2023-07-14T18:52:25.063 INFO:root:teuthology version: 0.0.1.dev145+gaa6a80e 2023-07-14T18:52:25.064 DEBUG:teuthology.run:Teuthology command: teuthology --name pdonnell-2023-07-14_16:44:47-fs:workload-wip-pdonnell-testing-20230714.134335-distro-default-smithi --archive /home/teuthworker/archive/pdonnell-2023-07-14_16:44:47-fs:workload-wip-pdonnell-testing-20230714.134335-distro-default-smithi/7338049 --verbose --description fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-comp omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/random export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/direct_io}} --owner scheduled_pdonnell@teuthology -- /home/teuthworker/archive/pdonnell-2023-07-14_16:44:47-fs:workload-wip-pdonnell-testing-20230714.134335-distro-default-smithi/7338049/orig.config.yaml 2023-07-14T18:52:25.121 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-07-14T18:52:30.229 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/pdonnell-2023-07-14_16:44:47-fs:workload-wip-pdonnell-testing-20230714.134335-distro-default-smithi/7338049 branch: wip-pdonnell-testing-20230714.134335 description: fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-comp omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/random export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/direct_io}} email: pdonnell@redhat.com first_in_suite: false job_id: '7338049' kernel: client: sha1: distro kdb: true sha1: distro ktype: distro last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi name: pdonnell-2023-07-14_16:44:47-fs:workload-wip-pdonnell-testing-20230714.134335-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 20 - machine: disk: 200 os_type: rhel os_version: '8.6' overrides: admin_socket: branch: wip-pdonnell-testing-20230714.134335 ceph: cephfs: max_mds: 5 session_timeout: 300 standby_replay: true conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 rados mon op timeout: 900 rados osd op timeout: 900 global: ms die on skipped message: false mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds bal fragment size max: 10000 mds bal merge size: 5 mds bal split bits: 3 mds bal split size: 100 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: cephsqlite lock renewal timeout: 900000 debug cephsqlite: 20 debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon op complaint time: 120 osd: bluestore block size: 96636764160 bluestore compression mode: aggressive bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 osd shutdown pgref assert: true osd_max_omap_entries_per_request: 10 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - overall HEALTH_ - \(FS_DEGRADED\) - \(MDS_FAILED\) - \(MDS_DEGRADED\) - \(FS_WITH_FAILED_MDS\) - \(MDS_DAMAGE\) - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(FS_INLINE_DATA_DEPRECATED\) - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding - slow metadata IO - SLOW_OPS - slow request sha1: 81c9387d2e7576da065747ec9c16d1c7d69016ec subvols: create: 2 subvol_options: --namespace-isolated --size 25000000000 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: osd default pool size: 2 ceph-fuse: client.0: mount_subvol_num: 0 check-counter: counters: mds: - mds.exported - mds.imported - mds.root_rsnaps - mds_server.req_mksnap_latency.avgcount - mds_server.req_rmsnap_latency.avgcount dry_run: true install: ceph: flavor: default sha1: 81c9387d2e7576da065747ec9c16d1c7d69016ec kclient: client.0: mount_subvol_num: 1 mntopts: - ms_mode=secure - wsync syntax: v2 selinux: whitelist: - scontext=system_u:system_r:logrotate_t:s0 - scontext=system_u:system_r:logrotate_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-pdonnell-testing-20230714.134335 sha1: 81c9387d2e7576da065747ec9c16d1c7d69016ec owner: scheduled_pdonnell@teuthology priority: 75 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.x - mds.a - mds.d - mds.g - mds.j - osd.0 - osd.3 - osd.6 - osd.9 - client.0 - - mon.b - mgr.y - mds.b - mds.e - mds.h - mds.k - osd.1 - osd.4 - osd.7 - osd.10 - - mon.c - mgr.z - mds.c - mds.f - mds.i - mds.l - osd.2 - osd.5 - osd.8 - osd.11 seed: 5179 sha1: 81c9387d2e7576da065747ec9c16d1c7d69016ec sleep_before_teardown: 0 subset: 285/512 suite: fs:workload suite_branch: wip-pdonnell-testing-20230714.134335 suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_81c9387d2e7576da065747ec9c16d1c7d69016ec/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: 81c9387d2e7576da065747ec9c16d1c7d69016ec syslog: ignorelist: - WARNING*.*check_session_state - WARNING*.*__ceph_remove_cap targets: smithi040.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCpUxo5ay/ucdkX9rleNMm8Ofp24XRXUtJPOK3Ji3XBpTiUgmmwneaY6316+3CG+k4K0p3IebanuyTvZApjm02/dNCvu8yD3rjEpWVgIuqYH8O3lEdN9DXcCX+7xHJE0m18+ZPsJQVsNrNJxY+5zqyNnBc8jpZq2LStsWwPOKPWhauANRkOBFSGDoLppWZbLx/yJs+R0qoNLLK9LpKQTUnqsapXvbD27OpPahzj/qbNxFPPyY6XrrldZ6qEO+EjJOujFLH3gzv0kJu10DCuzxrpz6ston4RPi1O3GgcHnF4Mqr/W9VwZ94UTMh+k4dIFRVqJEKrcwRHa5kFsvH/WhVx0Ygr/MiV/Q86mmV7JT+/MpzIRTtWyei5L9IBeZIE9cwzj+mOk6SD/t1roIVEGVV/uF2AyFDVBPr6F8oZ822OGpUyyaDsWTitPNdTX4TC3zdSrLc9tyDa1P74eH/ZLeTSf+C6pM/cepOhQaa6XzpDiPSuJUm9HOQxUlEAvcqa7Ac= smithi130.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLVvN3+LmzPZ4ixiwyCkS3KZLZ9R6MOPm+kPtaJbl8fExpzbk8TudXCcJWHBL7DGPWFvye2jT6sei9eiX0EUQYs= smithi191.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCpUxo5ay/ucdkX9rleNMm8Ofp24XRXUtJPOK3Ji3XBpTiUgmmwneaY6316+3CG+k4K0p3IebanuyTvZApjm02/dNCvu8yD3rjEpWVgIuqYH8O3lEdN9DXcCX+7xHJE0m18+ZPsJQVsNrNJxY+5zqyNnBc8jpZq2LStsWwPOKPWhauANRkOBFSGDoLppWZbLx/yJs+R0qoNLLK9LpKQTUnqsapXvbD27OpPahzj/qbNxFPPyY6XrrldZ6qEO+EjJOujFLH3gzv0kJu10DCuzxrpz6ston4RPi1O3GgcHnF4Mqr/W9VwZ94UTMh+k4dIFRVqJEKrcwRHa5kFsvH/WhVx0Ygr/MiV/Q86mmV7JT+/MpzIRTtWyei5L9IBeZIE9cwzj+mOk6SD/t1roIVEGVV/uF2AyFDVBPr6F8oZ822OGpUyyaDsWTitPNdTX4TC3zdSrLc9tyDa1P74eH/ZLeTSf+C6pM/cepOhQaa6XzpDiPSuJUm9HOQxUlEAvcqa7Ac= tasks: - pexec: all: - sudo cp /etc/containers/registries.conf /etc/containers/registries.conf.backup - sudo dnf -y module reset container-tools - sudo dnf -y module install container-tools:rhel8 --allowerasing --nobest - sudo cp /etc/containers/registries.conf.backup /etc/containers/registries.conf - install: extra_packages: deb: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror rpm: - python3-cephfs - cephfs-top - cephfs-mirror extra_system_packages: deb: - bison - flex - libelf-dev - libssl-dev - network-manager - iproute2 - util-linux - dump - indent - libaio-dev - libtool-bin - uuid-dev - xfslibs-dev - postgresql - postgresql-client - postgresql-common - postgresql-contrib rpm: - bison - flex - elfutils-libelf-devel - openssl-devel - NetworkManager - iproute - util-linux - libacl-devel - libaio-devel - libattr-devel - libtool - libuuid-devel - xfsdump - xfsprogs - xfsprogs-devel - libaio-devel - libtool - libuuid-devel - xfsprogs-devel - postgresql - postgresql-server - postgresql-contrib - cephadm: roleless: false - cephadm.shell: mon.a: - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - cephadm.shell: mon.a: - ceph fs dump - ceph osd dump - fs.ready: timeout: 300 - kclient: null - exec: mon.a: - ceph fs set cephfs balance_automate false - ceph fs subvolumegroup pin cephfs _nogroup random 0.01 - check-counter: {} - fwd_scrub: scrub_timeout: 900 sleep_between_iterations: 1 - exec: mon.a: - ceph mgr module enable snap_schedule - ceph config set mgr mgr/snap_schedule/allow_m_granularity true - ceph config set mgr mgr/snap_schedule/dump_on_update true - ceph fs snap-schedule add --fs=cephfs --path=/ --snap_schedule=1M - ceph fs snap-schedule retention add --fs=cephfs --path=/ --retention-spec-or-period=6M3h - ceph fs snap-schedule status --fs=cephfs --path=/ - ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true - date +%s > START_TIME - full_sequential_finally: - exec: mon.a: - date +%s > END_TIME - START_TIME=$(cat START_TIME); END_TIME=$(cat END_TIME); DIFF_TIME=$((600-(END_TIME-START_TIME))); if [ "$DIFF_TIME" -gt 0 ]; then sleep "$DIFF_TIME"; fi - ceph fs snap-schedule status --fs=cephfs --path=/ - ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true - background_exec: mon.a: - while sleep 13; do ceph tell mds.cephfs:0 flush journal; done - workunit: clients: all: - direct_io teuthology: fragments_dropped: [] meta: {} postmerge: - local function is_kupstream() return yaml.ktype == 'upstream' end - local function is_kdistro() return yaml.ktype == 'distro' end - local function is_fuse() return false end - local function syntax_version() return yaml.overrides.kclient.syntax end teuthology_branch: main teuthology_sha1: aa6a80e6cc655fff48452a7eca02070e61d074fc timestamp: 2023-07-14_16:44:47 tube: smithi user: pdonnell verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.23459 2023-07-14T18:52:30.229 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_81c9387d2e7576da065747ec9c16d1c7d69016ec/qa; will attempt to use it 2023-07-14T18:52:30.230 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_81c9387d2e7576da065747ec9c16d1c7d69016ec/qa/tasks 2023-07-14T18:52:30.230 INFO:teuthology.run_tasks:Running task internal.check_packages... 2023-07-14T18:52:30.232 INFO:teuthology.task.internal:Checking packages... 2023-07-14T18:52:30.261 INFO:teuthology.task.internal:Checking packages for os_type 'rhel', flavor 'default' and ceph hash '81c9387d2e7576da065747ec9c16d1c7d69016ec' 2023-07-14T18:52:30.261 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2023-07-14T18:52:30.262 INFO:teuthology.packaging:ref: None 2023-07-14T18:52:30.262 INFO:teuthology.packaging:tag: None 2023-07-14T18:52:30.262 INFO:teuthology.packaging:branch: wip-pdonnell-testing-20230714.134335 2023-07-14T18:52:30.262 INFO:teuthology.packaging:sha1: 81c9387d2e7576da065747ec9c16d1c7d69016ec 2023-07-14T18:52:30.263 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=wip-pdonnell-testing-20230714.134335 2023-07-14T18:52:35.591 INFO:teuthology.task.internal:Found packages for ceph version 18.0.0-4940.g81c9387d 2023-07-14T18:52:35.591 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2023-07-14T18:52:35.602 INFO:teuthology.task.internal:no buildpackages task found 2023-07-14T18:52:35.602 INFO:teuthology.run_tasks:Running task internal.save_config... 2023-07-14T18:52:35.640 INFO:teuthology.task.internal:Saving configuration 2023-07-14T18:52:35.662 INFO:teuthology.run_tasks:Running task internal.check_lock... 2023-07-14T18:52:35.668 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-07-14T18:52:40.698 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi040.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2023-07-14_16:44:47-fs:workload-wip-pdonnell-testing-20230714.134335-distro-default-smithi/7338049', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.6', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-07-14 18:46:28.583416', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLVvN3+LmzPZ4ixiwyCkS3KZLZ9R6MOPm+kPtaJbl8fExpzbk8TudXCcJWHBL7DGPWFvye2jT6sei9eiX0EUQYs='} 2023-07-14T18:52:48.773 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi130.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2023-07-14_16:44:47-fs:workload-wip-pdonnell-testing-20230714.134335-distro-default-smithi/7338049', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.6', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-07-14 18:46:28.580568', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLVvN3+LmzPZ4ixiwyCkS3KZLZ9R6MOPm+kPtaJbl8fExpzbk8TudXCcJWHBL7DGPWFvye2jT6sei9eiX0EUQYs='} 2023-07-14T18:52:48.796 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi191.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2023-07-14_16:44:47-fs:workload-wip-pdonnell-testing-20230714.134335-distro-default-smithi/7338049', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.6', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-07-14 18:46:28.582158', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLVvN3+LmzPZ4ixiwyCkS3KZLZ9R6MOPm+kPtaJbl8fExpzbk8TudXCcJWHBL7DGPWFvye2jT6sei9eiX0EUQYs='} 2023-07-14T18:52:48.796 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2023-07-14T18:52:48.803 INFO:teuthology.task.internal:roles: ubuntu@smithi040.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.d', 'mds.g', 'mds.j', 'osd.0', 'osd.3', 'osd.6', 'osd.9', 'client.0'] 2023-07-14T18:52:48.803 INFO:teuthology.task.internal:roles: ubuntu@smithi130.front.sepia.ceph.com - ['mon.b', 'mgr.y', 'mds.b', 'mds.e', 'mds.h', 'mds.k', 'osd.1', 'osd.4', 'osd.7', 'osd.10'] 2023-07-14T18:52:48.803 INFO:teuthology.task.internal:roles: ubuntu@smithi191.front.sepia.ceph.com - ['mon.c', 'mgr.z', 'mds.c', 'mds.f', 'mds.i', 'mds.l', 'osd.2', 'osd.5', 'osd.8', 'osd.11'] 2023-07-14T18:52:48.803 INFO:teuthology.run_tasks:Running task console_log... 2023-07-14T18:52:48.940 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fab23dd5430>, signals=[15]) 2023-07-14T18:52:48.941 INFO:teuthology.run_tasks:Running task internal.connect... 2023-07-14T18:52:48.948 INFO:teuthology.task.internal:Opening connections... 2023-07-14T18:52:48.948 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi040.front.sepia.ceph.com 2023-07-14T18:52:48.951 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi040.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-07-14T18:52:49.024 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi130.front.sepia.ceph.com 2023-07-14T18:52:49.025 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi130.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-07-14T18:52:49.093 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi191.front.sepia.ceph.com 2023-07-14T18:52:49.094 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi191.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-07-14T18:52:49.162 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2023-07-14T18:52:49.169 DEBUG:teuthology.orchestra.run.smithi040:> uname -m 2023-07-14T18:52:49.188 INFO:teuthology.orchestra.run.smithi040.stdout:x86_64 2023-07-14T18:52:49.189 DEBUG:teuthology.orchestra.run.smithi040:> cat /etc/os-release 2023-07-14T18:52:49.249 INFO:teuthology.orchestra.run.smithi040.stdout:NAME="Red Hat Enterprise Linux" 2023-07-14T18:52:49.249 INFO:teuthology.orchestra.run.smithi040.stdout:VERSION="8.6 (Ootpa)" 2023-07-14T18:52:49.250 INFO:teuthology.orchestra.run.smithi040.stdout:ID="rhel" 2023-07-14T18:52:49.250 INFO:teuthology.orchestra.run.smithi040.stdout:ID_LIKE="fedora" 2023-07-14T18:52:49.250 INFO:teuthology.orchestra.run.smithi040.stdout:VERSION_ID="8.6" 2023-07-14T18:52:49.250 INFO:teuthology.orchestra.run.smithi040.stdout:PLATFORM_ID="platform:el8" 2023-07-14T18:52:49.250 INFO:teuthology.orchestra.run.smithi040.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.6 (Ootpa)" 2023-07-14T18:52:49.250 INFO:teuthology.orchestra.run.smithi040.stdout:ANSI_COLOR="0;31" 2023-07-14T18:52:49.251 INFO:teuthology.orchestra.run.smithi040.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8::baseos" 2023-07-14T18:52:49.251 INFO:teuthology.orchestra.run.smithi040.stdout:HOME_URL="https://www.redhat.com/" 2023-07-14T18:52:49.251 INFO:teuthology.orchestra.run.smithi040.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/" 2023-07-14T18:52:49.251 INFO:teuthology.orchestra.run.smithi040.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-07-14T18:52:49.251 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-07-14T18:52:49.251 INFO:teuthology.orchestra.run.smithi040.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2023-07-14T18:52:49.251 INFO:teuthology.orchestra.run.smithi040.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.6 2023-07-14T18:52:49.252 INFO:teuthology.orchestra.run.smithi040.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2023-07-14T18:52:49.252 INFO:teuthology.orchestra.run.smithi040.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.6" 2023-07-14T18:52:49.252 INFO:teuthology.lock.ops:Updating smithi040.front.sepia.ceph.com on lock server 2023-07-14T18:52:59.305 DEBUG:teuthology.orchestra.run.smithi130:> uname -m 2023-07-14T18:52:59.326 INFO:teuthology.orchestra.run.smithi130.stdout:x86_64 2023-07-14T18:52:59.327 DEBUG:teuthology.orchestra.run.smithi130:> cat /etc/os-release 2023-07-14T18:52:59.385 INFO:teuthology.orchestra.run.smithi130.stdout:NAME="Red Hat Enterprise Linux" 2023-07-14T18:52:59.385 INFO:teuthology.orchestra.run.smithi130.stdout:VERSION="8.6 (Ootpa)" 2023-07-14T18:52:59.386 INFO:teuthology.orchestra.run.smithi130.stdout:ID="rhel" 2023-07-14T18:52:59.386 INFO:teuthology.orchestra.run.smithi130.stdout:ID_LIKE="fedora" 2023-07-14T18:52:59.386 INFO:teuthology.orchestra.run.smithi130.stdout:VERSION_ID="8.6" 2023-07-14T18:52:59.386 INFO:teuthology.orchestra.run.smithi130.stdout:PLATFORM_ID="platform:el8" 2023-07-14T18:52:59.386 INFO:teuthology.orchestra.run.smithi130.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.6 (Ootpa)" 2023-07-14T18:52:59.386 INFO:teuthology.orchestra.run.smithi130.stdout:ANSI_COLOR="0;31" 2023-07-14T18:52:59.387 INFO:teuthology.orchestra.run.smithi130.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8::baseos" 2023-07-14T18:52:59.387 INFO:teuthology.orchestra.run.smithi130.stdout:HOME_URL="https://www.redhat.com/" 2023-07-14T18:52:59.387 INFO:teuthology.orchestra.run.smithi130.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/" 2023-07-14T18:52:59.387 INFO:teuthology.orchestra.run.smithi130.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-07-14T18:52:59.387 INFO:teuthology.orchestra.run.smithi130.stdout: 2023-07-14T18:52:59.387 INFO:teuthology.orchestra.run.smithi130.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2023-07-14T18:52:59.387 INFO:teuthology.orchestra.run.smithi130.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.6 2023-07-14T18:52:59.387 INFO:teuthology.orchestra.run.smithi130.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2023-07-14T18:52:59.388 INFO:teuthology.orchestra.run.smithi130.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.6" 2023-07-14T18:52:59.388 INFO:teuthology.lock.ops:Updating smithi130.front.sepia.ceph.com on lock server 2023-07-14T18:53:04.420 DEBUG:teuthology.orchestra.run.smithi191:> uname -m 2023-07-14T18:53:04.441 INFO:teuthology.orchestra.run.smithi191.stdout:x86_64 2023-07-14T18:53:04.442 DEBUG:teuthology.orchestra.run.smithi191:> cat /etc/os-release 2023-07-14T18:53:04.500 INFO:teuthology.orchestra.run.smithi191.stdout:NAME="Red Hat Enterprise Linux" 2023-07-14T18:53:04.501 INFO:teuthology.orchestra.run.smithi191.stdout:VERSION="8.6 (Ootpa)" 2023-07-14T18:53:04.501 INFO:teuthology.orchestra.run.smithi191.stdout:ID="rhel" 2023-07-14T18:53:04.501 INFO:teuthology.orchestra.run.smithi191.stdout:ID_LIKE="fedora" 2023-07-14T18:53:04.501 INFO:teuthology.orchestra.run.smithi191.stdout:VERSION_ID="8.6" 2023-07-14T18:53:04.501 INFO:teuthology.orchestra.run.smithi191.stdout:PLATFORM_ID="platform:el8" 2023-07-14T18:53:04.501 INFO:teuthology.orchestra.run.smithi191.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.6 (Ootpa)" 2023-07-14T18:53:04.501 INFO:teuthology.orchestra.run.smithi191.stdout:ANSI_COLOR="0;31" 2023-07-14T18:53:04.502 INFO:teuthology.orchestra.run.smithi191.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8::baseos" 2023-07-14T18:53:04.502 INFO:teuthology.orchestra.run.smithi191.stdout:HOME_URL="https://www.redhat.com/" 2023-07-14T18:53:04.502 INFO:teuthology.orchestra.run.smithi191.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/" 2023-07-14T18:53:04.502 INFO:teuthology.orchestra.run.smithi191.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-07-14T18:53:04.502 INFO:teuthology.orchestra.run.smithi191.stdout: 2023-07-14T18:53:04.502 INFO:teuthology.orchestra.run.smithi191.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2023-07-14T18:53:04.502 INFO:teuthology.orchestra.run.smithi191.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.6 2023-07-14T18:53:04.503 INFO:teuthology.orchestra.run.smithi191.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2023-07-14T18:53:04.503 INFO:teuthology.orchestra.run.smithi191.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.6" 2023-07-14T18:53:04.503 INFO:teuthology.lock.ops:Updating smithi191.front.sepia.ceph.com on lock server 2023-07-14T18:53:09.537 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2023-07-14T18:53:09.548 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2023-07-14T18:53:09.580 INFO:teuthology.task.internal:Checking for old test directory... 2023-07-14T18:53:09.580 DEBUG:teuthology.orchestra.run.smithi040:> test '!' -e /home/ubuntu/cephtest 2023-07-14T18:53:09.583 DEBUG:teuthology.orchestra.run.smithi130:> test '!' -e /home/ubuntu/cephtest 2023-07-14T18:53:09.585 DEBUG:teuthology.orchestra.run.smithi191:> test '!' -e /home/ubuntu/cephtest 2023-07-14T18:53:09.604 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2023-07-14T18:53:09.613 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2023-07-14T18:53:09.614 DEBUG:teuthology.orchestra.run.smithi040:> test -z $(ls -A /var/lib/ceph) 2023-07-14T18:53:09.643 DEBUG:teuthology.orchestra.run.smithi130:> test -z $(ls -A /var/lib/ceph) 2023-07-14T18:53:09.645 DEBUG:teuthology.orchestra.run.smithi191:> test -z $(ls -A /var/lib/ceph) 2023-07-14T18:53:09.674 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2023-07-14T18:53:15.472 INFO:teuthology.run_tasks:Running task kernel... 2023-07-14T18:53:15.489 INFO:teuthology.task.kernel:normalize config orig: {'client': {'sha1': 'distro'}, 'kdb': True, 'sha1': 'distro'} 2023-07-14T18:53:15.490 INFO:teuthology.task.kernel:normalize config final: {'client.0': {'sha1': 'distro'}} 2023-07-14T18:53:15.490 INFO:teuthology.task.kernel:config {'client.0': {'sha1': 'distro'}}, timeout 300 2023-07-14T18:53:15.491 DEBUG:teuthology.orchestra.run.smithi040:> test -f /run/.containerenv -o -f /.dockerenv 2023-07-14T18:53:15.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-07-14T18:53:15.511 DEBUG:teuthology.orchestra.run.smithi040:> uname -r 2023-07-14T18:53:15.571 INFO:teuthology.orchestra.run.smithi040.stdout:4.18.0-372.9.1.el8.x86_64 2023-07-14T18:53:15.572 INFO:teuthology.task.kernel:Running kernel on smithi040: 4.18.0-372.9.1.el8.x86_64 2023-07-14T18:53:15.572 DEBUG:teuthology.orchestra.run.smithi040:> sudo yum install -y kernel 2023-07-14T18:53:16.884 INFO:teuthology.orchestra.run.smithi040.stdout:Updating Subscription Management repositories. 2023-07-14T18:53:16.884 INFO:teuthology.orchestra.run.smithi040.stdout:Unable to read consumer identity 2023-07-14T18:53:18.437 INFO:teuthology.orchestra.run.smithi040.stdout:Copr repo for python3-asyncssh owned by ceph 12 kB/s | 3.5 kB 00:00 2023-07-14T18:53:31.253 INFO:teuthology.orchestra.run.smithi040.stdout:Extra Packages for Enterprise Linux 1.2 MB/s | 16 MB 00:12 2023-07-14T18:53:34.994 INFO:teuthology.orchestra.run.smithi040.stdout:lab-extras 468 kB/s | 24 kB 00:00 2023-07-14T18:53:35.714 INFO:teuthology.orchestra.run.smithi040.stdout:Package kernel-4.18.0-372.9.1.el8.x86_64 is already installed. 2023-07-14T18:53:35.758 INFO:teuthology.orchestra.run.smithi040.stdout:Dependencies resolved. 2023-07-14T18:53:35.759 INFO:teuthology.orchestra.run.smithi040.stdout:Nothing to do. 2023-07-14T18:53:35.760 INFO:teuthology.orchestra.run.smithi040.stdout:Complete! 2023-07-14T18:53:35.910 DEBUG:teuthology.orchestra.run.smithi040:> echo no | sudo yum reinstall kernel || true 2023-07-14T18:53:36.260 INFO:teuthology.orchestra.run.smithi040.stdout:Updating Subscription Management repositories. 2023-07-14T18:53:36.261 INFO:teuthology.orchestra.run.smithi040.stdout:Unable to read consumer identity 2023-07-14T18:53:36.542 INFO:teuthology.orchestra.run.smithi040.stdout:Last metadata expiration check: 0:00:02 ago on Fri 14 Jul 2023 06:53:34 PM UTC. 2023-07-14T18:53:36.590 INFO:teuthology.orchestra.run.smithi040.stdout:Installed package kernel-4.18.0-372.9.1.el8.x86_64 (from anaconda) not available. 2023-07-14T18:53:36.629 INFO:teuthology.orchestra.run.smithi040.stderr:Error: No packages marked for reinstall. 2023-07-14T18:53:36.676 DEBUG:teuthology.orchestra.run.smithi040:> sudo yum reinstall -y kernel || true 2023-07-14T18:53:37.027 INFO:teuthology.orchestra.run.smithi040.stdout:Updating Subscription Management repositories. 2023-07-14T18:53:37.027 INFO:teuthology.orchestra.run.smithi040.stdout:Unable to read consumer identity 2023-07-14T18:53:37.302 INFO:teuthology.orchestra.run.smithi040.stdout:Last metadata expiration check: 0:00:03 ago on Fri 14 Jul 2023 06:53:34 PM UTC. 2023-07-14T18:53:37.345 INFO:teuthology.orchestra.run.smithi040.stdout:Installed package kernel-4.18.0-372.9.1.el8.x86_64 (from anaconda) not available. 2023-07-14T18:53:37.375 INFO:teuthology.orchestra.run.smithi040.stderr:Error: No packages marked for reinstall. 2023-07-14T18:53:37.425 DEBUG:teuthology.orchestra.run.smithi040:> rpm -q kernel | sort -rV | head -n 1 2023-07-14T18:53:37.508 INFO:teuthology.orchestra.run.smithi040.stdout:kernel-4.18.0-372.9.1.el8.x86_64 2023-07-14T18:53:37.508 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-372.9.1.el8.x86_64 2023-07-14T18:53:37.508 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2023-07-14T18:53:37.509 DEBUG:teuthology.parallel:result is None 2023-07-14T18:53:37.509 INFO:teuthology.run_tasks:Running task internal.base... 2023-07-14T18:53:37.516 INFO:teuthology.task.internal:Creating test directory... 2023-07-14T18:53:37.516 DEBUG:teuthology.orchestra.run.smithi040:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-07-14T18:53:37.518 DEBUG:teuthology.orchestra.run.smithi130:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-07-14T18:53:37.521 DEBUG:teuthology.orchestra.run.smithi191:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-07-14T18:53:37.540 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2023-07-14T18:53:37.546 INFO:teuthology.run_tasks:Running task internal.archive... 2023-07-14T18:53:37.574 INFO:teuthology.task.internal:Creating archive directory... 2023-07-14T18:53:37.574 DEBUG:teuthology.orchestra.run.smithi040:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-07-14T18:53:37.576 DEBUG:teuthology.orchestra.run.smithi130:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-07-14T18:53:37.580 DEBUG:teuthology.orchestra.run.smithi191:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-07-14T18:53:37.622 INFO:teuthology.run_tasks:Running task internal.coredump... 2023-07-14T18:53:37.628 INFO:teuthology.task.internal:Enabling coredump saving... 2023-07-14T18:53:37.629 DEBUG:teuthology.orchestra.run.smithi130:> test -f /run/.containerenv -o -f /.dockerenv 2023-07-14T18:53:37.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-07-14T18:53:37.669 DEBUG:teuthology.orchestra.run.smithi191:> test -f /run/.containerenv -o -f /.dockerenv 2023-07-14T18:53:37.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-07-14T18:53:37.688 DEBUG:teuthology.orchestra.run.smithi040:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-07-14T18:53:37.690 DEBUG:teuthology.orchestra.run.smithi130:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-07-14T18:53:37.711 DEBUG:teuthology.orchestra.run.smithi191:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-07-14T18:53:37.728 INFO:teuthology.orchestra.run.smithi040.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-07-14T18:53:37.740 INFO:teuthology.orchestra.run.smithi040.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-07-14T18:53:37.749 INFO:teuthology.orchestra.run.smithi130.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-07-14T18:53:37.781 INFO:teuthology.orchestra.run.smithi191.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-07-14T18:53:37.791 INFO:teuthology.orchestra.run.smithi130.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-07-14T18:53:37.819 INFO:teuthology.orchestra.run.smithi191.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-07-14T18:53:37.821 INFO:teuthology.run_tasks:Running task internal.sudo... 2023-07-14T18:53:37.828 INFO:teuthology.task.internal:Configuring sudo... 2023-07-14T18:53:37.829 DEBUG:teuthology.orchestra.run.smithi040:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-07-14T18:53:37.831 DEBUG:teuthology.orchestra.run.smithi130:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-07-14T18:53:37.834 DEBUG:teuthology.orchestra.run.smithi191:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-07-14T18:53:37.864 INFO:teuthology.run_tasks:Running task internal.syslog... 2023-07-14T18:53:37.892 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2023-07-14T18:53:37.893 DEBUG:teuthology.orchestra.run.smithi040:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-07-14T18:53:37.898 DEBUG:teuthology.orchestra.run.smithi130:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-07-14T18:53:37.903 DEBUG:teuthology.orchestra.run.smithi191:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-07-14T18:53:37.920 DEBUG:teuthology.orchestra.run.smithi040:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-07-14T18:53:42.026 DEBUG:teuthology.orchestra.run.smithi040:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-07-14T18:53:42.065 DEBUG:teuthology.orchestra.run.smithi040:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-07-14T18:53:42.152 DEBUG:teuthology.orchestra.run.smithi040:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-07-14T18:53:42.199 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2023-07-14T18:53:42.200 DEBUG:teuthology.orchestra.run.smithi040:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-07-14T18:53:42.298 DEBUG:teuthology.orchestra.run.smithi130:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-07-14T18:53:42.349 DEBUG:teuthology.orchestra.run.smithi130:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-07-14T18:53:42.410 DEBUG:teuthology.orchestra.run.smithi130:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-07-14T18:53:42.452 DEBUG:teuthology.orchestra.run.smithi130:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-07-14T18:53:42.499 DEBUG:teuthology.orchestra.run.smithi130:> set -ex 2023-07-14T18:53:42.499 DEBUG:teuthology.orchestra.run.smithi130:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-07-14T18:53:42.568 DEBUG:teuthology.orchestra.run.smithi191:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-07-14T18:53:42.619 DEBUG:teuthology.orchestra.run.smithi191:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-07-14T18:53:42.667 DEBUG:teuthology.orchestra.run.smithi191:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-07-14T18:53:42.755 DEBUG:teuthology.orchestra.run.smithi191:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-07-14T18:53:42.796 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2023-07-14T18:53:42.796 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-07-14T18:53:42.865 DEBUG:teuthology.orchestra.run.smithi040:> sudo service rsyslog restart 2023-07-14T18:53:42.867 DEBUG:teuthology.orchestra.run.smithi130:> sudo service rsyslog restart 2023-07-14T18:53:42.870 DEBUG:teuthology.orchestra.run.smithi191:> sudo service rsyslog restart 2023-07-14T18:53:42.934 INFO:teuthology.orchestra.run.smithi040.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-07-14T18:53:42.940 INFO:teuthology.orchestra.run.smithi130.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-07-14T18:53:42.978 INFO:teuthology.orchestra.run.smithi191.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-07-14T18:53:43.258 INFO:teuthology.run_tasks:Running task internal.timer... 2023-07-14T18:53:43.265 INFO:teuthology.task.internal:Starting timer... 2023-07-14T18:53:43.266 INFO:teuthology.run_tasks:Running task pcp... 2023-07-14T18:53:43.302 INFO:teuthology.run_tasks:Running task selinux... 2023-07-14T18:53:43.334 DEBUG:teuthology.task:Applying overrides for task selinux: {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0', 'scontext=system_u:system_r:logrotate_t:s0']} 2023-07-14T18:53:43.335 DEBUG:teuthology.orchestra.run.smithi040:> sudo service auditd rotate 2023-07-14T18:53:43.400 INFO:teuthology.orchestra.run.smithi040.stdout:Rotating logs: 2023-07-14T18:53:43.401 DEBUG:teuthology.orchestra.run.smithi130:> sudo service auditd rotate 2023-07-14T18:53:43.474 INFO:teuthology.orchestra.run.smithi130.stdout:Rotating logs: 2023-07-14T18:53:43.475 DEBUG:teuthology.orchestra.run.smithi191:> sudo service auditd rotate 2023-07-14T18:53:43.543 INFO:teuthology.orchestra.run.smithi191.stdout:Rotating logs: 2023-07-14T18:53:43.545 DEBUG:teuthology.task.selinux:Getting current SELinux state 2023-07-14T18:53:43.546 DEBUG:teuthology.orchestra.run.smithi040:> /usr/sbin/getenforce 2023-07-14T18:53:43.571 INFO:teuthology.orchestra.run.smithi040.stdout:Permissive 2023-07-14T18:53:43.572 DEBUG:teuthology.orchestra.run.smithi130:> /usr/sbin/getenforce 2023-07-14T18:53:43.596 INFO:teuthology.orchestra.run.smithi130.stdout:Permissive 2023-07-14T18:53:43.597 DEBUG:teuthology.orchestra.run.smithi191:> /usr/sbin/getenforce 2023-07-14T18:53:43.621 INFO:teuthology.orchestra.run.smithi191.stdout:Permissive 2023-07-14T18:53:43.622 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi040.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi130.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi191.front.sepia.ceph.com': 'permissive'} 2023-07-14T18:53:43.622 DEBUG:teuthology.orchestra.run.smithi040:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-07-14T18:53:43.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-07-14T18:53:43.653 DEBUG:teuthology.orchestra.run.smithi130:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-07-14T18:53:43.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-07-14T18:53:43.680 DEBUG:teuthology.orchestra.run.smithi191:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-07-14T18:53:43.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-07-14T18:53:43.709 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2023-07-14T18:53:43.709 DEBUG:teuthology.orchestra.run.smithi040:> sudo /usr/sbin/setenforce permissive 2023-07-14T18:53:43.738 DEBUG:teuthology.orchestra.run.smithi130:> sudo /usr/sbin/setenforce permissive 2023-07-14T18:53:43.765 DEBUG:teuthology.orchestra.run.smithi191:> sudo /usr/sbin/setenforce permissive 2023-07-14T18:53:43.791 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2023-07-14T18:53:58.734 INFO:teuthology.repo_utils:/home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main was just updated or references a specific commit; assuming it is current 2023-07-14T18:53:58.735 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2023-07-14T18:53:58.753 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2023-07-14T18:53:58.754 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi040.front.sepia.ceph.com,smithi130.front.sepia.ceph.com,smithi191.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2023-07-14T19:03:28.350 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi040.front.sepia.ceph.com'), Remote(name='ubuntu@smithi130.front.sepia.ceph.com'), Remote(name='ubuntu@smithi191.front.sepia.ceph.com')] 2023-07-14T19:03:28.352 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi040.front.sepia.ceph.com' 2023-07-14T19:03:28.353 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi040.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-07-14T19:03:33.428 DEBUG:teuthology.orchestra.run.smithi040:> true 2023-07-14T19:03:33.501 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi040.front.sepia.ceph.com' 2023-07-14T19:03:33.502 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi130.front.sepia.ceph.com' 2023-07-14T19:03:33.503 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi130.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-07-14T19:03:38.578 DEBUG:teuthology.orchestra.run.smithi130:> true 2023-07-14T19:03:38.606 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi130.front.sepia.ceph.com' 2023-07-14T19:03:38.606 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi191.front.sepia.ceph.com' 2023-07-14T19:03:38.607 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi191.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-07-14T19:03:38.677 DEBUG:teuthology.orchestra.run.smithi191:> true 2023-07-14T19:03:38.751 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi191.front.sepia.ceph.com' 2023-07-14T19:03:38.752 INFO:teuthology.run_tasks:Running task clock... 2023-07-14T19:03:38.762 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2023-07-14T19:03:38.763 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-07-14T19:03:38.764 DEBUG:teuthology.orchestra.run.smithi040:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-07-14T19:03:38.767 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-07-14T19:03:38.767 DEBUG:teuthology.orchestra.run.smithi130:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-07-14T19:03:38.770 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-07-14T19:03:38.771 DEBUG:teuthology.orchestra.run.smithi191:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-07-14T19:03:38.803 INFO:teuthology.orchestra.run.smithi040.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2023-07-14T19:03:38.804 INFO:teuthology.orchestra.run.smithi130.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2023-07-14T19:03:38.820 INFO:teuthology.orchestra.run.smithi040.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2023-07-14T19:03:38.821 INFO:teuthology.orchestra.run.smithi130.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2023-07-14T19:03:38.826 INFO:teuthology.orchestra.run.smithi191.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2023-07-14T19:03:38.842 INFO:teuthology.orchestra.run.smithi191.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2023-07-14T19:03:38.887 INFO:teuthology.orchestra.run.smithi040.stderr:sudo: ntpd: command not found 2023-07-14T19:03:38.898 INFO:teuthology.orchestra.run.smithi130.stderr:sudo: ntpd: command not found 2023-07-14T19:03:38.899 INFO:teuthology.orchestra.run.smithi040.stdout:506 Cannot talk to daemon 2023-07-14T19:03:38.911 INFO:teuthology.orchestra.run.smithi130.stdout:506 Cannot talk to daemon 2023-07-14T19:03:38.915 INFO:teuthology.orchestra.run.smithi040.stderr:Failed to start ntp.service: Unit ntp.service not found. 2023-07-14T19:03:38.925 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: ntpd: command not found 2023-07-14T19:03:38.926 INFO:teuthology.orchestra.run.smithi130.stderr:Failed to start ntp.service: Unit ntp.service not found. 2023-07-14T19:03:38.930 INFO:teuthology.orchestra.run.smithi040.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2023-07-14T19:03:38.937 INFO:teuthology.orchestra.run.smithi191.stdout:506 Cannot talk to daemon 2023-07-14T19:03:38.940 INFO:teuthology.orchestra.run.smithi130.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2023-07-14T19:03:38.952 INFO:teuthology.orchestra.run.smithi191.stderr:Failed to start ntp.service: Unit ntp.service not found. 2023-07-14T19:03:38.966 INFO:teuthology.orchestra.run.smithi191.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2023-07-14T19:03:38.998 INFO:teuthology.orchestra.run.smithi040.stderr:bash: ntpq: command not found 2023-07-14T19:03:39.002 INFO:teuthology.orchestra.run.smithi040.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-07-14T19:03:39.002 INFO:teuthology.orchestra.run.smithi040.stdout:=============================================================================== 2023-07-14T19:03:39.003 INFO:teuthology.orchestra.run.smithi040.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-07-14T19:03:39.003 INFO:teuthology.orchestra.run.smithi040.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-07-14T19:03:39.003 INFO:teuthology.orchestra.run.smithi040.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-07-14T19:03:39.003 INFO:teuthology.orchestra.run.smithi040.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-07-14T19:03:39.012 INFO:teuthology.orchestra.run.smithi130.stderr:bash: ntpq: command not found 2023-07-14T19:03:39.015 INFO:teuthology.orchestra.run.smithi130.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-07-14T19:03:39.015 INFO:teuthology.orchestra.run.smithi130.stdout:=============================================================================== 2023-07-14T19:03:39.015 INFO:teuthology.orchestra.run.smithi130.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-07-14T19:03:39.016 INFO:teuthology.orchestra.run.smithi130.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-07-14T19:03:39.016 INFO:teuthology.orchestra.run.smithi130.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-07-14T19:03:39.016 INFO:teuthology.orchestra.run.smithi130.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-07-14T19:03:39.044 INFO:teuthology.orchestra.run.smithi191.stderr:bash: ntpq: command not found 2023-07-14T19:03:39.047 INFO:teuthology.orchestra.run.smithi191.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-07-14T19:03:39.048 INFO:teuthology.orchestra.run.smithi191.stdout:=============================================================================== 2023-07-14T19:03:39.048 INFO:teuthology.orchestra.run.smithi191.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-07-14T19:03:39.048 INFO:teuthology.orchestra.run.smithi191.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-07-14T19:03:39.048 INFO:teuthology.orchestra.run.smithi191.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-07-14T19:03:39.048 INFO:teuthology.orchestra.run.smithi191.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-07-14T19:03:39.049 INFO:teuthology.run_tasks:Running task pexec... 2023-07-14T19:03:39.061 INFO:teuthology.task.pexec:Executing custom commands... 2023-07-14T19:03:39.062 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi040.front.sepia.ceph.com 2023-07-14T19:03:39.062 DEBUG:teuthology.orchestra.run.smithi040:> TESTDIR=/home/ubuntu/cephtest bash -s 2023-07-14T19:03:39.063 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi130.front.sepia.ceph.com 2023-07-14T19:03:39.063 DEBUG:teuthology.orchestra.run.smithi130:> TESTDIR=/home/ubuntu/cephtest bash -s 2023-07-14T19:03:39.064 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi191.front.sepia.ceph.com 2023-07-14T19:03:39.064 DEBUG:teuthology.orchestra.run.smithi191:> TESTDIR=/home/ubuntu/cephtest bash -s 2023-07-14T19:03:39.438 INFO:teuthology.orchestra.run.smithi130.stdout:Updating Subscription Management repositories. 2023-07-14T19:03:39.457 INFO:teuthology.orchestra.run.smithi040.stdout:Updating Subscription Management repositories. 2023-07-14T19:03:39.461 INFO:teuthology.orchestra.run.smithi191.stdout:Updating Subscription Management repositories. 2023-07-14T19:03:40.620 INFO:teuthology.orchestra.run.smithi130.stdout:Last metadata expiration check: 0:02:08 ago on Fri 14 Jul 2023 07:01:32 PM UTC. 2023-07-14T19:03:40.664 INFO:teuthology.orchestra.run.smithi191.stdout:Last metadata expiration check: 0:01:58 ago on Fri 14 Jul 2023 07:01:42 PM UTC. 2023-07-14T19:03:40.703 INFO:teuthology.orchestra.run.smithi040.stdout:Last metadata expiration check: 0:01:53 ago on Fri 14 Jul 2023 07:01:47 PM UTC. 2023-07-14T19:03:42.332 INFO:teuthology.orchestra.run.smithi130.stdout:Dependencies resolved. 2023-07-14T19:03:42.333 INFO:teuthology.orchestra.run.smithi130.stdout:================================================================================ 2023-07-14T19:03:42.334 INFO:teuthology.orchestra.run.smithi130.stdout: Package Architecture Version Repository Size 2023-07-14T19:03:42.334 INFO:teuthology.orchestra.run.smithi130.stdout:================================================================================ 2023-07-14T19:03:42.334 INFO:teuthology.orchestra.run.smithi130.stdout:Resetting modules: 2023-07-14T19:03:42.334 INFO:teuthology.orchestra.run.smithi130.stdout: container-tools 2023-07-14T19:03:42.334 INFO:teuthology.orchestra.run.smithi130.stdout: 2023-07-14T19:03:42.334 INFO:teuthology.orchestra.run.smithi130.stdout:Transaction Summary 2023-07-14T19:03:42.335 INFO:teuthology.orchestra.run.smithi130.stdout:================================================================================ 2023-07-14T19:03:42.335 INFO:teuthology.orchestra.run.smithi130.stdout: 2023-07-14T19:03:42.532 INFO:teuthology.orchestra.run.smithi191.stdout:Dependencies resolved. 2023-07-14T19:03:42.533 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2023-07-14T19:03:42.533 INFO:teuthology.orchestra.run.smithi191.stdout: Package Architecture Version Repository Size 2023-07-14T19:03:42.533 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2023-07-14T19:03:42.533 INFO:teuthology.orchestra.run.smithi191.stdout:Resetting modules: 2023-07-14T19:03:42.533 INFO:teuthology.orchestra.run.smithi191.stdout: container-tools 2023-07-14T19:03:42.534 INFO:teuthology.orchestra.run.smithi191.stdout: 2023-07-14T19:03:42.534 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction Summary 2023-07-14T19:03:42.534 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2023-07-14T19:03:42.534 INFO:teuthology.orchestra.run.smithi191.stdout: 2023-07-14T19:03:42.680 INFO:teuthology.orchestra.run.smithi040.stdout:Dependencies resolved. 2023-07-14T19:03:42.681 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2023-07-14T19:03:42.682 INFO:teuthology.orchestra.run.smithi040.stdout: Package Architecture Version Repository Size 2023-07-14T19:03:42.682 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2023-07-14T19:03:42.682 INFO:teuthology.orchestra.run.smithi040.stdout:Resetting modules: 2023-07-14T19:03:42.682 INFO:teuthology.orchestra.run.smithi040.stdout: container-tools 2023-07-14T19:03:42.682 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-07-14T19:03:42.682 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction Summary 2023-07-14T19:03:42.682 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2023-07-14T19:03:42.683 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-07-14T19:03:43.321 INFO:teuthology.orchestra.run.smithi130.stdout:Last metadata expiration check: 0:02:10 ago on Fri 14 Jul 2023 07:01:32 PM UTC. 2023-07-14T19:03:43.497 INFO:teuthology.orchestra.run.smithi191.stdout:Last metadata expiration check: 0:02:01 ago on Fri 14 Jul 2023 07:01:42 PM UTC. 2023-07-14T19:03:43.677 INFO:teuthology.orchestra.run.smithi040.stdout:Last metadata expiration check: 0:01:56 ago on Fri 14 Jul 2023 07:01:47 PM UTC. 2023-07-14T19:03:49.821 INFO:teuthology.orchestra.run.smithi191.stdout:Complete! 2023-07-14T19:03:50.074 INFO:teuthology.orchestra.run.smithi040.stdout:Complete! 2023-07-14T19:03:50.132 INFO:teuthology.orchestra.run.smithi130.stdout:Complete! 2023-07-14T19:03:50.289 INFO:teuthology.orchestra.run.smithi191.stdout:Updating Subscription Management repositories. 2023-07-14T19:03:50.559 INFO:teuthology.orchestra.run.smithi040.stdout:Updating Subscription Management repositories. 2023-07-14T19:03:50.610 INFO:teuthology.orchestra.run.smithi130.stdout:Updating Subscription Management repositories. 2023-07-14T19:03:52.170 INFO:teuthology.orchestra.run.smithi040.stdout:Last metadata expiration check: 0:02:04 ago on Fri 14 Jul 2023 07:01:47 PM UTC. 2023-07-14T19:03:52.196 INFO:teuthology.orchestra.run.smithi191.stdout:Last metadata expiration check: 0:02:09 ago on Fri 14 Jul 2023 07:01:42 PM UTC. 2023-07-14T19:03:52.204 INFO:teuthology.orchestra.run.smithi130.stdout:Last metadata expiration check: 0:02:19 ago on Fri 14 Jul 2023 07:01:32 PM UTC. 2023-07-14T19:03:53.967 INFO:teuthology.orchestra.run.smithi191.stdout:Dependencies resolved. 2023-07-14T19:03:54.059 INFO:teuthology.orchestra.run.smithi130.stdout:Dependencies resolved. 2023-07-14T19:03:54.060 INFO:teuthology.orchestra.run.smithi191.stdout:============================================================================================================ 2023-07-14T19:03:54.061 INFO:teuthology.orchestra.run.smithi191.stdout: Package Arch Version Repository Size 2023-07-14T19:03:54.061 INFO:teuthology.orchestra.run.smithi191.stdout:============================================================================================================ 2023-07-14T19:03:54.061 INFO:teuthology.orchestra.run.smithi191.stdout:Installing group/module packages: 2023-07-14T19:03:54.061 INFO:teuthology.orchestra.run.smithi191.stdout: buildah x86_64 1:1.26.2-1.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 8.0 M 2023-07-14T19:03:54.061 INFO:teuthology.orchestra.run.smithi191.stdout: cockpit-podman noarch 49.1-1.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 524 k 2023-07-14T19:03:54.061 INFO:teuthology.orchestra.run.smithi191.stdout: crun x86_64 1.5-1.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 213 k 2023-07-14T19:03:54.062 INFO:teuthology.orchestra.run.smithi191.stdout: python3-podman noarch 4.0.1-1.module+el8.6.0+16837+85b82211 rhel-8-for-x86_64-appstream-rpms 149 k 2023-07-14T19:03:54.062 INFO:teuthology.orchestra.run.smithi191.stdout: skopeo x86_64 2:1.9.1-1.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 7.0 M 2023-07-14T19:03:54.062 INFO:teuthology.orchestra.run.smithi191.stdout: toolbox x86_64 0.0.99.3-0.6.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 2.2 M 2023-07-14T19:03:54.062 INFO:teuthology.orchestra.run.smithi191.stdout: udica noarch 0.2.6-4.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 49 k 2023-07-14T19:03:54.062 INFO:teuthology.orchestra.run.smithi191.stdout:Installing dependencies: 2023-07-14T19:03:54.062 INFO:teuthology.orchestra.run.smithi191.stdout: python3-pytoml noarch 0.1.14-5.git7dea353.el8 rhel-8-for-x86_64-appstream-rpms 25 k 2023-07-14T19:03:54.062 INFO:teuthology.orchestra.run.smithi191.stdout: python3-pyxdg noarch 0.25-16.el8 rhel-8-for-x86_64-appstream-rpms 94 k 2023-07-14T19:03:54.062 INFO:teuthology.orchestra.run.smithi191.stdout: yajl x86_64 2.1.0-10.el8 rhel-8-for-x86_64-appstream-rpms 41 k 2023-07-14T19:03:54.063 INFO:teuthology.orchestra.run.smithi191.stdout:Installing module profiles: 2023-07-14T19:03:54.063 INFO:teuthology.orchestra.run.smithi191.stdout: container-tools/common 2023-07-14T19:03:54.063 INFO:teuthology.orchestra.run.smithi191.stdout: 2023-07-14T19:03:54.063 INFO:teuthology.orchestra.run.smithi191.stdout:Enabling module streams: 2023-07-14T19:03:54.063 INFO:teuthology.orchestra.run.smithi191.stdout: container-tools rhel8 2023-07-14T19:03:54.063 INFO:teuthology.orchestra.run.smithi191.stdout: 2023-07-14T19:03:54.063 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction Summary 2023-07-14T19:03:54.064 INFO:teuthology.orchestra.run.smithi191.stdout:============================================================================================================ 2023-07-14T19:03:54.064 INFO:teuthology.orchestra.run.smithi191.stdout:Install 10 Packages 2023-07-14T19:03:54.064 INFO:teuthology.orchestra.run.smithi191.stdout: 2023-07-14T19:03:54.064 INFO:teuthology.orchestra.run.smithi191.stdout:Total download size: 18 M 2023-07-14T19:03:54.065 INFO:teuthology.orchestra.run.smithi191.stdout:Installed size: 65 M 2023-07-14T19:03:54.065 INFO:teuthology.orchestra.run.smithi191.stdout:Downloading Packages: 2023-07-14T19:03:54.151 INFO:teuthology.orchestra.run.smithi040.stdout:Dependencies resolved. 2023-07-14T19:03:54.153 INFO:teuthology.orchestra.run.smithi130.stdout:============================================================================================================ 2023-07-14T19:03:54.153 INFO:teuthology.orchestra.run.smithi130.stdout: Package Arch Version Repository Size 2023-07-14T19:03:54.153 INFO:teuthology.orchestra.run.smithi130.stdout:============================================================================================================ 2023-07-14T19:03:54.154 INFO:teuthology.orchestra.run.smithi130.stdout:Installing group/module packages: 2023-07-14T19:03:54.154 INFO:teuthology.orchestra.run.smithi130.stdout: buildah x86_64 1:1.26.2-1.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 8.0 M 2023-07-14T19:03:54.154 INFO:teuthology.orchestra.run.smithi130.stdout: cockpit-podman noarch 49.1-1.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 524 k 2023-07-14T19:03:54.154 INFO:teuthology.orchestra.run.smithi130.stdout: crun x86_64 1.5-1.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 213 k 2023-07-14T19:03:54.154 INFO:teuthology.orchestra.run.smithi130.stdout: python3-podman noarch 4.0.1-1.module+el8.6.0+16837+85b82211 rhel-8-for-x86_64-appstream-rpms 149 k 2023-07-14T19:03:54.154 INFO:teuthology.orchestra.run.smithi130.stdout: skopeo x86_64 2:1.9.1-1.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 7.0 M 2023-07-14T19:03:54.154 INFO:teuthology.orchestra.run.smithi130.stdout: toolbox x86_64 0.0.99.3-0.6.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 2.2 M 2023-07-14T19:03:54.155 INFO:teuthology.orchestra.run.smithi130.stdout: udica noarch 0.2.6-4.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 49 k 2023-07-14T19:03:54.155 INFO:teuthology.orchestra.run.smithi130.stdout:Installing dependencies: 2023-07-14T19:03:54.155 INFO:teuthology.orchestra.run.smithi130.stdout: python3-pytoml noarch 0.1.14-5.git7dea353.el8 rhel-8-for-x86_64-appstream-rpms 25 k 2023-07-14T19:03:54.155 INFO:teuthology.orchestra.run.smithi130.stdout: python3-pyxdg noarch 0.25-16.el8 rhel-8-for-x86_64-appstream-rpms 94 k 2023-07-14T19:03:54.155 INFO:teuthology.orchestra.run.smithi130.stdout: yajl x86_64 2.1.0-10.el8 rhel-8-for-x86_64-appstream-rpms 41 k 2023-07-14T19:03:54.155 INFO:teuthology.orchestra.run.smithi130.stdout:Installing module profiles: 2023-07-14T19:03:54.155 INFO:teuthology.orchestra.run.smithi130.stdout: container-tools/common 2023-07-14T19:03:54.155 INFO:teuthology.orchestra.run.smithi130.stdout: 2023-07-14T19:03:54.156 INFO:teuthology.orchestra.run.smithi130.stdout:Enabling module streams: 2023-07-14T19:03:54.156 INFO:teuthology.orchestra.run.smithi130.stdout: container-tools rhel8 2023-07-14T19:03:54.156 INFO:teuthology.orchestra.run.smithi130.stdout: 2023-07-14T19:03:54.156 INFO:teuthology.orchestra.run.smithi130.stdout:Transaction Summary 2023-07-14T19:03:54.156 INFO:teuthology.orchestra.run.smithi130.stdout:============================================================================================================ 2023-07-14T19:03:54.156 INFO:teuthology.orchestra.run.smithi130.stdout:Install 10 Packages 2023-07-14T19:03:54.156 INFO:teuthology.orchestra.run.smithi130.stdout: 2023-07-14T19:03:54.157 INFO:teuthology.orchestra.run.smithi130.stdout:Total download size: 18 M 2023-07-14T19:03:54.157 INFO:teuthology.orchestra.run.smithi130.stdout:Installed size: 65 M 2023-07-14T19:03:54.157 INFO:teuthology.orchestra.run.smithi130.stdout:Downloading Packages: 2023-07-14T19:03:54.198 INFO:teuthology.orchestra.run.smithi191.stdout:(1/10): yajl-2.1.0-10.el8.x86_64.rpm 303 kB/s | 41 kB 00:00 2023-07-14T19:03:54.223 INFO:teuthology.orchestra.run.smithi191.stdout:(2/10): python3-pyxdg-0.25-16.el8.noarch.rpm 586 kB/s | 94 kB 00:00 2023-07-14T19:03:54.249 INFO:teuthology.orchestra.run.smithi191.stdout:(3/10): python3-pytoml-0.1.14-5.git7dea353.el8. 134 kB/s | 25 kB 00:00 2023-07-14T19:03:54.253 INFO:teuthology.orchestra.run.smithi040.stdout:============================================================================================================ 2023-07-14T19:03:54.254 INFO:teuthology.orchestra.run.smithi040.stdout: Package Arch Version Repository Size 2023-07-14T19:03:54.254 INFO:teuthology.orchestra.run.smithi040.stdout:============================================================================================================ 2023-07-14T19:03:54.254 INFO:teuthology.orchestra.run.smithi040.stdout:Installing group/module packages: 2023-07-14T19:03:54.254 INFO:teuthology.orchestra.run.smithi040.stdout: buildah x86_64 1:1.26.2-1.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 8.0 M 2023-07-14T19:03:54.254 INFO:teuthology.orchestra.run.smithi040.stdout: cockpit-podman noarch 49.1-1.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 524 k 2023-07-14T19:03:54.255 INFO:teuthology.orchestra.run.smithi040.stdout: crun x86_64 1.5-1.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 213 k 2023-07-14T19:03:54.255 INFO:teuthology.orchestra.run.smithi040.stdout: python3-podman noarch 4.0.1-1.module+el8.6.0+16837+85b82211 rhel-8-for-x86_64-appstream-rpms 149 k 2023-07-14T19:03:54.255 INFO:teuthology.orchestra.run.smithi040.stdout: skopeo x86_64 2:1.9.1-1.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 7.0 M 2023-07-14T19:03:54.255 INFO:teuthology.orchestra.run.smithi040.stdout: toolbox x86_64 0.0.99.3-0.6.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 2.2 M 2023-07-14T19:03:54.255 INFO:teuthology.orchestra.run.smithi040.stdout: udica noarch 0.2.6-4.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 49 k 2023-07-14T19:03:54.255 INFO:teuthology.orchestra.run.smithi040.stdout:Installing dependencies: 2023-07-14T19:03:54.255 INFO:teuthology.orchestra.run.smithi040.stdout: python3-pytoml noarch 0.1.14-5.git7dea353.el8 rhel-8-for-x86_64-appstream-rpms 25 k 2023-07-14T19:03:54.256 INFO:teuthology.orchestra.run.smithi040.stdout: python3-pyxdg noarch 0.25-16.el8 rhel-8-for-x86_64-appstream-rpms 94 k 2023-07-14T19:03:54.256 INFO:teuthology.orchestra.run.smithi040.stdout: yajl x86_64 2.1.0-10.el8 rhel-8-for-x86_64-appstream-rpms 41 k 2023-07-14T19:03:54.256 INFO:teuthology.orchestra.run.smithi040.stdout:Installing module profiles: 2023-07-14T19:03:54.256 INFO:teuthology.orchestra.run.smithi040.stdout: container-tools/common 2023-07-14T19:03:54.256 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-07-14T19:03:54.257 INFO:teuthology.orchestra.run.smithi040.stdout:Enabling module streams: 2023-07-14T19:03:54.257 INFO:teuthology.orchestra.run.smithi040.stdout: container-tools rhel8 2023-07-14T19:03:54.257 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-07-14T19:03:54.257 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction Summary 2023-07-14T19:03:54.258 INFO:teuthology.orchestra.run.smithi040.stdout:============================================================================================================ 2023-07-14T19:03:54.258 INFO:teuthology.orchestra.run.smithi040.stdout:Install 10 Packages 2023-07-14T19:03:54.258 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-07-14T19:03:54.259 INFO:teuthology.orchestra.run.smithi040.stdout:Total download size: 18 M 2023-07-14T19:03:54.259 INFO:teuthology.orchestra.run.smithi040.stdout:Installed size: 65 M 2023-07-14T19:03:54.260 INFO:teuthology.orchestra.run.smithi040.stdout:Downloading Packages: 2023-07-14T19:03:54.273 INFO:teuthology.orchestra.run.smithi130.stdout:(1/10): python3-pyxdg-0.25-16.el8.noarch.rpm 802 kB/s | 94 kB 00:00 2023-07-14T19:03:54.282 INFO:teuthology.orchestra.run.smithi191.stdout:(4/10): udica-0.2.6-4.module+el8.6.0+16771+28df 588 kB/s | 49 kB 00:00 2023-07-14T19:03:54.298 INFO:teuthology.orchestra.run.smithi130.stdout:(2/10): python3-pytoml-0.1.14-5.git7dea353.el8. 175 kB/s | 25 kB 00:00 2023-07-14T19:03:54.323 INFO:teuthology.orchestra.run.smithi130.stdout:(3/10): yajl-2.1.0-10.el8.x86_64.rpm 243 kB/s | 41 kB 00:00 2023-07-14T19:03:54.390 INFO:teuthology.orchestra.run.smithi130.stdout:(4/10): udica-0.2.6-4.module+el8.6.0+16771+28df 422 kB/s | 49 kB 00:00 2023-07-14T19:03:54.391 INFO:teuthology.orchestra.run.smithi191.stdout:(5/10): cockpit-podman-49.1-1.module+el8.6.0+16 3.1 MB/s | 524 kB 00:00 2023-07-14T19:03:54.412 INFO:teuthology.orchestra.run.smithi040.stdout:(1/10): python3-pyxdg-0.25-16.el8.noarch.rpm 604 kB/s | 94 kB 00:00 2023-07-14T19:03:54.441 INFO:teuthology.orchestra.run.smithi191.stdout:(6/10): python3-podman-4.0.1-1.module+el8.6.0+1 942 kB/s | 149 kB 00:00 2023-07-14T19:03:54.445 INFO:teuthology.orchestra.run.smithi040.stdout:(2/10): yajl-2.1.0-10.el8.x86_64.rpm 215 kB/s | 41 kB 00:00 2023-07-14T19:03:54.449 INFO:teuthology.orchestra.run.smithi130.stdout:(5/10): cockpit-podman-49.1-1.module+el8.6.0+16 3.4 MB/s | 524 kB 00:00 2023-07-14T19:03:54.471 INFO:teuthology.orchestra.run.smithi040.stdout:(3/10): python3-pytoml-0.1.14-5.git7dea353.el8. 116 kB/s | 25 kB 00:00 2023-07-14T19:03:54.474 INFO:teuthology.orchestra.run.smithi130.stdout:(6/10): python3-podman-4.0.1-1.module+el8.6.0+1 1.7 MB/s | 149 kB 00:00 2023-07-14T19:03:54.583 INFO:teuthology.orchestra.run.smithi191.stdout:(7/10): buildah-1.26.2-1.module+el8.6.0+16771+2 24 MB/s | 8.0 MB 00:00 2023-07-14T19:03:54.600 INFO:teuthology.orchestra.run.smithi191.stdout:(8/10): crun-1.5-1.module+el8.6.0+16771+28dfca7 1.0 MB/s | 213 kB 00:00 2023-07-14T19:03:54.630 INFO:teuthology.orchestra.run.smithi040.stdout:(4/10): udica-0.2.6-4.module+el8.6.0+16771+28df 227 kB/s | 49 kB 00:00 2023-07-14T19:03:54.691 INFO:teuthology.orchestra.run.smithi130.stdout:(7/10): buildah-1.26.2-1.module+el8.6.0+16771+2 22 MB/s | 8.0 MB 00:00 2023-07-14T19:03:54.713 INFO:teuthology.orchestra.run.smithi040.stdout:(5/10): cockpit-podman-49.1-1.module+el8.6.0+16 2.1 MB/s | 524 kB 00:00 2023-07-14T19:03:54.717 INFO:teuthology.orchestra.run.smithi191.stdout:(9/10): toolbox-0.0.99.3-0.6.module+el8.6.0+167 16 MB/s | 2.2 MB 00:00 2023-07-14T19:03:54.766 INFO:teuthology.orchestra.run.smithi130.stdout:(8/10): crun-1.5-1.module+el8.6.0+16771+28dfca7 671 kB/s | 213 kB 00:00 2023-07-14T19:03:54.788 INFO:teuthology.orchestra.run.smithi040.stdout:(6/10): python3-podman-4.0.1-1.module+el8.6.0+1 943 kB/s | 149 kB 00:00 2023-07-14T19:03:54.847 INFO:teuthology.orchestra.run.smithi191.stdout:(10/10): skopeo-1.9.1-1.module+el8.6.0+16771+28 17 MB/s | 7.0 MB 00:00 2023-07-14T19:03:54.848 INFO:teuthology.orchestra.run.smithi191.stdout:-------------------------------------------------------------------------------- 2023-07-14T19:03:54.848 INFO:teuthology.orchestra.run.smithi191.stdout:Total 23 MB/s | 18 MB 00:00 2023-07-14T19:03:54.875 INFO:teuthology.orchestra.run.smithi130.stdout:(9/10): skopeo-1.9.1-1.module+el8.6.0+16771+28d 18 MB/s | 7.0 MB 00:00 2023-07-14T19:03:54.897 INFO:teuthology.orchestra.run.smithi040.stdout:(7/10): buildah-1.26.2-1.module+el8.6.0+16771+2 19 MB/s | 8.0 MB 00:00 2023-07-14T19:03:54.922 INFO:teuthology.orchestra.run.smithi040.stdout:(8/10): crun-1.5-1.module+el8.6.0+16771+28dfca7 1.0 MB/s | 213 kB 00:00 2023-07-14T19:03:54.933 INFO:teuthology.orchestra.run.smithi130.stdout:(10/10): toolbox-0.0.99.3-0.6.module+el8.6.0+16 9.1 MB/s | 2.2 MB 00:00 2023-07-14T19:03:54.934 INFO:teuthology.orchestra.run.smithi130.stdout:-------------------------------------------------------------------------------- 2023-07-14T19:03:54.934 INFO:teuthology.orchestra.run.smithi130.stdout:Total 23 MB/s | 18 MB 00:00 2023-07-14T19:03:55.049 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction check 2023-07-14T19:03:55.064 INFO:teuthology.orchestra.run.smithi040.stdout:(9/10): skopeo-1.9.1-1.module+el8.6.0+16771+28d 25 MB/s | 7.0 MB 00:00 2023-07-14T19:03:55.067 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction check succeeded. 2023-07-14T19:03:55.067 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction test 2023-07-14T19:03:55.131 INFO:teuthology.orchestra.run.smithi040.stdout:(10/10): toolbox-0.0.99.3-0.6.module+el8.6.0+16 9.4 MB/s | 2.2 MB 00:00 2023-07-14T19:03:55.131 INFO:teuthology.orchestra.run.smithi040.stdout:-------------------------------------------------------------------------------- 2023-07-14T19:03:55.132 INFO:teuthology.orchestra.run.smithi040.stdout:Total 21 MB/s | 18 MB 00:00 2023-07-14T19:03:55.133 INFO:teuthology.orchestra.run.smithi130.stdout:Running transaction check 2023-07-14T19:03:55.152 INFO:teuthology.orchestra.run.smithi130.stdout:Transaction check succeeded. 2023-07-14T19:03:55.152 INFO:teuthology.orchestra.run.smithi130.stdout:Running transaction test 2023-07-14T19:03:55.248 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction test succeeded. 2023-07-14T19:03:55.252 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction 2023-07-14T19:03:55.333 INFO:teuthology.orchestra.run.smithi130.stdout:Transaction test succeeded. 2023-07-14T19:03:55.336 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction check 2023-07-14T19:03:55.337 INFO:teuthology.orchestra.run.smithi130.stdout:Running transaction 2023-07-14T19:03:55.357 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction check succeeded. 2023-07-14T19:03:55.357 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction test 2023-07-14T19:03:55.551 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction test succeeded. 2023-07-14T19:03:55.555 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction 2023-07-14T19:03:55.740 INFO:teuthology.orchestra.run.smithi191.stdout: Preparing : 1/1 2023-07-14T19:03:55.859 INFO:teuthology.orchestra.run.smithi130.stdout: Preparing : 1/1 2023-07-14T19:03:55.938 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 1/10 2023-07-14T19:03:56.048 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 1/10 2023-07-14T19:03:56.065 INFO:teuthology.orchestra.run.smithi040.stdout: Preparing : 1/1 2023-07-14T19:03:56.102 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : yajl-2.1.0-10.el8.x86_64 2/10 2023-07-14T19:03:56.214 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : yajl-2.1.0-10.el8.x86_64 2/10 2023-07-14T19:03:56.261 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 1/10 2023-07-14T19:03:56.297 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-pyxdg-0.25-16.el8.noarch 3/10 2023-07-14T19:03:56.458 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-pyxdg-0.25-16.el8.noarch 3/10 2023-07-14T19:03:56.477 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : yajl-2.1.0-10.el8.x86_64 2/10 2023-07-14T19:03:56.507 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-podman-4.0.1-1.module+el8.6.0+16837+85b822 4/10 2023-07-14T19:03:56.661 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-podman-4.0.1-1.module+el8.6.0+16837+85b822 4/10 2023-07-14T19:03:56.667 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-pyxdg-0.25-16.el8.noarch 3/10 2023-07-14T19:03:56.847 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : crun-1.5-1.module+el8.6.0+16771+28dfca77.x86_64 5/10 2023-07-14T19:03:56.889 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-podman-4.0.1-1.module+el8.6.0+16837+85b822 4/10 2023-07-14T19:03:57.035 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : crun-1.5-1.module+el8.6.0+16771+28dfca77.x86_64 5/10 2023-07-14T19:03:57.276 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : crun-1.5-1.module+el8.6.0+16771+28dfca77.x86_64 5/10 2023-07-14T19:03:57.642 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : toolbox-0.0.99.3-0.6.module+el8.6.0+16771+28dfca77 6/10 2023-07-14T19:03:57.821 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : toolbox-0.0.99.3-0.6.module+el8.6.0+16771+28dfca77 6/10 2023-07-14T19:03:58.115 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : toolbox-0.0.99.3-0.6.module+el8.6.0+16771+28dfca77 6/10 2023-07-14T19:03:58.525 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : skopeo-2:1.9.1-1.module+el8.6.0+16771+28dfca77.x86 7/10 2023-07-14T19:03:58.713 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : skopeo-2:1.9.1-1.module+el8.6.0+16771+28dfca77.x86 7/10 2023-07-14T19:03:58.780 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : buildah-1:1.26.2-1.module+el8.6.0+16771+28dfca77.x 8/10 2023-07-14T19:03:58.946 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : buildah-1:1.26.2-1.module+el8.6.0+16771+28dfca77.x 8/10 2023-07-14T19:03:58.967 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : cockpit-podman-49.1-1.module+el8.6.0+16771+28dfca7 9/10 2023-07-14T19:03:59.054 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : skopeo-2:1.9.1-1.module+el8.6.0+16771+28dfca77.x86 7/10 2023-07-14T19:03:59.105 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : cockpit-podman-49.1-1.module+el8.6.0+16771+28dfca7 9/10 2023-07-14T19:03:59.145 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : udica-0.2.6-4.module+el8.6.0+16771+28dfca77.noarch 10/10 2023-07-14T19:03:59.285 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : buildah-1:1.26.2-1.module+el8.6.0+16771+28dfca77.x 8/10 2023-07-14T19:03:59.299 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : udica-0.2.6-4.module+el8.6.0+16771+28dfca77.noarch 10/10 2023-07-14T19:03:59.413 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: udica-0.2.6-4.module+el8.6.0+16771+28dfca77.noarch 10/10 2023-07-14T19:03:59.414 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-pyxdg-0.25-16.el8.noarch 1/10 2023-07-14T19:03:59.414 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : yajl-2.1.0-10.el8.x86_64 2/10 2023-07-14T19:03:59.414 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 3/10 2023-07-14T19:03:59.415 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : udica-0.2.6-4.module+el8.6.0+16771+28dfca77.noarch 4/10 2023-07-14T19:03:59.415 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : cockpit-podman-49.1-1.module+el8.6.0+16771+28dfca7 5/10 2023-07-14T19:03:59.415 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : buildah-1:1.26.2-1.module+el8.6.0+16771+28dfca77.x 6/10 2023-07-14T19:03:59.416 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-podman-4.0.1-1.module+el8.6.0+16837+85b822 7/10 2023-07-14T19:03:59.416 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : crun-1.5-1.module+el8.6.0+16771+28dfca77.x86_64 8/10 2023-07-14T19:03:59.416 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : skopeo-2:1.9.1-1.module+el8.6.0+16771+28dfca77.x86 9/10 2023-07-14T19:03:59.441 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : cockpit-podman-49.1-1.module+el8.6.0+16771+28dfca7 9/10 2023-07-14T19:03:59.561 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: udica-0.2.6-4.module+el8.6.0+16771+28dfca77.noarch 10/10 2023-07-14T19:03:59.562 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-pyxdg-0.25-16.el8.noarch 1/10 2023-07-14T19:03:59.562 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : yajl-2.1.0-10.el8.x86_64 2/10 2023-07-14T19:03:59.562 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 3/10 2023-07-14T19:03:59.562 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : udica-0.2.6-4.module+el8.6.0+16771+28dfca77.noarch 4/10 2023-07-14T19:03:59.562 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : cockpit-podman-49.1-1.module+el8.6.0+16771+28dfca7 5/10 2023-07-14T19:03:59.562 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : buildah-1:1.26.2-1.module+el8.6.0+16771+28dfca77.x 6/10 2023-07-14T19:03:59.562 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-podman-4.0.1-1.module+el8.6.0+16837+85b822 7/10 2023-07-14T19:03:59.563 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : crun-1.5-1.module+el8.6.0+16771+28dfca77.x86_64 8/10 2023-07-14T19:03:59.563 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : skopeo-2:1.9.1-1.module+el8.6.0+16771+28dfca77.x86 9/10 2023-07-14T19:03:59.650 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : udica-0.2.6-4.module+el8.6.0+16771+28dfca77.noarch 10/10 2023-07-14T19:03:59.932 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: udica-0.2.6-4.module+el8.6.0+16771+28dfca77.noarch 10/10 2023-07-14T19:03:59.932 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-pyxdg-0.25-16.el8.noarch 1/10 2023-07-14T19:03:59.933 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : yajl-2.1.0-10.el8.x86_64 2/10 2023-07-14T19:03:59.933 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 3/10 2023-07-14T19:03:59.934 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : udica-0.2.6-4.module+el8.6.0+16771+28dfca77.noarch 4/10 2023-07-14T19:03:59.934 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : cockpit-podman-49.1-1.module+el8.6.0+16771+28dfca7 5/10 2023-07-14T19:03:59.934 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : buildah-1:1.26.2-1.module+el8.6.0+16771+28dfca77.x 6/10 2023-07-14T19:03:59.934 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-podman-4.0.1-1.module+el8.6.0+16837+85b822 7/10 2023-07-14T19:03:59.934 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : crun-1.5-1.module+el8.6.0+16771+28dfca77.x86_64 8/10 2023-07-14T19:03:59.934 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : skopeo-2:1.9.1-1.module+el8.6.0+16771+28dfca77.x86 9/10 2023-07-14T19:04:00.600 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : toolbox-0.0.99.3-0.6.module+el8.6.0+16771+28dfca77 10/10 2023-07-14T19:04:00.600 INFO:teuthology.orchestra.run.smithi191.stdout:Last metadata expiration check: 0:02:18 ago on Fri 14 Jul 2023 07:01:42 PM UTC. 2023-07-14T19:04:00.715 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : toolbox-0.0.99.3-0.6.module+el8.6.0+16771+28dfca77 10/10 2023-07-14T19:04:00.716 INFO:teuthology.orchestra.run.smithi130.stdout:Last metadata expiration check: 0:02:28 ago on Fri 14 Jul 2023 07:01:32 PM UTC. 2023-07-14T19:04:01.074 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : toolbox-0.0.99.3-0.6.module+el8.6.0+16771+28dfca77 10/10 2023-07-14T19:04:01.075 INFO:teuthology.orchestra.run.smithi040.stdout:Last metadata expiration check: 0:02:13 ago on Fri 14 Jul 2023 07:01:47 PM UTC. 2023-07-14T19:04:08.222 INFO:teuthology.orchestra.run.smithi191.stdout:Installed products updated. 2023-07-14T19:04:08.414 INFO:teuthology.orchestra.run.smithi191.stdout: 2023-07-14T19:04:08.415 INFO:teuthology.orchestra.run.smithi191.stdout:Installed: 2023-07-14T19:04:08.415 INFO:teuthology.orchestra.run.smithi191.stdout: buildah-1:1.26.2-1.module+el8.6.0+16771+28dfca77.x86_64 2023-07-14T19:04:08.415 INFO:teuthology.orchestra.run.smithi191.stdout: cockpit-podman-49.1-1.module+el8.6.0+16771+28dfca77.noarch 2023-07-14T19:04:08.415 INFO:teuthology.orchestra.run.smithi191.stdout: crun-1.5-1.module+el8.6.0+16771+28dfca77.x86_64 2023-07-14T19:04:08.415 INFO:teuthology.orchestra.run.smithi191.stdout: python3-podman-4.0.1-1.module+el8.6.0+16837+85b82211.noarch 2023-07-14T19:04:08.415 INFO:teuthology.orchestra.run.smithi191.stdout: python3-pytoml-0.1.14-5.git7dea353.el8.noarch 2023-07-14T19:04:08.415 INFO:teuthology.orchestra.run.smithi191.stdout: python3-pyxdg-0.25-16.el8.noarch 2023-07-14T19:04:08.416 INFO:teuthology.orchestra.run.smithi191.stdout: skopeo-2:1.9.1-1.module+el8.6.0+16771+28dfca77.x86_64 2023-07-14T19:04:08.416 INFO:teuthology.orchestra.run.smithi191.stdout: toolbox-0.0.99.3-0.6.module+el8.6.0+16771+28dfca77.x86_64 2023-07-14T19:04:08.416 INFO:teuthology.orchestra.run.smithi191.stdout: udica-0.2.6-4.module+el8.6.0+16771+28dfca77.noarch 2023-07-14T19:04:08.416 INFO:teuthology.orchestra.run.smithi191.stdout: yajl-2.1.0-10.el8.x86_64 2023-07-14T19:04:08.417 INFO:teuthology.orchestra.run.smithi191.stdout: 2023-07-14T19:04:08.417 INFO:teuthology.orchestra.run.smithi191.stdout:Complete! 2023-07-14T19:04:08.555 DEBUG:teuthology.parallel:result is None 2023-07-14T19:04:08.824 INFO:teuthology.orchestra.run.smithi040.stdout:Installed products updated. 2023-07-14T19:04:08.858 INFO:teuthology.orchestra.run.smithi130.stdout:Installed products updated. 2023-07-14T19:04:09.042 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-07-14T19:04:09.042 INFO:teuthology.orchestra.run.smithi040.stdout:Installed: 2023-07-14T19:04:09.042 INFO:teuthology.orchestra.run.smithi040.stdout: buildah-1:1.26.2-1.module+el8.6.0+16771+28dfca77.x86_64 2023-07-14T19:04:09.043 INFO:teuthology.orchestra.run.smithi040.stdout: cockpit-podman-49.1-1.module+el8.6.0+16771+28dfca77.noarch 2023-07-14T19:04:09.043 INFO:teuthology.orchestra.run.smithi040.stdout: crun-1.5-1.module+el8.6.0+16771+28dfca77.x86_64 2023-07-14T19:04:09.043 INFO:teuthology.orchestra.run.smithi040.stdout: python3-podman-4.0.1-1.module+el8.6.0+16837+85b82211.noarch 2023-07-14T19:04:09.043 INFO:teuthology.orchestra.run.smithi040.stdout: python3-pytoml-0.1.14-5.git7dea353.el8.noarch 2023-07-14T19:04:09.043 INFO:teuthology.orchestra.run.smithi040.stdout: python3-pyxdg-0.25-16.el8.noarch 2023-07-14T19:04:09.043 INFO:teuthology.orchestra.run.smithi040.stdout: skopeo-2:1.9.1-1.module+el8.6.0+16771+28dfca77.x86_64 2023-07-14T19:04:09.043 INFO:teuthology.orchestra.run.smithi040.stdout: toolbox-0.0.99.3-0.6.module+el8.6.0+16771+28dfca77.x86_64 2023-07-14T19:04:09.044 INFO:teuthology.orchestra.run.smithi040.stdout: udica-0.2.6-4.module+el8.6.0+16771+28dfca77.noarch 2023-07-14T19:04:09.044 INFO:teuthology.orchestra.run.smithi040.stdout: yajl-2.1.0-10.el8.x86_64 2023-07-14T19:04:09.044 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-07-14T19:04:09.044 INFO:teuthology.orchestra.run.smithi040.stdout:Complete! 2023-07-14T19:04:09.060 INFO:teuthology.orchestra.run.smithi130.stdout: 2023-07-14T19:04:09.060 INFO:teuthology.orchestra.run.smithi130.stdout:Installed: 2023-07-14T19:04:09.060 INFO:teuthology.orchestra.run.smithi130.stdout: buildah-1:1.26.2-1.module+el8.6.0+16771+28dfca77.x86_64 2023-07-14T19:04:09.060 INFO:teuthology.orchestra.run.smithi130.stdout: cockpit-podman-49.1-1.module+el8.6.0+16771+28dfca77.noarch 2023-07-14T19:04:09.060 INFO:teuthology.orchestra.run.smithi130.stdout: crun-1.5-1.module+el8.6.0+16771+28dfca77.x86_64 2023-07-14T19:04:09.061 INFO:teuthology.orchestra.run.smithi130.stdout: python3-podman-4.0.1-1.module+el8.6.0+16837+85b82211.noarch 2023-07-14T19:04:09.061 INFO:teuthology.orchestra.run.smithi130.stdout: python3-pytoml-0.1.14-5.git7dea353.el8.noarch 2023-07-14T19:04:09.061 INFO:teuthology.orchestra.run.smithi130.stdout: python3-pyxdg-0.25-16.el8.noarch 2023-07-14T19:04:09.061 INFO:teuthology.orchestra.run.smithi130.stdout: skopeo-2:1.9.1-1.module+el8.6.0+16771+28dfca77.x86_64 2023-07-14T19:04:09.061 INFO:teuthology.orchestra.run.smithi130.stdout: toolbox-0.0.99.3-0.6.module+el8.6.0+16771+28dfca77.x86_64 2023-07-14T19:04:09.061 INFO:teuthology.orchestra.run.smithi130.stdout: udica-0.2.6-4.module+el8.6.0+16771+28dfca77.noarch 2023-07-14T19:04:09.061 INFO:teuthology.orchestra.run.smithi130.stdout: yajl-2.1.0-10.el8.x86_64 2023-07-14T19:04:09.061 INFO:teuthology.orchestra.run.smithi130.stdout: 2023-07-14T19:04:09.062 INFO:teuthology.orchestra.run.smithi130.stdout:Complete! 2023-07-14T19:04:09.206 DEBUG:teuthology.parallel:result is None 2023-07-14T19:04:09.286 DEBUG:teuthology.parallel:result is None 2023-07-14T19:04:09.286 INFO:teuthology.run_tasks:Running task install... 2023-07-14T19:04:09.295 DEBUG:teuthology.task.install:project ceph 2023-07-14T19:04:09.295 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '81c9387d2e7576da065747ec9c16d1c7d69016ec'}} 2023-07-14T19:04:09.295 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': '81c9387d2e7576da065747ec9c16d1c7d69016ec'} 2023-07-14T19:04:09.296 INFO:teuthology.task.install:Using flavor: default 2023-07-14T19:04:09.301 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2023-07-14T19:04:09.302 INFO:teuthology.task.install:extra packages: {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']} 2023-07-14T19:04:09.302 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '81c9387d2e7576da065747ec9c16d1c7d69016ec', 'tag': None, 'wait_for_package': False} 2023-07-14T19:04:09.303 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=81c9387d2e7576da065747ec9c16d1c7d69016ec 2023-07-14T19:04:09.305 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '81c9387d2e7576da065747ec9c16d1c7d69016ec', 'tag': None, 'wait_for_package': False} 2023-07-14T19:04:09.305 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=81c9387d2e7576da065747ec9c16d1c7d69016ec 2023-07-14T19:04:09.306 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '81c9387d2e7576da065747ec9c16d1c7d69016ec', 'tag': None, 'wait_for_package': False} 2023-07-14T19:04:09.307 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=81c9387d2e7576da065747ec9c16d1c7d69016ec 2023-07-14T19:04:09.799 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20230714.134335/81c9387d2e7576da065747ec9c16d1c7d69016ec/centos/8/flavors/default/ 2023-07-14T19:04:09.800 INFO:teuthology.task.install.rpm:Package version is 18.0.0-4940.g81c9387d 2023-07-14T19:04:09.806 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20230714.134335/81c9387d2e7576da065747ec9c16d1c7d69016ec/centos/8/flavors/default/ 2023-07-14T19:04:09.807 INFO:teuthology.task.install.rpm:Package version is 18.0.0-4940.g81c9387d 2023-07-14T19:04:09.810 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20230714.134335/81c9387d2e7576da065747ec9c16d1c7d69016ec/centos/8/flavors/default/ 2023-07-14T19:04:09.811 INFO:teuthology.task.install.rpm:Package version is 18.0.0-4940.g81c9387d 2023-07-14T19:04:10.154 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20230714.134335/81c9387d2e7576da065747ec9c16d1c7d69016ec/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20230714.134335/81c9387d2e7576da065747ec9c16d1c7d69016ec/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20230714.134335/81c9387d2e7576da065747ec9c16d1c7d69016ec/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2023-07-14T19:04:10.155 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2023-07-14T19:04:10.155 DEBUG:teuthology.orchestra.run.smithi040:> sudo dd of=/etc/yum.repos.d/ceph.repo 2023-07-14T19:04:10.157 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20230714.134335/81c9387d2e7576da065747ec9c16d1c7d69016ec/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20230714.134335/81c9387d2e7576da065747ec9c16d1c7d69016ec/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20230714.134335/81c9387d2e7576da065747ec9c16d1c7d69016ec/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2023-07-14T19:04:10.158 DEBUG:teuthology.orchestra.run.smithi130:> set -ex 2023-07-14T19:04:10.158 DEBUG:teuthology.orchestra.run.smithi130:> sudo dd of=/etc/yum.repos.d/ceph.repo 2023-07-14T19:04:10.190 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2023-07-14T19:04:10.190 DEBUG:teuthology.orchestra.run.smithi130:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/81c9387d2e7576da065747ec9c16d1c7d69016ec/;g' /etc/yum.repos.d/ceph.repo ; fi 2023-07-14T19:04:10.192 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2023-07-14T19:04:10.192 DEBUG:teuthology.orchestra.run.smithi040:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/81c9387d2e7576da065747ec9c16d1c7d69016ec/;g' /etc/yum.repos.d/ceph.repo ; fi 2023-07-14T19:04:10.264 DEBUG:teuthology.orchestra.run.smithi130:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2023-07-14T19:04:10.266 DEBUG:teuthology.orchestra.run.smithi040:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2023-07-14T19:04:10.345 DEBUG:teuthology.orchestra.run.smithi130:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2023-07-14T19:04:10.350 DEBUG:teuthology.orchestra.run.smithi040:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2023-07-14T19:04:10.411 INFO:teuthology.orchestra.run.smithi130.stdout:check_obsoletes = 1 2023-07-14T19:04:10.412 DEBUG:teuthology.orchestra.run.smithi130:> sudo yum clean all 2023-07-14T19:04:10.418 INFO:teuthology.orchestra.run.smithi040.stdout:check_obsoletes = 1 2023-07-14T19:04:10.420 DEBUG:teuthology.orchestra.run.smithi040:> sudo yum clean all 2023-07-14T19:04:10.655 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20230714.134335/81c9387d2e7576da065747ec9c16d1c7d69016ec/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20230714.134335/81c9387d2e7576da065747ec9c16d1c7d69016ec/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20230714.134335/81c9387d2e7576da065747ec9c16d1c7d69016ec/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2023-07-14T19:04:10.656 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2023-07-14T19:04:10.656 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd of=/etc/yum.repos.d/ceph.repo 2023-07-14T19:04:10.689 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2023-07-14T19:04:10.690 DEBUG:teuthology.orchestra.run.smithi191:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/81c9387d2e7576da065747ec9c16d1c7d69016ec/;g' /etc/yum.repos.d/ceph.repo ; fi 2023-07-14T19:04:10.762 DEBUG:teuthology.orchestra.run.smithi191:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2023-07-14T19:04:10.784 INFO:teuthology.orchestra.run.smithi130.stdout:Updating Subscription Management repositories. 2023-07-14T19:04:10.820 INFO:teuthology.orchestra.run.smithi040.stdout:Updating Subscription Management repositories. 2023-07-14T19:04:10.841 DEBUG:teuthology.orchestra.run.smithi191:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2023-07-14T19:04:10.908 INFO:teuthology.orchestra.run.smithi191.stdout:check_obsoletes = 1 2023-07-14T19:04:10.909 DEBUG:teuthology.orchestra.run.smithi191:> sudo yum clean all 2023-07-14T19:04:11.280 INFO:teuthology.orchestra.run.smithi191.stdout:Updating Subscription Management repositories. 2023-07-14T19:04:11.824 INFO:teuthology.orchestra.run.smithi130.stdout:44 files removed 2023-07-14T19:04:11.828 INFO:teuthology.orchestra.run.smithi040.stdout:44 files removed 2023-07-14T19:04:11.876 DEBUG:teuthology.orchestra.run.smithi130:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2023-07-14T19:04:11.884 DEBUG:teuthology.orchestra.run.smithi040:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2023-07-14T19:04:12.208 INFO:teuthology.orchestra.run.smithi130.stdout:Updating Subscription Management repositories. 2023-07-14T19:04:12.237 INFO:teuthology.orchestra.run.smithi040.stdout:Updating Subscription Management repositories. 2023-07-14T19:04:12.342 INFO:teuthology.orchestra.run.smithi191.stdout:44 files removed 2023-07-14T19:04:12.396 DEBUG:teuthology.orchestra.run.smithi191:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2023-07-14T19:04:12.729 INFO:teuthology.orchestra.run.smithi191.stdout:Updating Subscription Management repositories. 2023-07-14T19:04:14.184 INFO:teuthology.orchestra.run.smithi130.stdout:ceph packages for x86_64 119 kB/s | 81 kB 00:00 2023-07-14T19:04:14.224 INFO:teuthology.orchestra.run.smithi040.stdout:ceph packages for x86_64 119 kB/s | 81 kB 00:00 2023-07-14T19:04:14.677 INFO:teuthology.orchestra.run.smithi191.stdout:ceph packages for x86_64 127 kB/s | 81 kB 00:00 2023-07-14T19:04:14.743 INFO:teuthology.orchestra.run.smithi130.stdout:ceph noarch packages 18 kB/s | 9.4 kB 00:00 2023-07-14T19:04:14.792 INFO:teuthology.orchestra.run.smithi040.stdout:ceph noarch packages 18 kB/s | 9.4 kB 00:00 2023-07-14T19:04:15.228 INFO:teuthology.orchestra.run.smithi191.stdout:ceph noarch packages 18 kB/s | 9.4 kB 00:00 2023-07-14T19:04:15.269 INFO:teuthology.orchestra.run.smithi130.stdout:ceph source packages 3.0 kB/s | 1.5 kB 00:00 2023-07-14T19:04:15.343 INFO:teuthology.orchestra.run.smithi040.stdout:ceph source packages 2.8 kB/s | 1.5 kB 00:00 2023-07-14T19:04:15.537 INFO:teuthology.orchestra.run.smithi130.stdout:Copr repo for python3-asyncssh owned by ceph 14 kB/s | 3.5 kB 00:00 2023-07-14T19:04:15.593 INFO:teuthology.orchestra.run.smithi040.stdout:Copr repo for python3-asyncssh owned by ceph 15 kB/s | 3.5 kB 00:00 2023-07-14T19:04:15.754 INFO:teuthology.orchestra.run.smithi191.stdout:ceph source packages 3.0 kB/s | 1.5 kB 00:00 2023-07-14T19:04:16.037 INFO:teuthology.orchestra.run.smithi191.stdout:Copr repo for python3-asyncssh owned by ceph 13 kB/s | 3.5 kB 00:00 2023-07-14T19:04:16.554 INFO:teuthology.orchestra.run.smithi130.stdout:Extra Packages for Enterprise Linux 16 MB/s | 16 MB 00:00 2023-07-14T19:04:17.296 INFO:teuthology.orchestra.run.smithi191.stdout:Extra Packages for Enterprise Linux 13 MB/s | 16 MB 00:01 2023-07-14T19:04:19.987 INFO:teuthology.orchestra.run.smithi130.stdout:lab-extras 441 kB/s | 24 kB 00:00 2023-07-14T19:04:20.738 INFO:teuthology.orchestra.run.smithi191.stdout:lab-extras 429 kB/s | 24 kB 00:00 2023-07-14T19:04:21.097 INFO:teuthology.orchestra.run.smithi130.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 46 MB/s | 47 MB 00:01 2023-07-14T19:04:21.998 INFO:teuthology.orchestra.run.smithi191.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 45 MB/s | 53 MB 00:01 2023-07-14T19:04:23.671 INFO:teuthology.orchestra.run.smithi040.stdout:Extra Packages for Enterprise Linux 1.9 MB/s | 16 MB 00:08 2023-07-14T19:04:27.343 INFO:teuthology.orchestra.run.smithi040.stdout:lab-extras 389 kB/s | 24 kB 00:00 2023-07-14T19:04:28.653 INFO:teuthology.orchestra.run.smithi040.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 43 MB/s | 53 MB 00:01 2023-07-14T19:04:29.793 INFO:teuthology.orchestra.run.smithi130.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 36 MB/s | 53 MB 00:01 2023-07-14T19:04:30.741 INFO:teuthology.orchestra.run.smithi191.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 35 MB/s | 47 MB 00:01 2023-07-14T19:04:37.676 INFO:teuthology.orchestra.run.smithi040.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 41 MB/s | 47 MB 00:01 2023-07-14T19:04:40.874 INFO:teuthology.orchestra.run.smithi130.stdout:Package librados2-1:12.2.7-9.el8.x86_64 is already installed. 2023-07-14T19:04:40.875 INFO:teuthology.orchestra.run.smithi130.stdout:Package librbd1-1:12.2.7-9.el8.x86_64 is already installed. 2023-07-14T19:04:40.890 INFO:teuthology.orchestra.run.smithi130.stdout:Package NetworkManager-1:1.36.0-4.el8.x86_64 is already installed. 2023-07-14T19:04:40.891 INFO:teuthology.orchestra.run.smithi130.stdout:Package iproute-5.15.0-4.el8.x86_64 is already installed. 2023-07-14T19:04:40.892 INFO:teuthology.orchestra.run.smithi130.stdout:Package util-linux-2.32.1-35.el8.x86_64 is already installed. 2023-07-14T19:04:40.893 INFO:teuthology.orchestra.run.smithi130.stdout:Package libacl-devel-2.2.53-1.el8.x86_64 is already installed. 2023-07-14T19:04:40.895 INFO:teuthology.orchestra.run.smithi130.stdout:Package libattr-devel-2.4.48-3.el8.x86_64 is already installed. 2023-07-14T19:04:40.899 INFO:teuthology.orchestra.run.smithi130.stdout:Package xfsdump-3.1.8-4.el8.x86_64 is already installed. 2023-07-14T19:04:40.900 INFO:teuthology.orchestra.run.smithi130.stdout:Package xfsprogs-5.0.0-10.el8.x86_64 is already installed. 2023-07-14T19:04:41.033 INFO:teuthology.orchestra.run.smithi130.stdout:Dependencies resolved. 2023-07-14T19:04:41.046 INFO:teuthology.orchestra.run.smithi130.stdout:=========================================================================================================================== 2023-07-14T19:04:41.046 INFO:teuthology.orchestra.run.smithi130.stdout: Package Arch Version Repository Size 2023-07-14T19:04:41.046 INFO:teuthology.orchestra.run.smithi130.stdout:=========================================================================================================================== 2023-07-14T19:04:41.046 INFO:teuthology.orchestra.run.smithi130.stdout:Installing: 2023-07-14T19:04:41.047 INFO:teuthology.orchestra.run.smithi130.stdout: NetworkManager-initscripts-updown noarch 1:1.36.0-9.el8_6 rhel-8-for-x86_64-baseos-rpms 138 k 2023-07-14T19:04:41.047 INFO:teuthology.orchestra.run.smithi130.stdout: bison x86_64 3.0.4-10.el8 rhel-8-for-x86_64-appstream-rpms 688 k 2023-07-14T19:04:41.047 INFO:teuthology.orchestra.run.smithi130.stdout: ceph x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 6.4 k 2023-07-14T19:04:41.047 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-base x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 5.4 M 2023-07-14T19:04:41.047 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-fuse x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 850 k 2023-07-14T19:04:41.047 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-immutable-object-cache x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 167 k 2023-07-14T19:04:41.047 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-mgr x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 1.6 M 2023-07-14T19:04:41.048 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-mgr-cephadm noarch 2:18.0.0-4940.g81c9387d.el8 ceph-noarch 129 k 2023-07-14T19:04:41.048 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-mgr-dashboard noarch 2:18.0.0-4940.g81c9387d.el8 ceph-noarch 1.7 M 2023-07-14T19:04:41.048 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-mgr-diskprediction-local noarch 2:18.0.0-4940.g81c9387d.el8 ceph-noarch 7.4 M 2023-07-14T19:04:41.048 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-mgr-rook noarch 2:18.0.0-4940.g81c9387d.el8 ceph-noarch 47 k 2023-07-14T19:04:41.048 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-radosgw x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 14 M 2023-07-14T19:04:41.048 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-test x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 48 M 2023-07-14T19:04:41.048 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-volume noarch 2:18.0.0-4940.g81c9387d.el8 ceph-noarch 258 k 2023-07-14T19:04:41.049 INFO:teuthology.orchestra.run.smithi130.stdout: cephadm noarch 2:18.0.0-4940.g81c9387d.el8 ceph-noarch 165 k 2023-07-14T19:04:41.049 INFO:teuthology.orchestra.run.smithi130.stdout: cephfs-mirror x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 260 k 2023-07-14T19:04:41.049 INFO:teuthology.orchestra.run.smithi130.stdout: cephfs-top noarch 2:18.0.0-4940.g81c9387d.el8 ceph-noarch 18 k 2023-07-14T19:04:41.049 INFO:teuthology.orchestra.run.smithi130.stdout: elfutils-libelf-devel x86_64 0.186-1.el8 rhel-8-for-x86_64-baseos-rpms 60 k 2023-07-14T19:04:41.049 INFO:teuthology.orchestra.run.smithi130.stdout: flex x86_64 2.6.1-9.el8 rhel-8-for-x86_64-appstream-rpms 320 k 2023-07-14T19:04:41.049 INFO:teuthology.orchestra.run.smithi130.stdout: libaio-devel x86_64 0.3.112-1.el8 rhel-8-for-x86_64-baseos-rpms 19 k 2023-07-14T19:04:41.049 INFO:teuthology.orchestra.run.smithi130.stdout: libcephfs-devel x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 31 k 2023-07-14T19:04:41.050 INFO:teuthology.orchestra.run.smithi130.stdout: libcephfs2 x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 772 k 2023-07-14T19:04:41.050 INFO:teuthology.orchestra.run.smithi130.stdout: librados-devel x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 139 k 2023-07-14T19:04:41.050 INFO:teuthology.orchestra.run.smithi130.stdout: libtool x86_64 2.4.6-25.el8 rhel-8-for-x86_64-appstream-rpms 709 k 2023-07-14T19:04:41.050 INFO:teuthology.orchestra.run.smithi130.stdout: libuuid-devel x86_64 2.32.1-35.el8 rhel-8-for-x86_64-baseos-rpms 98 k 2023-07-14T19:04:41.050 INFO:teuthology.orchestra.run.smithi130.stdout: openssl-devel x86_64 1:1.1.1k-7.el8_6 rhel-8-for-x86_64-baseos-rpms 2.3 M 2023-07-14T19:04:41.050 INFO:teuthology.orchestra.run.smithi130.stdout: postgresql x86_64 10.21-2.module+el8.6.0+15342+53518fac rhel-8-for-x86_64-appstream-rpms 1.5 M 2023-07-14T19:04:41.050 INFO:teuthology.orchestra.run.smithi130.stdout: postgresql-contrib x86_64 10.21-2.module+el8.6.0+15342+53518fac rhel-8-for-x86_64-appstream-rpms 808 k 2023-07-14T19:04:41.050 INFO:teuthology.orchestra.run.smithi130.stdout: postgresql-server x86_64 10.21-2.module+el8.6.0+15342+53518fac rhel-8-for-x86_64-appstream-rpms 5.1 M 2023-07-14T19:04:41.051 INFO:teuthology.orchestra.run.smithi130.stdout: python3-cephfs x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 197 k 2023-07-14T19:04:41.051 INFO:teuthology.orchestra.run.smithi130.stdout: python3-rados x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 371 k 2023-07-14T19:04:41.051 INFO:teuthology.orchestra.run.smithi130.stdout: python3-rbd x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 364 k 2023-07-14T19:04:41.051 INFO:teuthology.orchestra.run.smithi130.stdout: python3-rgw x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 107 k 2023-07-14T19:04:41.051 INFO:teuthology.orchestra.run.smithi130.stdout: rbd-fuse x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 94 k 2023-07-14T19:04:41.051 INFO:teuthology.orchestra.run.smithi130.stdout: rbd-mirror x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 3.5 M 2023-07-14T19:04:41.051 INFO:teuthology.orchestra.run.smithi130.stdout: rbd-nbd x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 179 k 2023-07-14T19:04:41.052 INFO:teuthology.orchestra.run.smithi130.stdout: xfsprogs-devel x86_64 5.0.0-10.el8 rhel-8-for-x86_64-baseos-rpms 72 k 2023-07-14T19:04:41.052 INFO:teuthology.orchestra.run.smithi130.stdout:Upgrading: 2023-07-14T19:04:41.052 INFO:teuthology.orchestra.run.smithi130.stdout: iproute x86_64 5.15.0-4.el8_6.1 rhel-8-for-x86_64-baseos-rpms 798 k 2023-07-14T19:04:41.052 INFO:teuthology.orchestra.run.smithi130.stdout: librados2 x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 3.7 M 2023-07-14T19:04:41.052 INFO:teuthology.orchestra.run.smithi130.stdout: librbd1 x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 3.6 M 2023-07-14T19:04:41.052 INFO:teuthology.orchestra.run.smithi130.stdout:Installing dependencies: 2023-07-14T19:04:41.052 INFO:teuthology.orchestra.run.smithi130.stdout: automake noarch 1.16.1-7.el8 rhel-8-for-x86_64-appstream-rpms 713 k 2023-07-14T19:04:41.053 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-common x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 22 M 2023-07-14T19:04:41.053 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-grafana-dashboards noarch 2:18.0.0-4940.g81c9387d.el8 ceph-noarch 23 k 2023-07-14T19:04:41.053 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-mds x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 2.3 M 2023-07-14T19:04:41.053 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-mgr-modules-core noarch 2:18.0.0-4940.g81c9387d.el8 ceph-noarch 241 k 2023-07-14T19:04:41.053 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-mon x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 4.6 M 2023-07-14T19:04:41.053 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-osd x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 18 M 2023-07-14T19:04:41.053 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-prometheus-alerts noarch 2:18.0.0-4940.g81c9387d.el8 ceph-noarch 15 k 2023-07-14T19:04:41.054 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-selinux x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 24 k 2023-07-14T19:04:41.054 INFO:teuthology.orchestra.run.smithi130.stdout: fuse x86_64 2.9.7-15.el8 rhel-8-for-x86_64-baseos-rpms 83 k 2023-07-14T19:04:41.054 INFO:teuthology.orchestra.run.smithi130.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2023-07-14T19:04:41.054 INFO:teuthology.orchestra.run.smithi130.stdout: jq x86_64 1.6-3.el8 rhel-8-for-x86_64-appstream-rpms 202 k 2023-07-14T19:04:41.054 INFO:teuthology.orchestra.run.smithi130.stdout: keyutils-libs-devel x86_64 1.5.10-9.el8 rhel-8-for-x86_64-baseos-rpms 48 k 2023-07-14T19:04:41.054 INFO:teuthology.orchestra.run.smithi130.stdout: krb5-devel x86_64 1.18.2-14.el8 rhel-8-for-x86_64-baseos-rpms 560 k 2023-07-14T19:04:41.054 INFO:teuthology.orchestra.run.smithi130.stdout: libcephsqlite x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 135 k 2023-07-14T19:04:41.054 INFO:teuthology.orchestra.run.smithi130.stdout: libcom_err-devel x86_64 1.45.6-4.el8 rhel-8-for-x86_64-baseos-rpms 39 k 2023-07-14T19:04:41.055 INFO:teuthology.orchestra.run.smithi130.stdout: libgfortran x86_64 8.5.0-10.1.el8_6 rhel-8-for-x86_64-baseos-rpms 644 k 2023-07-14T19:04:41.055 INFO:teuthology.orchestra.run.smithi130.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2023-07-14T19:04:41.055 INFO:teuthology.orchestra.run.smithi130.stdout: libpq x86_64 13.5-1.el8 rhel-8-for-x86_64-appstream-rpms 198 k 2023-07-14T19:04:41.055 INFO:teuthology.orchestra.run.smithi130.stdout: libquadmath x86_64 8.5.0-10.1.el8_6 rhel-8-for-x86_64-baseos-rpms 171 k 2023-07-14T19:04:41.055 INFO:teuthology.orchestra.run.smithi130.stdout: librabbitmq x86_64 0.9.0-3.el8 rhel-8-for-x86_64-baseos-rpms 47 k 2023-07-14T19:04:41.055 INFO:teuthology.orchestra.run.smithi130.stdout: libradosstriper1 x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 473 k 2023-07-14T19:04:41.055 INFO:teuthology.orchestra.run.smithi130.stdout: librdkafka x86_64 0.11.4-3.el8 rhel-8-for-x86_64-appstream-rpms 354 k 2023-07-14T19:04:41.056 INFO:teuthology.orchestra.run.smithi130.stdout: librgw2 x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 6.8 M 2023-07-14T19:04:41.056 INFO:teuthology.orchestra.run.smithi130.stdout: libselinux-devel x86_64 2.9-5.el8 rhel-8-for-x86_64-baseos-rpms 200 k 2023-07-14T19:04:41.056 INFO:teuthology.orchestra.run.smithi130.stdout: libsepol-devel x86_64 2.9-3.el8 rhel-8-for-x86_64-baseos-rpms 87 k 2023-07-14T19:04:41.056 INFO:teuthology.orchestra.run.smithi130.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2023-07-14T19:04:41.056 INFO:teuthology.orchestra.run.smithi130.stdout: libverto-devel x86_64 0.3.0-5.el8 rhel-8-for-x86_64-baseos-rpms 18 k 2023-07-14T19:04:41.056 INFO:teuthology.orchestra.run.smithi130.stdout: lttng-ust x86_64 2.8.1-11.el8 rhel-8-for-x86_64-appstream-rpms 259 k 2023-07-14T19:04:41.056 INFO:teuthology.orchestra.run.smithi130.stdout: oniguruma x86_64 6.8.2-2.el8 rhel-8-for-x86_64-appstream-rpms 187 k 2023-07-14T19:04:41.056 INFO:teuthology.orchestra.run.smithi130.stdout: openblas x86_64 0.3.15-3.el8 rhel-8-for-x86_64-appstream-rpms 4.7 M 2023-07-14T19:04:41.057 INFO:teuthology.orchestra.run.smithi130.stdout: openblas-threads x86_64 0.3.15-3.el8 rhel-8-for-x86_64-appstream-rpms 4.9 M 2023-07-14T19:04:41.057 INFO:teuthology.orchestra.run.smithi130.stdout: pcre2-devel x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 605 k 2023-07-14T19:04:41.057 INFO:teuthology.orchestra.run.smithi130.stdout: pcre2-utf16 x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 229 k 2023-07-14T19:04:41.057 INFO:teuthology.orchestra.run.smithi130.stdout: pcre2-utf32 x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 220 k 2023-07-14T19:04:41.057 INFO:teuthology.orchestra.run.smithi130.stdout: python3-asyncssh noarch 2.7.0-2.el8 epel 476 k 2023-07-14T19:04:41.057 INFO:teuthology.orchestra.run.smithi130.stdout: python3-babel noarch 2.5.1-7.el8 rhel-8-for-x86_64-appstream-rpms 4.8 M 2023-07-14T19:04:41.057 INFO:teuthology.orchestra.run.smithi130.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2023-07-14T19:04:41.057 INFO:teuthology.orchestra.run.smithi130.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2023-07-14T19:04:41.058 INFO:teuthology.orchestra.run.smithi130.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2023-07-14T19:04:41.058 INFO:teuthology.orchestra.run.smithi130.stdout: python3-ceph-argparse x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 45 k 2023-07-14T19:04:41.058 INFO:teuthology.orchestra.run.smithi130.stdout: python3-ceph-common x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 122 k 2023-07-14T19:04:41.058 INFO:teuthology.orchestra.run.smithi130.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2023-07-14T19:04:41.058 INFO:teuthology.orchestra.run.smithi130.stdout: python3-cffi x86_64 1.11.5-5.el8 rhel-8-for-x86_64-baseos-rpms 238 k 2023-07-14T19:04:41.058 INFO:teuthology.orchestra.run.smithi130.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2023-07-14T19:04:41.058 INFO:teuthology.orchestra.run.smithi130.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2023-07-14T19:04:41.059 INFO:teuthology.orchestra.run.smithi130.stdout: python3-cryptography x86_64 3.2.1-5.el8 rhel-8-for-x86_64-baseos-rpms 559 k 2023-07-14T19:04:41.059 INFO:teuthology.orchestra.run.smithi130.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2023-07-14T19:04:41.059 INFO:teuthology.orchestra.run.smithi130.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2023-07-14T19:04:41.059 INFO:teuthology.orchestra.run.smithi130.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2023-07-14T19:04:41.059 INFO:teuthology.orchestra.run.smithi130.stdout: python3-jinja2 noarch 2.10.1-3.el8 rhel-8-for-x86_64-appstream-rpms 538 k 2023-07-14T19:04:41.059 INFO:teuthology.orchestra.run.smithi130.stdout: python3-jsonpatch noarch 1.21-2.el8 rhel-8-for-x86_64-appstream-rpms 27 k 2023-07-14T19:04:41.059 INFO:teuthology.orchestra.run.smithi130.stdout: python3-jsonpointer noarch 1.10-11.el8 rhel-8-for-x86_64-appstream-rpms 20 k 2023-07-14T19:04:41.059 INFO:teuthology.orchestra.run.smithi130.stdout: python3-jwt noarch 1.6.1-2.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2023-07-14T19:04:41.060 INFO:teuthology.orchestra.run.smithi130.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2023-07-14T19:04:41.060 INFO:teuthology.orchestra.run.smithi130.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2023-07-14T19:04:41.060 INFO:teuthology.orchestra.run.smithi130.stdout: python3-mako noarch 1.0.6-13.el8 rhel-8-for-x86_64-appstream-rpms 157 k 2023-07-14T19:04:41.060 INFO:teuthology.orchestra.run.smithi130.stdout: python3-markupsafe x86_64 0.23-19.el8 rhel-8-for-x86_64-appstream-rpms 39 k 2023-07-14T19:04:41.060 INFO:teuthology.orchestra.run.smithi130.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2023-07-14T19:04:41.060 INFO:teuthology.orchestra.run.smithi130.stdout: python3-natsort noarch 7.1.1-2.el8 epel 60 k 2023-07-14T19:04:41.060 INFO:teuthology.orchestra.run.smithi130.stdout: python3-numpy x86_64 1:1.14.3-10.el8 rhel-8-for-x86_64-appstream-rpms 3.7 M 2023-07-14T19:04:41.061 INFO:teuthology.orchestra.run.smithi130.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 rhel-8-for-x86_64-appstream-rpms 225 k 2023-07-14T19:04:41.061 INFO:teuthology.orchestra.run.smithi130.stdout: python3-oauthlib noarch 2.1.0-1.el8 rhel-8-for-x86_64-baseos-rpms 155 k 2023-07-14T19:04:41.061 INFO:teuthology.orchestra.run.smithi130.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2023-07-14T19:04:41.061 INFO:teuthology.orchestra.run.smithi130.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2023-07-14T19:04:41.061 INFO:teuthology.orchestra.run.smithi130.stdout: python3-prettytable noarch 0.7.2-14.el8 rhel-8-for-x86_64-appstream-rpms 44 k 2023-07-14T19:04:41.061 INFO:teuthology.orchestra.run.smithi130.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 rhel-8-for-x86_64-appstream-rpms 103 k 2023-07-14T19:04:41.061 INFO:teuthology.orchestra.run.smithi130.stdout: python3-pyasn1 noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 126 k 2023-07-14T19:04:41.062 INFO:teuthology.orchestra.run.smithi130.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 110 k 2023-07-14T19:04:41.062 INFO:teuthology.orchestra.run.smithi130.stdout: python3-pycparser noarch 2.14-14.el8 rhel-8-for-x86_64-baseos-rpms 109 k 2023-07-14T19:04:41.062 INFO:teuthology.orchestra.run.smithi130.stdout: python3-pytz noarch 2017.2-9.el8 rhel-8-for-x86_64-appstream-rpms 54 k 2023-07-14T19:04:41.062 INFO:teuthology.orchestra.run.smithi130.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2023-07-14T19:04:41.062 INFO:teuthology.orchestra.run.smithi130.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2023-07-14T19:04:41.062 INFO:teuthology.orchestra.run.smithi130.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2023-07-14T19:04:41.062 INFO:teuthology.orchestra.run.smithi130.stdout: python3-rsa noarch 4.8-1.el8 epel 61 k 2023-07-14T19:04:41.063 INFO:teuthology.orchestra.run.smithi130.stdout: python3-scipy x86_64 1.0.0-21.module+el8.5.0+10916+41bd434d rhel-8-for-x86_64-appstream-rpms 14 M 2023-07-14T19:04:41.063 INFO:teuthology.orchestra.run.smithi130.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2023-07-14T19:04:41.063 INFO:teuthology.orchestra.run.smithi130.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2023-07-14T19:04:41.063 INFO:teuthology.orchestra.run.smithi130.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2023-07-14T19:04:41.063 INFO:teuthology.orchestra.run.smithi130.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2023-07-14T19:04:41.063 INFO:teuthology.orchestra.run.smithi130.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2023-07-14T19:04:41.063 INFO:teuthology.orchestra.run.smithi130.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2023-07-14T19:04:41.063 INFO:teuthology.orchestra.run.smithi130.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2023-07-14T19:04:41.064 INFO:teuthology.orchestra.run.smithi130.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2023-07-14T19:04:41.064 INFO:teuthology.orchestra.run.smithi130.stdout: python3-werkzeug noarch 0.12.2-4.el8 rhel-8-for-x86_64-appstream-rpms 457 k 2023-07-14T19:04:41.064 INFO:teuthology.orchestra.run.smithi130.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2023-07-14T19:04:41.064 INFO:teuthology.orchestra.run.smithi130.stdout: socat x86_64 1.7.4.1-1.el8 rhel-8-for-x86_64-appstream-rpms 323 k 2023-07-14T19:04:41.064 INFO:teuthology.orchestra.run.smithi130.stdout: thrift x86_64 0.13.0-2.el8 epel 1.7 M 2023-07-14T19:04:41.064 INFO:teuthology.orchestra.run.smithi130.stdout: uuid x86_64 1.6.2-43.el8 rhel-8-for-x86_64-appstream-rpms 64 k 2023-07-14T19:04:41.064 INFO:teuthology.orchestra.run.smithi130.stdout: xmlstarlet x86_64 1.6.1-20.el8 lab-extras 70 k 2023-07-14T19:04:41.065 INFO:teuthology.orchestra.run.smithi130.stdout: zlib-devel x86_64 1.2.11-18.el8_5 rhel-8-for-x86_64-baseos-rpms 58 k 2023-07-14T19:04:41.065 INFO:teuthology.orchestra.run.smithi130.stdout:Enabling module streams: 2023-07-14T19:04:41.065 INFO:teuthology.orchestra.run.smithi130.stdout: postgresql 10 2023-07-14T19:04:41.065 INFO:teuthology.orchestra.run.smithi130.stdout: 2023-07-14T19:04:41.065 INFO:teuthology.orchestra.run.smithi130.stdout:Transaction Summary 2023-07-14T19:04:41.065 INFO:teuthology.orchestra.run.smithi130.stdout:=========================================================================================================================== 2023-07-14T19:04:41.065 INFO:teuthology.orchestra.run.smithi130.stdout:Install 128 Packages 2023-07-14T19:04:41.065 INFO:teuthology.orchestra.run.smithi130.stdout:Upgrade 3 Packages 2023-07-14T19:04:41.066 INFO:teuthology.orchestra.run.smithi130.stdout: 2023-07-14T19:04:41.066 INFO:teuthology.orchestra.run.smithi130.stdout:Total download size: 206 M 2023-07-14T19:04:41.066 INFO:teuthology.orchestra.run.smithi130.stdout:Downloading Packages: 2023-07-14T19:04:41.402 INFO:teuthology.orchestra.run.smithi130.stdout:(1/131): ceph-18.0.0-4940.g81c9387d.el8.x86_64. 28 kB/s | 6.4 kB 00:00 2023-07-14T19:04:41.706 INFO:teuthology.orchestra.run.smithi191.stdout:Package librados2-1:12.2.7-9.el8.x86_64 is already installed. 2023-07-14T19:04:41.707 INFO:teuthology.orchestra.run.smithi191.stdout:Package librbd1-1:12.2.7-9.el8.x86_64 is already installed. 2023-07-14T19:04:41.722 INFO:teuthology.orchestra.run.smithi191.stdout:Package NetworkManager-1:1.36.0-4.el8.x86_64 is already installed. 2023-07-14T19:04:41.723 INFO:teuthology.orchestra.run.smithi191.stdout:Package iproute-5.15.0-4.el8.x86_64 is already installed. 2023-07-14T19:04:41.724 INFO:teuthology.orchestra.run.smithi191.stdout:Package util-linux-2.32.1-35.el8.x86_64 is already installed. 2023-07-14T19:04:41.725 INFO:teuthology.orchestra.run.smithi191.stdout:Package libacl-devel-2.2.53-1.el8.x86_64 is already installed. 2023-07-14T19:04:41.727 INFO:teuthology.orchestra.run.smithi191.stdout:Package libattr-devel-2.4.48-3.el8.x86_64 is already installed. 2023-07-14T19:04:41.731 INFO:teuthology.orchestra.run.smithi191.stdout:Package xfsdump-3.1.8-4.el8.x86_64 is already installed. 2023-07-14T19:04:41.732 INFO:teuthology.orchestra.run.smithi191.stdout:Package xfsprogs-5.0.0-10.el8.x86_64 is already installed. 2023-07-14T19:04:41.864 INFO:teuthology.orchestra.run.smithi191.stdout:Dependencies resolved. 2023-07-14T19:04:41.876 INFO:teuthology.orchestra.run.smithi191.stdout:=========================================================================================================================== 2023-07-14T19:04:41.876 INFO:teuthology.orchestra.run.smithi191.stdout: Package Arch Version Repository Size 2023-07-14T19:04:41.877 INFO:teuthology.orchestra.run.smithi191.stdout:=========================================================================================================================== 2023-07-14T19:04:41.877 INFO:teuthology.orchestra.run.smithi191.stdout:Installing: 2023-07-14T19:04:41.877 INFO:teuthology.orchestra.run.smithi191.stdout: NetworkManager-initscripts-updown noarch 1:1.36.0-9.el8_6 rhel-8-for-x86_64-baseos-rpms 138 k 2023-07-14T19:04:41.877 INFO:teuthology.orchestra.run.smithi191.stdout: bison x86_64 3.0.4-10.el8 rhel-8-for-x86_64-appstream-rpms 688 k 2023-07-14T19:04:41.877 INFO:teuthology.orchestra.run.smithi191.stdout: ceph x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 6.4 k 2023-07-14T19:04:41.877 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-base x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 5.4 M 2023-07-14T19:04:41.877 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-fuse x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 850 k 2023-07-14T19:04:41.878 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-immutable-object-cache x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 167 k 2023-07-14T19:04:41.878 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-mgr x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 1.6 M 2023-07-14T19:04:41.878 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-mgr-cephadm noarch 2:18.0.0-4940.g81c9387d.el8 ceph-noarch 129 k 2023-07-14T19:04:41.878 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-mgr-dashboard noarch 2:18.0.0-4940.g81c9387d.el8 ceph-noarch 1.7 M 2023-07-14T19:04:41.878 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-mgr-diskprediction-local noarch 2:18.0.0-4940.g81c9387d.el8 ceph-noarch 7.4 M 2023-07-14T19:04:41.878 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-mgr-rook noarch 2:18.0.0-4940.g81c9387d.el8 ceph-noarch 47 k 2023-07-14T19:04:41.878 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-radosgw x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 14 M 2023-07-14T19:04:41.879 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-test x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 48 M 2023-07-14T19:04:41.879 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-volume noarch 2:18.0.0-4940.g81c9387d.el8 ceph-noarch 258 k 2023-07-14T19:04:41.879 INFO:teuthology.orchestra.run.smithi191.stdout: cephadm noarch 2:18.0.0-4940.g81c9387d.el8 ceph-noarch 165 k 2023-07-14T19:04:41.879 INFO:teuthology.orchestra.run.smithi191.stdout: cephfs-mirror x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 260 k 2023-07-14T19:04:41.879 INFO:teuthology.orchestra.run.smithi191.stdout: cephfs-top noarch 2:18.0.0-4940.g81c9387d.el8 ceph-noarch 18 k 2023-07-14T19:04:41.879 INFO:teuthology.orchestra.run.smithi191.stdout: elfutils-libelf-devel x86_64 0.186-1.el8 rhel-8-for-x86_64-baseos-rpms 60 k 2023-07-14T19:04:41.879 INFO:teuthology.orchestra.run.smithi191.stdout: flex x86_64 2.6.1-9.el8 rhel-8-for-x86_64-appstream-rpms 320 k 2023-07-14T19:04:41.879 INFO:teuthology.orchestra.run.smithi191.stdout: libaio-devel x86_64 0.3.112-1.el8 rhel-8-for-x86_64-baseos-rpms 19 k 2023-07-14T19:04:41.880 INFO:teuthology.orchestra.run.smithi191.stdout: libcephfs-devel x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 31 k 2023-07-14T19:04:41.880 INFO:teuthology.orchestra.run.smithi191.stdout: libcephfs2 x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 772 k 2023-07-14T19:04:41.880 INFO:teuthology.orchestra.run.smithi191.stdout: librados-devel x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 139 k 2023-07-14T19:04:41.880 INFO:teuthology.orchestra.run.smithi191.stdout: libtool x86_64 2.4.6-25.el8 rhel-8-for-x86_64-appstream-rpms 709 k 2023-07-14T19:04:41.880 INFO:teuthology.orchestra.run.smithi191.stdout: libuuid-devel x86_64 2.32.1-35.el8 rhel-8-for-x86_64-baseos-rpms 98 k 2023-07-14T19:04:41.880 INFO:teuthology.orchestra.run.smithi191.stdout: openssl-devel x86_64 1:1.1.1k-7.el8_6 rhel-8-for-x86_64-baseos-rpms 2.3 M 2023-07-14T19:04:41.880 INFO:teuthology.orchestra.run.smithi191.stdout: postgresql x86_64 10.21-2.module+el8.6.0+15342+53518fac rhel-8-for-x86_64-appstream-rpms 1.5 M 2023-07-14T19:04:41.881 INFO:teuthology.orchestra.run.smithi191.stdout: postgresql-contrib x86_64 10.21-2.module+el8.6.0+15342+53518fac rhel-8-for-x86_64-appstream-rpms 808 k 2023-07-14T19:04:41.881 INFO:teuthology.orchestra.run.smithi191.stdout: postgresql-server x86_64 10.21-2.module+el8.6.0+15342+53518fac rhel-8-for-x86_64-appstream-rpms 5.1 M 2023-07-14T19:04:41.881 INFO:teuthology.orchestra.run.smithi191.stdout: python3-cephfs x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 197 k 2023-07-14T19:04:41.881 INFO:teuthology.orchestra.run.smithi191.stdout: python3-rados x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 371 k 2023-07-14T19:04:41.881 INFO:teuthology.orchestra.run.smithi191.stdout: python3-rbd x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 364 k 2023-07-14T19:04:41.881 INFO:teuthology.orchestra.run.smithi191.stdout: python3-rgw x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 107 k 2023-07-14T19:04:41.881 INFO:teuthology.orchestra.run.smithi191.stdout: rbd-fuse x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 94 k 2023-07-14T19:04:41.882 INFO:teuthology.orchestra.run.smithi191.stdout: rbd-mirror x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 3.5 M 2023-07-14T19:04:41.882 INFO:teuthology.orchestra.run.smithi191.stdout: rbd-nbd x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 179 k 2023-07-14T19:04:41.882 INFO:teuthology.orchestra.run.smithi191.stdout: xfsprogs-devel x86_64 5.0.0-10.el8 rhel-8-for-x86_64-baseos-rpms 72 k 2023-07-14T19:04:41.882 INFO:teuthology.orchestra.run.smithi191.stdout:Upgrading: 2023-07-14T19:04:41.882 INFO:teuthology.orchestra.run.smithi191.stdout: iproute x86_64 5.15.0-4.el8_6.1 rhel-8-for-x86_64-baseos-rpms 798 k 2023-07-14T19:04:41.882 INFO:teuthology.orchestra.run.smithi191.stdout: librados2 x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 3.7 M 2023-07-14T19:04:41.882 INFO:teuthology.orchestra.run.smithi191.stdout: librbd1 x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 3.6 M 2023-07-14T19:04:41.882 INFO:teuthology.orchestra.run.smithi191.stdout:Installing dependencies: 2023-07-14T19:04:41.883 INFO:teuthology.orchestra.run.smithi191.stdout: automake noarch 1.16.1-7.el8 rhel-8-for-x86_64-appstream-rpms 713 k 2023-07-14T19:04:41.883 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-common x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 22 M 2023-07-14T19:04:41.883 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-grafana-dashboards noarch 2:18.0.0-4940.g81c9387d.el8 ceph-noarch 23 k 2023-07-14T19:04:41.883 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-mds x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 2.3 M 2023-07-14T19:04:41.883 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-mgr-modules-core noarch 2:18.0.0-4940.g81c9387d.el8 ceph-noarch 241 k 2023-07-14T19:04:41.883 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-mon x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 4.6 M 2023-07-14T19:04:41.883 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-osd x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 18 M 2023-07-14T19:04:41.884 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-prometheus-alerts noarch 2:18.0.0-4940.g81c9387d.el8 ceph-noarch 15 k 2023-07-14T19:04:41.884 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-selinux x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 24 k 2023-07-14T19:04:41.884 INFO:teuthology.orchestra.run.smithi191.stdout: fuse x86_64 2.9.7-15.el8 rhel-8-for-x86_64-baseos-rpms 83 k 2023-07-14T19:04:41.884 INFO:teuthology.orchestra.run.smithi191.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2023-07-14T19:04:41.884 INFO:teuthology.orchestra.run.smithi191.stdout: jq x86_64 1.6-3.el8 rhel-8-for-x86_64-appstream-rpms 202 k 2023-07-14T19:04:41.884 INFO:teuthology.orchestra.run.smithi191.stdout: keyutils-libs-devel x86_64 1.5.10-9.el8 rhel-8-for-x86_64-baseos-rpms 48 k 2023-07-14T19:04:41.884 INFO:teuthology.orchestra.run.smithi191.stdout: krb5-devel x86_64 1.18.2-14.el8 rhel-8-for-x86_64-baseos-rpms 560 k 2023-07-14T19:04:41.884 INFO:teuthology.orchestra.run.smithi191.stdout: libcephsqlite x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 135 k 2023-07-14T19:04:41.885 INFO:teuthology.orchestra.run.smithi191.stdout: libcom_err-devel x86_64 1.45.6-4.el8 rhel-8-for-x86_64-baseos-rpms 39 k 2023-07-14T19:04:41.885 INFO:teuthology.orchestra.run.smithi191.stdout: libgfortran x86_64 8.5.0-10.1.el8_6 rhel-8-for-x86_64-baseos-rpms 644 k 2023-07-14T19:04:41.885 INFO:teuthology.orchestra.run.smithi191.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2023-07-14T19:04:41.885 INFO:teuthology.orchestra.run.smithi191.stdout: libpq x86_64 13.5-1.el8 rhel-8-for-x86_64-appstream-rpms 198 k 2023-07-14T19:04:41.885 INFO:teuthology.orchestra.run.smithi191.stdout: libquadmath x86_64 8.5.0-10.1.el8_6 rhel-8-for-x86_64-baseos-rpms 171 k 2023-07-14T19:04:41.885 INFO:teuthology.orchestra.run.smithi191.stdout: librabbitmq x86_64 0.9.0-3.el8 rhel-8-for-x86_64-baseos-rpms 47 k 2023-07-14T19:04:41.885 INFO:teuthology.orchestra.run.smithi191.stdout: libradosstriper1 x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 473 k 2023-07-14T19:04:41.886 INFO:teuthology.orchestra.run.smithi191.stdout: librdkafka x86_64 0.11.4-3.el8 rhel-8-for-x86_64-appstream-rpms 354 k 2023-07-14T19:04:41.886 INFO:teuthology.orchestra.run.smithi191.stdout: librgw2 x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 6.8 M 2023-07-14T19:04:41.886 INFO:teuthology.orchestra.run.smithi191.stdout: libselinux-devel x86_64 2.9-5.el8 rhel-8-for-x86_64-baseos-rpms 200 k 2023-07-14T19:04:41.886 INFO:teuthology.orchestra.run.smithi191.stdout: libsepol-devel x86_64 2.9-3.el8 rhel-8-for-x86_64-baseos-rpms 87 k 2023-07-14T19:04:41.886 INFO:teuthology.orchestra.run.smithi191.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2023-07-14T19:04:41.886 INFO:teuthology.orchestra.run.smithi191.stdout: libverto-devel x86_64 0.3.0-5.el8 rhel-8-for-x86_64-baseos-rpms 18 k 2023-07-14T19:04:41.886 INFO:teuthology.orchestra.run.smithi191.stdout: lttng-ust x86_64 2.8.1-11.el8 rhel-8-for-x86_64-appstream-rpms 259 k 2023-07-14T19:04:41.886 INFO:teuthology.orchestra.run.smithi191.stdout: oniguruma x86_64 6.8.2-2.el8 rhel-8-for-x86_64-appstream-rpms 187 k 2023-07-14T19:04:41.887 INFO:teuthology.orchestra.run.smithi191.stdout: openblas x86_64 0.3.15-3.el8 rhel-8-for-x86_64-appstream-rpms 4.7 M 2023-07-14T19:04:41.887 INFO:teuthology.orchestra.run.smithi191.stdout: openblas-threads x86_64 0.3.15-3.el8 rhel-8-for-x86_64-appstream-rpms 4.9 M 2023-07-14T19:04:41.887 INFO:teuthology.orchestra.run.smithi191.stdout: pcre2-devel x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 605 k 2023-07-14T19:04:41.887 INFO:teuthology.orchestra.run.smithi191.stdout: pcre2-utf16 x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 229 k 2023-07-14T19:04:41.887 INFO:teuthology.orchestra.run.smithi191.stdout: pcre2-utf32 x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 220 k 2023-07-14T19:04:41.887 INFO:teuthology.orchestra.run.smithi191.stdout: python3-asyncssh noarch 2.7.0-2.el8 epel 476 k 2023-07-14T19:04:41.887 INFO:teuthology.orchestra.run.smithi191.stdout: python3-babel noarch 2.5.1-7.el8 rhel-8-for-x86_64-appstream-rpms 4.8 M 2023-07-14T19:04:41.888 INFO:teuthology.orchestra.run.smithi191.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2023-07-14T19:04:41.888 INFO:teuthology.orchestra.run.smithi191.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2023-07-14T19:04:41.888 INFO:teuthology.orchestra.run.smithi191.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2023-07-14T19:04:41.888 INFO:teuthology.orchestra.run.smithi191.stdout: python3-ceph-argparse x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 45 k 2023-07-14T19:04:41.888 INFO:teuthology.orchestra.run.smithi191.stdout: python3-ceph-common x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 122 k 2023-07-14T19:04:41.888 INFO:teuthology.orchestra.run.smithi191.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2023-07-14T19:04:41.888 INFO:teuthology.orchestra.run.smithi191.stdout: python3-cffi x86_64 1.11.5-5.el8 rhel-8-for-x86_64-baseos-rpms 238 k 2023-07-14T19:04:41.888 INFO:teuthology.orchestra.run.smithi191.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2023-07-14T19:04:41.889 INFO:teuthology.orchestra.run.smithi191.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2023-07-14T19:04:41.889 INFO:teuthology.orchestra.run.smithi191.stdout: python3-cryptography x86_64 3.2.1-5.el8 rhel-8-for-x86_64-baseos-rpms 559 k 2023-07-14T19:04:41.889 INFO:teuthology.orchestra.run.smithi191.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2023-07-14T19:04:41.889 INFO:teuthology.orchestra.run.smithi191.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2023-07-14T19:04:41.889 INFO:teuthology.orchestra.run.smithi191.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2023-07-14T19:04:41.889 INFO:teuthology.orchestra.run.smithi191.stdout: python3-jinja2 noarch 2.10.1-3.el8 rhel-8-for-x86_64-appstream-rpms 538 k 2023-07-14T19:04:41.889 INFO:teuthology.orchestra.run.smithi191.stdout: python3-jsonpatch noarch 1.21-2.el8 rhel-8-for-x86_64-appstream-rpms 27 k 2023-07-14T19:04:41.890 INFO:teuthology.orchestra.run.smithi191.stdout: python3-jsonpointer noarch 1.10-11.el8 rhel-8-for-x86_64-appstream-rpms 20 k 2023-07-14T19:04:41.890 INFO:teuthology.orchestra.run.smithi191.stdout: python3-jwt noarch 1.6.1-2.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2023-07-14T19:04:41.890 INFO:teuthology.orchestra.run.smithi191.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2023-07-14T19:04:41.890 INFO:teuthology.orchestra.run.smithi191.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2023-07-14T19:04:41.890 INFO:teuthology.orchestra.run.smithi191.stdout: python3-mako noarch 1.0.6-13.el8 rhel-8-for-x86_64-appstream-rpms 157 k 2023-07-14T19:04:41.890 INFO:teuthology.orchestra.run.smithi191.stdout: python3-markupsafe x86_64 0.23-19.el8 rhel-8-for-x86_64-appstream-rpms 39 k 2023-07-14T19:04:41.890 INFO:teuthology.orchestra.run.smithi191.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2023-07-14T19:04:41.890 INFO:teuthology.orchestra.run.smithi191.stdout: python3-natsort noarch 7.1.1-2.el8 epel 60 k 2023-07-14T19:04:41.891 INFO:teuthology.orchestra.run.smithi191.stdout: python3-numpy x86_64 1:1.14.3-10.el8 rhel-8-for-x86_64-appstream-rpms 3.7 M 2023-07-14T19:04:41.891 INFO:teuthology.orchestra.run.smithi191.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 rhel-8-for-x86_64-appstream-rpms 225 k 2023-07-14T19:04:41.891 INFO:teuthology.orchestra.run.smithi191.stdout: python3-oauthlib noarch 2.1.0-1.el8 rhel-8-for-x86_64-baseos-rpms 155 k 2023-07-14T19:04:41.891 INFO:teuthology.orchestra.run.smithi191.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2023-07-14T19:04:41.891 INFO:teuthology.orchestra.run.smithi191.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2023-07-14T19:04:41.891 INFO:teuthology.orchestra.run.smithi191.stdout: python3-prettytable noarch 0.7.2-14.el8 rhel-8-for-x86_64-appstream-rpms 44 k 2023-07-14T19:04:41.891 INFO:teuthology.orchestra.run.smithi191.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 rhel-8-for-x86_64-appstream-rpms 103 k 2023-07-14T19:04:41.892 INFO:teuthology.orchestra.run.smithi191.stdout: python3-pyasn1 noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 126 k 2023-07-14T19:04:41.892 INFO:teuthology.orchestra.run.smithi191.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 110 k 2023-07-14T19:04:41.892 INFO:teuthology.orchestra.run.smithi191.stdout: python3-pycparser noarch 2.14-14.el8 rhel-8-for-x86_64-baseos-rpms 109 k 2023-07-14T19:04:41.892 INFO:teuthology.orchestra.run.smithi191.stdout: python3-pytz noarch 2017.2-9.el8 rhel-8-for-x86_64-appstream-rpms 54 k 2023-07-14T19:04:41.892 INFO:teuthology.orchestra.run.smithi191.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2023-07-14T19:04:41.892 INFO:teuthology.orchestra.run.smithi191.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2023-07-14T19:04:41.892 INFO:teuthology.orchestra.run.smithi191.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2023-07-14T19:04:41.892 INFO:teuthology.orchestra.run.smithi191.stdout: python3-rsa noarch 4.8-1.el8 epel 61 k 2023-07-14T19:04:41.893 INFO:teuthology.orchestra.run.smithi191.stdout: python3-scipy x86_64 1.0.0-21.module+el8.5.0+10916+41bd434d rhel-8-for-x86_64-appstream-rpms 14 M 2023-07-14T19:04:41.893 INFO:teuthology.orchestra.run.smithi191.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2023-07-14T19:04:41.893 INFO:teuthology.orchestra.run.smithi191.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2023-07-14T19:04:41.893 INFO:teuthology.orchestra.run.smithi191.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2023-07-14T19:04:41.893 INFO:teuthology.orchestra.run.smithi191.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2023-07-14T19:04:41.893 INFO:teuthology.orchestra.run.smithi191.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2023-07-14T19:04:41.893 INFO:teuthology.orchestra.run.smithi191.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2023-07-14T19:04:41.894 INFO:teuthology.orchestra.run.smithi191.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2023-07-14T19:04:41.894 INFO:teuthology.orchestra.run.smithi191.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2023-07-14T19:04:41.894 INFO:teuthology.orchestra.run.smithi191.stdout: python3-werkzeug noarch 0.12.2-4.el8 rhel-8-for-x86_64-appstream-rpms 457 k 2023-07-14T19:04:41.894 INFO:teuthology.orchestra.run.smithi191.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2023-07-14T19:04:41.894 INFO:teuthology.orchestra.run.smithi191.stdout: socat x86_64 1.7.4.1-1.el8 rhel-8-for-x86_64-appstream-rpms 323 k 2023-07-14T19:04:41.894 INFO:teuthology.orchestra.run.smithi191.stdout: thrift x86_64 0.13.0-2.el8 epel 1.7 M 2023-07-14T19:04:41.894 INFO:teuthology.orchestra.run.smithi191.stdout: uuid x86_64 1.6.2-43.el8 rhel-8-for-x86_64-appstream-rpms 64 k 2023-07-14T19:04:41.894 INFO:teuthology.orchestra.run.smithi191.stdout: xmlstarlet x86_64 1.6.1-20.el8 lab-extras 70 k 2023-07-14T19:04:41.895 INFO:teuthology.orchestra.run.smithi191.stdout: zlib-devel x86_64 1.2.11-18.el8_5 rhel-8-for-x86_64-baseos-rpms 58 k 2023-07-14T19:04:41.895 INFO:teuthology.orchestra.run.smithi191.stdout:Enabling module streams: 2023-07-14T19:04:41.895 INFO:teuthology.orchestra.run.smithi191.stdout: postgresql 10 2023-07-14T19:04:41.895 INFO:teuthology.orchestra.run.smithi191.stdout: 2023-07-14T19:04:41.895 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction Summary 2023-07-14T19:04:41.895 INFO:teuthology.orchestra.run.smithi191.stdout:=========================================================================================================================== 2023-07-14T19:04:41.895 INFO:teuthology.orchestra.run.smithi191.stdout:Install 128 Packages 2023-07-14T19:04:41.896 INFO:teuthology.orchestra.run.smithi191.stdout:Upgrade 3 Packages 2023-07-14T19:04:41.896 INFO:teuthology.orchestra.run.smithi191.stdout: 2023-07-14T19:04:41.896 INFO:teuthology.orchestra.run.smithi191.stdout:Total download size: 206 M 2023-07-14T19:04:41.896 INFO:teuthology.orchestra.run.smithi191.stdout:Downloading Packages: 2023-07-14T19:04:42.226 INFO:teuthology.orchestra.run.smithi130.stdout:(2/131): ceph-fuse-18.0.0-4940.g81c9387d.el8.x8 1.0 MB/s | 850 kB 00:00 2023-07-14T19:04:42.377 INFO:teuthology.orchestra.run.smithi130.stdout:(3/131): ceph-immutable-object-cache-18.0.0-494 1.1 MB/s | 167 kB 00:00 2023-07-14T19:04:42.433 INFO:teuthology.orchestra.run.smithi191.stdout:(1/131): ceph-18.0.0-4940.g81c9387d.el8.x86_64. 23 kB/s | 6.4 kB 00:00 2023-07-14T19:04:43.560 INFO:teuthology.orchestra.run.smithi130.stdout:(4/131): ceph-mds-18.0.0-4940.g81c9387d.el8.x86 1.9 MB/s | 2.3 MB 00:01 2023-07-14T19:04:43.716 INFO:teuthology.orchestra.run.smithi191.stdout:(2/131): ceph-fuse-18.0.0-4940.g81c9387d.el8.x8 663 kB/s | 850 kB 00:01 2023-07-14T19:04:43.950 INFO:teuthology.orchestra.run.smithi191.stdout:(3/131): ceph-immutable-object-cache-18.0.0-494 716 kB/s | 167 kB 00:00 2023-07-14T19:04:44.052 INFO:teuthology.orchestra.run.smithi130.stdout:(5/131): ceph-mgr-18.0.0-4940.g81c9387d.el8.x86 3.2 MB/s | 1.6 MB 00:00 2023-07-14T19:04:44.142 INFO:teuthology.orchestra.run.smithi130.stdout:(6/131): ceph-base-18.0.0-4940.g81c9387d.el8.x8 1.8 MB/s | 5.4 MB 00:02 2023-07-14T19:04:45.253 INFO:teuthology.orchestra.run.smithi130.stdout:(7/131): ceph-mon-18.0.0-4940.g81c9387d.el8.x86 3.8 MB/s | 4.6 MB 00:01 2023-07-14T19:04:46.808 INFO:teuthology.orchestra.run.smithi191.stdout:(4/131): ceph-mds-18.0.0-4940.g81c9387d.el8.x86 822 kB/s | 2.3 MB 00:02 2023-07-14T19:04:47.103 INFO:teuthology.orchestra.run.smithi130.stdout:(8/131): ceph-radosgw-18.0.0-4940.g81c9387d.el8 7.3 MB/s | 14 MB 00:01 2023-07-14T19:04:47.187 INFO:teuthology.orchestra.run.smithi130.stdout:(9/131): ceph-selinux-18.0.0-4940.g81c9387d.el8 293 kB/s | 24 kB 00:00 2023-07-14T19:04:48.441 INFO:teuthology.orchestra.run.smithi191.stdout:(5/131): ceph-mgr-18.0.0-4940.g81c9387d.el8.x86 973 kB/s | 1.6 MB 00:01 2023-07-14T19:04:49.304 INFO:teuthology.orchestra.run.smithi040.stdout:Package librados2-1:12.2.7-9.el8.x86_64 is already installed. 2023-07-14T19:04:49.305 INFO:teuthology.orchestra.run.smithi040.stdout:Package librbd1-1:12.2.7-9.el8.x86_64 is already installed. 2023-07-14T19:04:49.320 INFO:teuthology.orchestra.run.smithi040.stdout:Package NetworkManager-1:1.36.0-4.el8.x86_64 is already installed. 2023-07-14T19:04:49.321 INFO:teuthology.orchestra.run.smithi040.stdout:Package iproute-5.15.0-4.el8.x86_64 is already installed. 2023-07-14T19:04:49.322 INFO:teuthology.orchestra.run.smithi040.stdout:Package util-linux-2.32.1-35.el8.x86_64 is already installed. 2023-07-14T19:04:49.323 INFO:teuthology.orchestra.run.smithi040.stdout:Package libacl-devel-2.2.53-1.el8.x86_64 is already installed. 2023-07-14T19:04:49.326 INFO:teuthology.orchestra.run.smithi040.stdout:Package libattr-devel-2.4.48-3.el8.x86_64 is already installed. 2023-07-14T19:04:49.329 INFO:teuthology.orchestra.run.smithi040.stdout:Package xfsdump-3.1.8-4.el8.x86_64 is already installed. 2023-07-14T19:04:49.330 INFO:teuthology.orchestra.run.smithi040.stdout:Package xfsprogs-5.0.0-10.el8.x86_64 is already installed. 2023-07-14T19:04:49.472 INFO:teuthology.orchestra.run.smithi040.stdout:Dependencies resolved. 2023-07-14T19:04:49.485 INFO:teuthology.orchestra.run.smithi040.stdout:=========================================================================================================================== 2023-07-14T19:04:49.486 INFO:teuthology.orchestra.run.smithi040.stdout: Package Arch Version Repository Size 2023-07-14T19:04:49.486 INFO:teuthology.orchestra.run.smithi040.stdout:=========================================================================================================================== 2023-07-14T19:04:49.486 INFO:teuthology.orchestra.run.smithi040.stdout:Installing: 2023-07-14T19:04:49.487 INFO:teuthology.orchestra.run.smithi040.stdout: NetworkManager-initscripts-updown noarch 1:1.36.0-9.el8_6 rhel-8-for-x86_64-baseos-rpms 138 k 2023-07-14T19:04:49.487 INFO:teuthology.orchestra.run.smithi040.stdout: bison x86_64 3.0.4-10.el8 rhel-8-for-x86_64-appstream-rpms 688 k 2023-07-14T19:04:49.487 INFO:teuthology.orchestra.run.smithi040.stdout: ceph x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 6.4 k 2023-07-14T19:04:49.488 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-base x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 5.4 M 2023-07-14T19:04:49.488 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-fuse x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 850 k 2023-07-14T19:04:49.488 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-immutable-object-cache x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 167 k 2023-07-14T19:04:49.489 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-mgr x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 1.6 M 2023-07-14T19:04:49.489 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-mgr-cephadm noarch 2:18.0.0-4940.g81c9387d.el8 ceph-noarch 129 k 2023-07-14T19:04:49.489 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-mgr-dashboard noarch 2:18.0.0-4940.g81c9387d.el8 ceph-noarch 1.7 M 2023-07-14T19:04:49.489 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-mgr-diskprediction-local noarch 2:18.0.0-4940.g81c9387d.el8 ceph-noarch 7.4 M 2023-07-14T19:04:49.490 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-mgr-rook noarch 2:18.0.0-4940.g81c9387d.el8 ceph-noarch 47 k 2023-07-14T19:04:49.490 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-radosgw x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 14 M 2023-07-14T19:04:49.490 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-test x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 48 M 2023-07-14T19:04:49.491 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-volume noarch 2:18.0.0-4940.g81c9387d.el8 ceph-noarch 258 k 2023-07-14T19:04:49.491 INFO:teuthology.orchestra.run.smithi040.stdout: cephadm noarch 2:18.0.0-4940.g81c9387d.el8 ceph-noarch 165 k 2023-07-14T19:04:49.491 INFO:teuthology.orchestra.run.smithi040.stdout: cephfs-mirror x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 260 k 2023-07-14T19:04:49.491 INFO:teuthology.orchestra.run.smithi040.stdout: cephfs-top noarch 2:18.0.0-4940.g81c9387d.el8 ceph-noarch 18 k 2023-07-14T19:04:49.491 INFO:teuthology.orchestra.run.smithi040.stdout: elfutils-libelf-devel x86_64 0.186-1.el8 rhel-8-for-x86_64-baseos-rpms 60 k 2023-07-14T19:04:49.492 INFO:teuthology.orchestra.run.smithi040.stdout: flex x86_64 2.6.1-9.el8 rhel-8-for-x86_64-appstream-rpms 320 k 2023-07-14T19:04:49.492 INFO:teuthology.orchestra.run.smithi040.stdout: libaio-devel x86_64 0.3.112-1.el8 rhel-8-for-x86_64-baseos-rpms 19 k 2023-07-14T19:04:49.492 INFO:teuthology.orchestra.run.smithi040.stdout: libcephfs-devel x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 31 k 2023-07-14T19:04:49.492 INFO:teuthology.orchestra.run.smithi040.stdout: libcephfs2 x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 772 k 2023-07-14T19:04:49.493 INFO:teuthology.orchestra.run.smithi040.stdout: librados-devel x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 139 k 2023-07-14T19:04:49.493 INFO:teuthology.orchestra.run.smithi040.stdout: libtool x86_64 2.4.6-25.el8 rhel-8-for-x86_64-appstream-rpms 709 k 2023-07-14T19:04:49.493 INFO:teuthology.orchestra.run.smithi040.stdout: libuuid-devel x86_64 2.32.1-35.el8 rhel-8-for-x86_64-baseos-rpms 98 k 2023-07-14T19:04:49.494 INFO:teuthology.orchestra.run.smithi040.stdout: openssl-devel x86_64 1:1.1.1k-7.el8_6 rhel-8-for-x86_64-baseos-rpms 2.3 M 2023-07-14T19:04:49.494 INFO:teuthology.orchestra.run.smithi040.stdout: postgresql x86_64 10.21-2.module+el8.6.0+15342+53518fac rhel-8-for-x86_64-appstream-rpms 1.5 M 2023-07-14T19:04:49.494 INFO:teuthology.orchestra.run.smithi040.stdout: postgresql-contrib x86_64 10.21-2.module+el8.6.0+15342+53518fac rhel-8-for-x86_64-appstream-rpms 808 k 2023-07-14T19:04:49.494 INFO:teuthology.orchestra.run.smithi040.stdout: postgresql-server x86_64 10.21-2.module+el8.6.0+15342+53518fac rhel-8-for-x86_64-appstream-rpms 5.1 M 2023-07-14T19:04:49.495 INFO:teuthology.orchestra.run.smithi040.stdout: python3-cephfs x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 197 k 2023-07-14T19:04:49.495 INFO:teuthology.orchestra.run.smithi040.stdout: python3-rados x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 371 k 2023-07-14T19:04:49.495 INFO:teuthology.orchestra.run.smithi040.stdout: python3-rbd x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 364 k 2023-07-14T19:04:49.495 INFO:teuthology.orchestra.run.smithi040.stdout: python3-rgw x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 107 k 2023-07-14T19:04:49.495 INFO:teuthology.orchestra.run.smithi040.stdout: rbd-fuse x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 94 k 2023-07-14T19:04:49.496 INFO:teuthology.orchestra.run.smithi040.stdout: rbd-mirror x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 3.5 M 2023-07-14T19:04:49.496 INFO:teuthology.orchestra.run.smithi040.stdout: rbd-nbd x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 179 k 2023-07-14T19:04:49.496 INFO:teuthology.orchestra.run.smithi040.stdout: xfsprogs-devel x86_64 5.0.0-10.el8 rhel-8-for-x86_64-baseos-rpms 72 k 2023-07-14T19:04:49.496 INFO:teuthology.orchestra.run.smithi040.stdout:Upgrading: 2023-07-14T19:04:49.496 INFO:teuthology.orchestra.run.smithi040.stdout: iproute x86_64 5.15.0-4.el8_6.1 rhel-8-for-x86_64-baseos-rpms 798 k 2023-07-14T19:04:49.496 INFO:teuthology.orchestra.run.smithi040.stdout: librados2 x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 3.7 M 2023-07-14T19:04:49.497 INFO:teuthology.orchestra.run.smithi040.stdout: librbd1 x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 3.6 M 2023-07-14T19:04:49.497 INFO:teuthology.orchestra.run.smithi040.stdout:Installing dependencies: 2023-07-14T19:04:49.497 INFO:teuthology.orchestra.run.smithi040.stdout: automake noarch 1.16.1-7.el8 rhel-8-for-x86_64-appstream-rpms 713 k 2023-07-14T19:04:49.497 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-common x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 22 M 2023-07-14T19:04:49.497 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-grafana-dashboards noarch 2:18.0.0-4940.g81c9387d.el8 ceph-noarch 23 k 2023-07-14T19:04:49.497 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-mds x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 2.3 M 2023-07-14T19:04:49.497 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-mgr-modules-core noarch 2:18.0.0-4940.g81c9387d.el8 ceph-noarch 241 k 2023-07-14T19:04:49.497 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-mon x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 4.6 M 2023-07-14T19:04:49.498 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-osd x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 18 M 2023-07-14T19:04:49.498 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-prometheus-alerts noarch 2:18.0.0-4940.g81c9387d.el8 ceph-noarch 15 k 2023-07-14T19:04:49.498 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-selinux x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 24 k 2023-07-14T19:04:49.498 INFO:teuthology.orchestra.run.smithi040.stdout: fuse x86_64 2.9.7-15.el8 rhel-8-for-x86_64-baseos-rpms 83 k 2023-07-14T19:04:49.498 INFO:teuthology.orchestra.run.smithi040.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2023-07-14T19:04:49.498 INFO:teuthology.orchestra.run.smithi040.stdout: jq x86_64 1.6-3.el8 rhel-8-for-x86_64-appstream-rpms 202 k 2023-07-14T19:04:49.498 INFO:teuthology.orchestra.run.smithi040.stdout: keyutils-libs-devel x86_64 1.5.10-9.el8 rhel-8-for-x86_64-baseos-rpms 48 k 2023-07-14T19:04:49.499 INFO:teuthology.orchestra.run.smithi040.stdout: krb5-devel x86_64 1.18.2-14.el8 rhel-8-for-x86_64-baseos-rpms 560 k 2023-07-14T19:04:49.499 INFO:teuthology.orchestra.run.smithi040.stdout: libcephsqlite x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 135 k 2023-07-14T19:04:49.499 INFO:teuthology.orchestra.run.smithi040.stdout: libcom_err-devel x86_64 1.45.6-4.el8 rhel-8-for-x86_64-baseos-rpms 39 k 2023-07-14T19:04:49.499 INFO:teuthology.orchestra.run.smithi040.stdout: libgfortran x86_64 8.5.0-10.1.el8_6 rhel-8-for-x86_64-baseos-rpms 644 k 2023-07-14T19:04:49.499 INFO:teuthology.orchestra.run.smithi040.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2023-07-14T19:04:49.499 INFO:teuthology.orchestra.run.smithi040.stdout: libpq x86_64 13.5-1.el8 rhel-8-for-x86_64-appstream-rpms 198 k 2023-07-14T19:04:49.499 INFO:teuthology.orchestra.run.smithi040.stdout: libquadmath x86_64 8.5.0-10.1.el8_6 rhel-8-for-x86_64-baseos-rpms 171 k 2023-07-14T19:04:49.499 INFO:teuthology.orchestra.run.smithi040.stdout: librabbitmq x86_64 0.9.0-3.el8 rhel-8-for-x86_64-baseos-rpms 47 k 2023-07-14T19:04:49.500 INFO:teuthology.orchestra.run.smithi040.stdout: libradosstriper1 x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 473 k 2023-07-14T19:04:49.500 INFO:teuthology.orchestra.run.smithi040.stdout: librdkafka x86_64 0.11.4-3.el8 rhel-8-for-x86_64-appstream-rpms 354 k 2023-07-14T19:04:49.500 INFO:teuthology.orchestra.run.smithi040.stdout: librgw2 x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 6.8 M 2023-07-14T19:04:49.500 INFO:teuthology.orchestra.run.smithi040.stdout: libselinux-devel x86_64 2.9-5.el8 rhel-8-for-x86_64-baseos-rpms 200 k 2023-07-14T19:04:49.500 INFO:teuthology.orchestra.run.smithi040.stdout: libsepol-devel x86_64 2.9-3.el8 rhel-8-for-x86_64-baseos-rpms 87 k 2023-07-14T19:04:49.500 INFO:teuthology.orchestra.run.smithi040.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2023-07-14T19:04:49.501 INFO:teuthology.orchestra.run.smithi040.stdout: libverto-devel x86_64 0.3.0-5.el8 rhel-8-for-x86_64-baseos-rpms 18 k 2023-07-14T19:04:49.501 INFO:teuthology.orchestra.run.smithi040.stdout: lttng-ust x86_64 2.8.1-11.el8 rhel-8-for-x86_64-appstream-rpms 259 k 2023-07-14T19:04:49.501 INFO:teuthology.orchestra.run.smithi040.stdout: oniguruma x86_64 6.8.2-2.el8 rhel-8-for-x86_64-appstream-rpms 187 k 2023-07-14T19:04:49.501 INFO:teuthology.orchestra.run.smithi040.stdout: openblas x86_64 0.3.15-3.el8 rhel-8-for-x86_64-appstream-rpms 4.7 M 2023-07-14T19:04:49.501 INFO:teuthology.orchestra.run.smithi040.stdout: openblas-threads x86_64 0.3.15-3.el8 rhel-8-for-x86_64-appstream-rpms 4.9 M 2023-07-14T19:04:49.501 INFO:teuthology.orchestra.run.smithi040.stdout: pcre2-devel x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 605 k 2023-07-14T19:04:49.501 INFO:teuthology.orchestra.run.smithi040.stdout: pcre2-utf16 x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 229 k 2023-07-14T19:04:49.501 INFO:teuthology.orchestra.run.smithi040.stdout: pcre2-utf32 x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 220 k 2023-07-14T19:04:49.502 INFO:teuthology.orchestra.run.smithi040.stdout: python3-asyncssh noarch 2.7.0-2.el8 epel 476 k 2023-07-14T19:04:49.502 INFO:teuthology.orchestra.run.smithi040.stdout: python3-babel noarch 2.5.1-7.el8 rhel-8-for-x86_64-appstream-rpms 4.8 M 2023-07-14T19:04:49.502 INFO:teuthology.orchestra.run.smithi040.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2023-07-14T19:04:49.502 INFO:teuthology.orchestra.run.smithi040.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2023-07-14T19:04:49.502 INFO:teuthology.orchestra.run.smithi040.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2023-07-14T19:04:49.503 INFO:teuthology.orchestra.run.smithi040.stdout: python3-ceph-argparse x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 45 k 2023-07-14T19:04:49.503 INFO:teuthology.orchestra.run.smithi040.stdout: python3-ceph-common x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 122 k 2023-07-14T19:04:49.503 INFO:teuthology.orchestra.run.smithi040.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2023-07-14T19:04:49.503 INFO:teuthology.orchestra.run.smithi040.stdout: python3-cffi x86_64 1.11.5-5.el8 rhel-8-for-x86_64-baseos-rpms 238 k 2023-07-14T19:04:49.503 INFO:teuthology.orchestra.run.smithi040.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2023-07-14T19:04:49.503 INFO:teuthology.orchestra.run.smithi040.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2023-07-14T19:04:49.504 INFO:teuthology.orchestra.run.smithi040.stdout: python3-cryptography x86_64 3.2.1-5.el8 rhel-8-for-x86_64-baseos-rpms 559 k 2023-07-14T19:04:49.504 INFO:teuthology.orchestra.run.smithi040.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2023-07-14T19:04:49.504 INFO:teuthology.orchestra.run.smithi040.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2023-07-14T19:04:49.504 INFO:teuthology.orchestra.run.smithi040.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2023-07-14T19:04:49.504 INFO:teuthology.orchestra.run.smithi040.stdout: python3-jinja2 noarch 2.10.1-3.el8 rhel-8-for-x86_64-appstream-rpms 538 k 2023-07-14T19:04:49.504 INFO:teuthology.orchestra.run.smithi040.stdout: python3-jsonpatch noarch 1.21-2.el8 rhel-8-for-x86_64-appstream-rpms 27 k 2023-07-14T19:04:49.504 INFO:teuthology.orchestra.run.smithi040.stdout: python3-jsonpointer noarch 1.10-11.el8 rhel-8-for-x86_64-appstream-rpms 20 k 2023-07-14T19:04:49.504 INFO:teuthology.orchestra.run.smithi040.stdout: python3-jwt noarch 1.6.1-2.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2023-07-14T19:04:49.505 INFO:teuthology.orchestra.run.smithi040.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2023-07-14T19:04:49.505 INFO:teuthology.orchestra.run.smithi040.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2023-07-14T19:04:49.505 INFO:teuthology.orchestra.run.smithi040.stdout: python3-mako noarch 1.0.6-13.el8 rhel-8-for-x86_64-appstream-rpms 157 k 2023-07-14T19:04:49.505 INFO:teuthology.orchestra.run.smithi040.stdout: python3-markupsafe x86_64 0.23-19.el8 rhel-8-for-x86_64-appstream-rpms 39 k 2023-07-14T19:04:49.505 INFO:teuthology.orchestra.run.smithi040.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2023-07-14T19:04:49.505 INFO:teuthology.orchestra.run.smithi040.stdout: python3-natsort noarch 7.1.1-2.el8 epel 60 k 2023-07-14T19:04:49.505 INFO:teuthology.orchestra.run.smithi040.stdout: python3-numpy x86_64 1:1.14.3-10.el8 rhel-8-for-x86_64-appstream-rpms 3.7 M 2023-07-14T19:04:49.506 INFO:teuthology.orchestra.run.smithi040.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 rhel-8-for-x86_64-appstream-rpms 225 k 2023-07-14T19:04:49.506 INFO:teuthology.orchestra.run.smithi040.stdout: python3-oauthlib noarch 2.1.0-1.el8 rhel-8-for-x86_64-baseos-rpms 155 k 2023-07-14T19:04:49.506 INFO:teuthology.orchestra.run.smithi040.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2023-07-14T19:04:49.506 INFO:teuthology.orchestra.run.smithi040.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2023-07-14T19:04:49.507 INFO:teuthology.orchestra.run.smithi040.stdout: python3-prettytable noarch 0.7.2-14.el8 rhel-8-for-x86_64-appstream-rpms 44 k 2023-07-14T19:04:49.507 INFO:teuthology.orchestra.run.smithi040.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 rhel-8-for-x86_64-appstream-rpms 103 k 2023-07-14T19:04:49.507 INFO:teuthology.orchestra.run.smithi040.stdout: python3-pyasn1 noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 126 k 2023-07-14T19:04:49.507 INFO:teuthology.orchestra.run.smithi040.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 110 k 2023-07-14T19:04:49.507 INFO:teuthology.orchestra.run.smithi040.stdout: python3-pycparser noarch 2.14-14.el8 rhel-8-for-x86_64-baseos-rpms 109 k 2023-07-14T19:04:49.508 INFO:teuthology.orchestra.run.smithi040.stdout: python3-pytz noarch 2017.2-9.el8 rhel-8-for-x86_64-appstream-rpms 54 k 2023-07-14T19:04:49.508 INFO:teuthology.orchestra.run.smithi040.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2023-07-14T19:04:49.508 INFO:teuthology.orchestra.run.smithi040.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2023-07-14T19:04:49.508 INFO:teuthology.orchestra.run.smithi040.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2023-07-14T19:04:49.508 INFO:teuthology.orchestra.run.smithi040.stdout: python3-rsa noarch 4.8-1.el8 epel 61 k 2023-07-14T19:04:49.508 INFO:teuthology.orchestra.run.smithi040.stdout: python3-scipy x86_64 1.0.0-21.module+el8.5.0+10916+41bd434d rhel-8-for-x86_64-appstream-rpms 14 M 2023-07-14T19:04:49.508 INFO:teuthology.orchestra.run.smithi040.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2023-07-14T19:04:49.509 INFO:teuthology.orchestra.run.smithi040.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2023-07-14T19:04:49.509 INFO:teuthology.orchestra.run.smithi040.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2023-07-14T19:04:49.509 INFO:teuthology.orchestra.run.smithi040.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2023-07-14T19:04:49.509 INFO:teuthology.orchestra.run.smithi040.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2023-07-14T19:04:49.509 INFO:teuthology.orchestra.run.smithi040.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2023-07-14T19:04:49.509 INFO:teuthology.orchestra.run.smithi040.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2023-07-14T19:04:49.509 INFO:teuthology.orchestra.run.smithi040.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2023-07-14T19:04:49.509 INFO:teuthology.orchestra.run.smithi040.stdout: python3-werkzeug noarch 0.12.2-4.el8 rhel-8-for-x86_64-appstream-rpms 457 k 2023-07-14T19:04:49.510 INFO:teuthology.orchestra.run.smithi040.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2023-07-14T19:04:49.510 INFO:teuthology.orchestra.run.smithi040.stdout: socat x86_64 1.7.4.1-1.el8 rhel-8-for-x86_64-appstream-rpms 323 k 2023-07-14T19:04:49.510 INFO:teuthology.orchestra.run.smithi040.stdout: thrift x86_64 0.13.0-2.el8 epel 1.7 M 2023-07-14T19:04:49.510 INFO:teuthology.orchestra.run.smithi040.stdout: uuid x86_64 1.6.2-43.el8 rhel-8-for-x86_64-appstream-rpms 64 k 2023-07-14T19:04:49.510 INFO:teuthology.orchestra.run.smithi040.stdout: xmlstarlet x86_64 1.6.1-20.el8 lab-extras 70 k 2023-07-14T19:04:49.510 INFO:teuthology.orchestra.run.smithi040.stdout: zlib-devel x86_64 1.2.11-18.el8_5 rhel-8-for-x86_64-baseos-rpms 58 k 2023-07-14T19:04:49.510 INFO:teuthology.orchestra.run.smithi040.stdout:Enabling module streams: 2023-07-14T19:04:49.510 INFO:teuthology.orchestra.run.smithi040.stdout: postgresql 10 2023-07-14T19:04:49.511 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-07-14T19:04:49.511 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction Summary 2023-07-14T19:04:49.511 INFO:teuthology.orchestra.run.smithi040.stdout:=========================================================================================================================== 2023-07-14T19:04:49.511 INFO:teuthology.orchestra.run.smithi040.stdout:Install 128 Packages 2023-07-14T19:04:49.511 INFO:teuthology.orchestra.run.smithi040.stdout:Upgrade 3 Packages 2023-07-14T19:04:49.511 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-07-14T19:04:49.512 INFO:teuthology.orchestra.run.smithi040.stdout:Total download size: 206 M 2023-07-14T19:04:49.512 INFO:teuthology.orchestra.run.smithi040.stdout:Downloading Packages: 2023-07-14T19:04:50.500 INFO:teuthology.orchestra.run.smithi191.stdout:(6/131): ceph-base-18.0.0-4940.g81c9387d.el8.x8 660 kB/s | 5.4 MB 00:08 2023-07-14T19:04:50.638 INFO:teuthology.orchestra.run.smithi040.stdout:(1/131): ceph-18.0.0-4940.g81c9387d.el8.x86_64. 13 kB/s | 6.4 kB 00:00 2023-07-14T19:04:52.330 INFO:teuthology.orchestra.run.smithi040.stdout:(2/131): ceph-fuse-18.0.0-4940.g81c9387d.el8.x8 503 kB/s | 850 kB 00:01 2023-07-14T19:04:52.345 INFO:teuthology.orchestra.run.smithi130.stdout:(10/131): ceph-test-18.0.0-4940.g81c9387d.el8.x 9.3 MB/s | 48 MB 00:05 2023-07-14T19:04:52.495 INFO:teuthology.orchestra.run.smithi130.stdout:(11/131): cephfs-mirror-18.0.0-4940.g81c9387d.e 1.7 MB/s | 260 kB 00:00 2023-07-14T19:04:52.572 INFO:teuthology.orchestra.run.smithi040.stdout:(3/131): ceph-immutable-object-cache-18.0.0-494 692 kB/s | 167 kB 00:00 2023-07-14T19:04:52.645 INFO:teuthology.orchestra.run.smithi130.stdout:(12/131): libcephfs-devel-18.0.0-4940.g81c9387d 207 kB/s | 31 kB 00:00 2023-07-14T19:04:52.920 INFO:teuthology.orchestra.run.smithi130.stdout:(13/131): libcephfs2-18.0.0-4940.g81c9387d.el8. 2.7 MB/s | 772 kB 00:00 2023-07-14T19:04:53.071 INFO:teuthology.orchestra.run.smithi130.stdout:(14/131): libcephsqlite-18.0.0-4940.g81c9387d.e 903 kB/s | 135 kB 00:00 2023-07-14T19:04:53.221 INFO:teuthology.orchestra.run.smithi130.stdout:(15/131): librados-devel-18.0.0-4940.g81c9387d. 928 kB/s | 139 kB 00:00 2023-07-14T19:04:53.363 INFO:teuthology.orchestra.run.smithi130.stdout:(16/131): libradosstriper1-18.0.0-4940.g81c9387 3.3 MB/s | 473 kB 00:00 2023-07-14T19:04:53.391 INFO:teuthology.orchestra.run.smithi191.stdout:(7/131): ceph-mon-18.0.0-4940.g81c9387d.el8.x86 955 kB/s | 4.6 MB 00:04 2023-07-14T19:04:54.230 INFO:teuthology.orchestra.run.smithi130.stdout:(17/131): ceph-common-18.0.0-4940.g81c9387d.el8 1.7 MB/s | 22 MB 00:13 2023-07-14T19:04:54.264 INFO:teuthology.orchestra.run.smithi040.stdout:(4/131): ceph-base-18.0.0-4940.g81c9387d.el8.x8 1.3 MB/s | 5.4 MB 00:04 2023-07-14T19:04:54.314 INFO:teuthology.orchestra.run.smithi130.stdout:(18/131): librgw2-18.0.0-4940.g81c9387d.el8.x86 7.1 MB/s | 6.8 MB 00:00 2023-07-14T19:04:54.389 INFO:teuthology.orchestra.run.smithi130.stdout:(19/131): python3-ceph-argparse-18.0.0-4940.g81 282 kB/s | 45 kB 00:00 2023-07-14T19:04:54.473 INFO:teuthology.orchestra.run.smithi130.stdout:(20/131): python3-ceph-common-18.0.0-4940.g81c9 773 kB/s | 122 kB 00:00 2023-07-14T19:04:54.564 INFO:teuthology.orchestra.run.smithi040.stdout:(5/131): ceph-mds-18.0.0-4940.g81c9387d.el8.x86 1.2 MB/s | 2.3 MB 00:01 2023-07-14T19:04:54.681 INFO:teuthology.orchestra.run.smithi130.stdout:(21/131): python3-rados-18.0.0-4940.g81c9387d.e 1.7 MB/s | 371 kB 00:00 2023-07-14T19:04:54.707 INFO:teuthology.orchestra.run.smithi130.stdout:(22/131): python3-cephfs-18.0.0-4940.g81c9387d. 620 kB/s | 197 kB 00:00 2023-07-14T19:04:54.840 INFO:teuthology.orchestra.run.smithi130.stdout:(23/131): python3-rbd-18.0.0-4940.g81c9387d.el8 2.2 MB/s | 364 kB 00:00 2023-07-14T19:04:54.865 INFO:teuthology.orchestra.run.smithi130.stdout:(24/131): python3-rgw-18.0.0-4940.g81c9387d.el8 673 kB/s | 107 kB 00:00 2023-07-14T19:04:54.873 INFO:teuthology.orchestra.run.smithi040.stdout:(6/131): ceph-mgr-18.0.0-4940.g81c9387d.el8.x86 2.5 MB/s | 1.6 MB 00:00 2023-07-14T19:04:54.982 INFO:teuthology.orchestra.run.smithi130.stdout:(25/131): rbd-fuse-18.0.0-4940.g81c9387d.el8.x8 660 kB/s | 94 kB 00:00 2023-07-14T19:04:55.133 INFO:teuthology.orchestra.run.smithi130.stdout:(26/131): rbd-nbd-18.0.0-4940.g81c9387d.el8.x86 1.2 MB/s | 179 kB 00:00 2023-07-14T19:04:55.300 INFO:teuthology.orchestra.run.smithi130.stdout:(27/131): ceph-grafana-dashboards-18.0.0-4940.g 138 kB/s | 23 kB 00:00 2023-07-14T19:04:55.467 INFO:teuthology.orchestra.run.smithi130.stdout:(28/131): ceph-mgr-cephadm-18.0.0-4940.g81c9387 776 kB/s | 129 kB 00:00 2023-07-14T19:04:55.942 INFO:teuthology.orchestra.run.smithi130.stdout:(29/131): ceph-mgr-dashboard-18.0.0-4940.g81c93 3.6 MB/s | 1.7 MB 00:00 2023-07-14T19:04:56.723 INFO:teuthology.orchestra.run.smithi040.stdout:(7/131): ceph-mon-18.0.0-4940.g81c9387d.el8.x86 2.1 MB/s | 4.6 MB 00:02 2023-07-14T19:04:56.734 INFO:teuthology.orchestra.run.smithi130.stdout:(30/131): ceph-osd-18.0.0-4940.g81c9387d.el8.x8 1.4 MB/s | 18 MB 00:12 2023-07-14T19:04:57.084 INFO:teuthology.orchestra.run.smithi130.stdout:(31/131): ceph-mgr-modules-core-18.0.0-4940.g81 689 kB/s | 241 kB 00:00 2023-07-14T19:04:57.209 INFO:teuthology.orchestra.run.smithi130.stdout:(32/131): ceph-mgr-diskprediction-local-18.0.0- 5.8 MB/s | 7.4 MB 00:01 2023-07-14T19:04:57.259 INFO:teuthology.orchestra.run.smithi130.stdout:(33/131): ceph-mgr-rook-18.0.0-4940.g81c9387d.e 270 kB/s | 47 kB 00:00 2023-07-14T19:04:57.376 INFO:teuthology.orchestra.run.smithi130.stdout:(34/131): ceph-prometheus-alerts-18.0.0-4940.g8 87 kB/s | 15 kB 00:00 2023-07-14T19:04:57.585 INFO:teuthology.orchestra.run.smithi130.stdout:(35/131): ceph-volume-18.0.0-4940.g81c9387d.el8 794 kB/s | 258 kB 00:00 2023-07-14T19:04:57.735 INFO:teuthology.orchestra.run.smithi130.stdout:(36/131): cephfs-top-18.0.0-4940.g81c9387d.el8. 120 kB/s | 18 kB 00:00 2023-07-14T19:04:58.327 INFO:teuthology.orchestra.run.smithi130.stdout:(37/131): rbd-mirror-18.0.0-4940.g81c9387d.el8. 1.0 MB/s | 3.5 MB 00:03 2023-07-14T19:04:58.644 INFO:teuthology.orchestra.run.smithi130.stdout:(38/131): gperftools-libs-2.7-9.el8.x86_64.rpm 337 kB/s | 306 kB 00:00 2023-07-14T19:04:58.736 INFO:teuthology.orchestra.run.smithi130.stdout:(39/131): cephadm-18.0.0-4940.g81c9387d.el8.noa 122 kB/s | 165 kB 00:01 2023-07-14T19:04:58.878 INFO:teuthology.orchestra.run.smithi130.stdout:(40/131): liboath-2.6.2-3.el8.x86_64.rpm 107 kB/s | 59 kB 00:00 2023-07-14T19:04:58.903 INFO:teuthology.orchestra.run.smithi130.stdout:(41/131): libunwind-1.3.1-3.el8.x86_64.rpm 290 kB/s | 75 kB 00:00 2023-07-14T19:04:59.029 INFO:teuthology.orchestra.run.smithi130.stdout:(42/131): python3-bcrypt-3.1.6-2.el8.1.x86_64.r 299 kB/s | 44 kB 00:00 2023-07-14T19:04:59.162 INFO:teuthology.orchestra.run.smithi130.stdout:(43/131): python3-cachetools-3.1.1-4.el8.noarch 248 kB/s | 33 kB 00:00 2023-07-14T19:04:59.258 INFO:teuthology.orchestra.run.smithi191.stdout:(8/131): ceph-radosgw-18.0.0-4940.g81c9387d.el8 2.3 MB/s | 14 MB 00:05 2023-07-14T19:04:59.279 INFO:teuthology.orchestra.run.smithi130.stdout:(44/131): python3-beautifulsoup4-4.6.3-2.el8.1. 493 kB/s | 185 kB 00:00 2023-07-14T19:04:59.305 INFO:teuthology.orchestra.run.smithi130.stdout:(45/131): python3-certifi-2018.10.15-7.el8.noar 115 kB/s | 16 kB 00:00 2023-07-14T19:04:59.391 INFO:teuthology.orchestra.run.smithi191.stdout:(9/131): ceph-selinux-18.0.0-4940.g81c9387d.el8 183 kB/s | 24 kB 00:00 2023-07-14T19:04:59.440 INFO:teuthology.orchestra.run.smithi040.stdout:(8/131): ceph-radosgw-18.0.0-4940.g81c9387d.el8 5.0 MB/s | 14 MB 00:02 2023-07-14T19:04:59.590 INFO:teuthology.orchestra.run.smithi040.stdout:(9/131): ceph-selinux-18.0.0-4940.g81c9387d.el8 162 kB/s | 24 kB 00:00 2023-07-14T19:04:59.877 INFO:teuthology.orchestra.run.smithi130.stdout:(46/131): python3-cheroot-8.5.2-1.el8.noarch.rp 289 kB/s | 173 kB 00:00 2023-07-14T19:04:59.948 INFO:teuthology.orchestra.run.smithi040.stdout:(10/131): ceph-osd-18.0.0-4940.g81c9387d.el8.x8 3.5 MB/s | 18 MB 00:05 2023-07-14T19:05:00.031 INFO:teuthology.orchestra.run.smithi130.stdout:(47/131): python3-asyncssh-2.7.0-2.el8.noarch.r 368 kB/s | 476 kB 00:01 2023-07-14T19:05:00.056 INFO:teuthology.orchestra.run.smithi130.stdout:(48/131): python3-cherrypy-18.4.0-1.el8.noarch. 512 kB/s | 384 kB 00:00 2023-07-14T19:05:00.124 INFO:teuthology.orchestra.run.smithi040.stdout:(11/131): cephfs-mirror-18.0.0-4940.g81c9387d.e 1.5 MB/s | 260 kB 00:00 2023-07-14T19:05:00.164 INFO:teuthology.orchestra.run.smithi130.stdout:(49/131): python3-jaraco-6.2-6.el8.noarch.rpm 83 kB/s | 11 kB 00:00 2023-07-14T19:05:00.190 INFO:teuthology.orchestra.run.smithi130.stdout:(50/131): python3-jaraco-functools-2.0-4.el8.no 131 kB/s | 18 kB 00:00 2023-07-14T19:05:00.282 INFO:teuthology.orchestra.run.smithi040.stdout:(12/131): libcephfs-devel-18.0.0-4940.g81c9387d 196 kB/s | 31 kB 00:00 2023-07-14T19:05:00.348 INFO:teuthology.orchestra.run.smithi130.stdout:(51/131): python3-logutils-0.3.5-11.el8.noarch. 308 kB/s | 49 kB 00:00 2023-07-14T19:05:00.507 INFO:teuthology.orchestra.run.smithi130.stdout:(52/131): python3-more-itertools-7.2.0-3.el8.no 371 kB/s | 59 kB 00:00 2023-07-14T19:05:00.541 INFO:teuthology.orchestra.run.smithi130.stdout:(53/131): python3-kubernetes-11.0.0-6.el8.noarc 3.1 MB/s | 1.2 MB 00:00 2023-07-14T19:05:00.607 INFO:teuthology.orchestra.run.smithi040.stdout:(13/131): libcephfs2-18.0.0-4940.g81c9387d.el8. 2.3 MB/s | 772 kB 00:00 2023-07-14T19:05:00.641 INFO:teuthology.orchestra.run.smithi130.stdout:(54/131): python3-natsort-7.1.1-2.el8.noarch.rp 451 kB/s | 60 kB 00:00 2023-07-14T19:05:00.674 INFO:teuthology.orchestra.run.smithi130.stdout:(55/131): python3-pecan-1.3.2-9.el8.noarch.rpm 2.1 MB/s | 283 kB 00:00 2023-07-14T19:05:00.733 INFO:teuthology.orchestra.run.smithi040.stdout:(14/131): libcephsqlite-18.0.0-4940.g81c9387d.e 1.1 MB/s | 135 kB 00:00 2023-07-14T19:05:00.762 INFO:teuthology.orchestra.run.smithi130.stdout:(56/131): python3-portend-2.6-1.el8.noarch.rpm 129 kB/s | 16 kB 00:00 2023-07-14T19:05:00.792 INFO:teuthology.orchestra.run.smithi130.stdout:(57/131): python3-google-auth-1.1.1-10.el8.noar 89 kB/s | 81 kB 00:00 2023-07-14T19:05:00.875 INFO:teuthology.orchestra.run.smithi191.stdout:(10/131): ceph-common-18.0.0-4940.g81c9387d.el8 1.2 MB/s | 22 MB 00:18 2023-07-14T19:05:00.883 INFO:teuthology.orchestra.run.smithi040.stdout:(15/131): librados-devel-18.0.0-4940.g81c9387d. 927 kB/s | 139 kB 00:00 2023-07-14T19:05:00.975 INFO:teuthology.orchestra.run.smithi130.stdout:(58/131): python3-rsa-4.8-1.el8.noarch.rpm 335 kB/s | 61 kB 00:00 2023-07-14T19:05:01.084 INFO:teuthology.orchestra.run.smithi130.stdout:(59/131): python3-simplegeneric-0.8.1-17.el8.no 173 kB/s | 19 kB 00:00 2023-07-14T19:05:01.126 INFO:teuthology.orchestra.run.smithi130.stdout:(60/131): python3-routes-2.4.1-12.el8.noarch.rp 540 kB/s | 196 kB 00:00 2023-07-14T19:05:01.151 INFO:teuthology.orchestra.run.smithi130.stdout:(61/131): python3-repoze-lru-0.7-6.el8.noarch.r 71 kB/s | 34 kB 00:00 2023-07-14T19:05:01.158 INFO:teuthology.orchestra.run.smithi191.stdout:(11/131): cephfs-mirror-18.0.0-4940.g81c9387d.e 919 kB/s | 260 kB 00:00 2023-07-14T19:05:01.201 INFO:teuthology.orchestra.run.smithi130.stdout:(62/131): python3-singledispatch-3.4.0.3-18.el8 204 kB/s | 24 kB 00:00 2023-07-14T19:05:01.226 INFO:teuthology.orchestra.run.smithi130.stdout:(63/131): python3-tempora-1.14.1-5.el8.noarch.r 291 kB/s | 29 kB 00:00 2023-07-14T19:05:01.252 INFO:teuthology.orchestra.run.smithi130.stdout:(64/131): python3-trustme-0.6.0-4.el8.noarch.rp 274 kB/s | 27 kB 00:00 2023-07-14T19:05:01.275 INFO:teuthology.orchestra.run.smithi191.stdout:(12/131): libcephfs-devel-18.0.0-4940.g81c9387d 266 kB/s | 31 kB 00:00 2023-07-14T19:05:01.361 INFO:teuthology.orchestra.run.smithi130.stdout:(65/131): python3-websocket-client-0.56.0-5.el8 566 kB/s | 61 kB 00:00 2023-07-14T19:05:01.486 INFO:teuthology.orchestra.run.smithi130.stdout:(66/131): python3-webtest-2.0.33-1.el8.noarch.r 684 kB/s | 85 kB 00:00 2023-07-14T19:05:01.636 INFO:teuthology.orchestra.run.smithi130.stdout:(67/131): python3-zc-lockfile-2.0-2.el8.noarch. 153 kB/s | 23 kB 00:00 2023-07-14T19:05:01.725 INFO:teuthology.orchestra.run.smithi191.stdout:(13/131): libcephfs2-18.0.0-4940.g81c9387d.el8. 1.7 MB/s | 772 kB 00:00 2023-07-14T19:05:01.736 INFO:teuthology.orchestra.run.smithi130.stdout:(68/131): python3-waitress-1.4.3-1.el8.noarch.r 477 kB/s | 255 kB 00:00 2023-07-14T19:05:01.762 INFO:teuthology.orchestra.run.smithi130.stdout:(69/131): python3-webob-1.8.5-1.el8.1.noarch.rp 470 kB/s | 251 kB 00:00 2023-07-14T19:05:01.787 INFO:teuthology.orchestra.run.smithi130.stdout:(70/131): xmlstarlet-1.6.1-20.el8.x86_64.rpm 1.4 MB/s | 70 kB 00:00 2023-07-14T19:05:01.946 INFO:teuthology.orchestra.run.smithi130.stdout:(71/131): python3-pyasn1-modules-0.3.7-6.el8.no 722 kB/s | 110 kB 00:00 2023-07-14T19:05:01.988 INFO:teuthology.orchestra.run.smithi130.stdout:(72/131): thrift-0.13.0-2.el8.x86_64.rpm 4.9 MB/s | 1.7 MB 00:00 2023-07-14T19:05:01.992 INFO:teuthology.orchestra.run.smithi191.stdout:(14/131): ceph-osd-18.0.0-4940.g81c9387d.el8.x8 1.5 MB/s | 18 MB 00:11 2023-07-14T19:05:02.017 INFO:teuthology.orchestra.run.smithi191.stdout:(15/131): libcephsqlite-18.0.0-4940.g81c9387d.e 464 kB/s | 135 kB 00:00 2023-07-14T19:05:02.113 INFO:teuthology.orchestra.run.smithi130.stdout:(73/131): python3-pytz-2017.2-9.el8.noarch.rpm 324 kB/s | 54 kB 00:00 2023-07-14T19:05:02.126 INFO:teuthology.orchestra.run.smithi191.stdout:(16/131): librados-devel-18.0.0-4940.g81c9387d. 1.0 MB/s | 139 kB 00:00 2023-07-14T19:05:02.280 INFO:teuthology.orchestra.run.smithi130.stdout:(74/131): python3-scipy-1.0.0-21.module+el8.5.0 26 MB/s | 14 MB 00:00 2023-07-14T19:05:02.305 INFO:teuthology.orchestra.run.smithi130.stdout:(75/131): lttng-ust-2.8.1-11.el8.x86_64.rpm 817 kB/s | 259 kB 00:00 2023-07-14T19:05:02.331 INFO:teuthology.orchestra.run.smithi130.stdout:(76/131): socat-1.7.4.1-1.el8.x86_64.rpm 1.5 MB/s | 323 kB 00:00 2023-07-14T19:05:02.356 INFO:teuthology.orchestra.run.smithi130.stdout:(77/131): python3-markupsafe-0.23-19.el8.x86_64 516 kB/s | 39 kB 00:00 2023-07-14T19:05:02.400 INFO:teuthology.orchestra.run.smithi040.stdout:(16/131): libradosstriper1-18.0.0-4940.g81c9387 312 kB/s | 473 kB 00:01 2023-07-14T19:05:02.476 INFO:teuthology.orchestra.run.smithi191.stdout:(17/131): libradosstriper1-18.0.0-4940.g81c9387 1.0 MB/s | 473 kB 00:00 2023-07-14T19:05:02.489 INFO:teuthology.orchestra.run.smithi130.stdout:(78/131): python3-jinja2-2.10.1-3.el8.noarch.rp 3.3 MB/s | 538 kB 00:00 2023-07-14T19:05:02.531 INFO:teuthology.orchestra.run.smithi130.stdout:(79/131): bison-3.0.4-10.el8.x86_64.rpm 3.0 MB/s | 688 kB 00:00 2023-07-14T19:05:02.573 INFO:teuthology.orchestra.run.smithi130.stdout:(80/131): python3-mako-1.0.6-13.el8.noarch.rpm 722 kB/s | 157 kB 00:00 2023-07-14T19:05:02.618 INFO:teuthology.orchestra.run.smithi191.stdout:(18/131): python3-ceph-argparse-18.0.0-4940.g81 315 kB/s | 45 kB 00:00 2023-07-14T19:05:02.682 INFO:teuthology.orchestra.run.smithi130.stdout:(81/131): automake-1.16.1-7.el8.noarch.rpm 4.6 MB/s | 713 kB 00:00 2023-07-14T19:05:02.707 INFO:teuthology.orchestra.run.smithi130.stdout:(82/131): oniguruma-6.8.2-2.el8.x86_64.rpm 861 kB/s | 187 kB 00:00 2023-07-14T19:05:02.769 INFO:teuthology.orchestra.run.smithi191.stdout:(19/131): python3-ceph-common-18.0.0-4940.g81c9 817 kB/s | 122 kB 00:00 2023-07-14T19:05:02.774 INFO:teuthology.orchestra.run.smithi130.stdout:(83/131): python3-jsonpatch-1.21-2.el8.noarch.r 289 kB/s | 27 kB 00:00 2023-07-14T19:05:02.866 INFO:teuthology.orchestra.run.smithi130.stdout:(84/131): python3-babel-2.5.1-7.el8.noarch.rpm 16 MB/s | 4.8 MB 00:00 2023-07-14T19:05:02.900 INFO:teuthology.orchestra.run.smithi130.stdout:(85/131): flex-2.6.1-9.el8.x86_64.rpm 1.6 MB/s | 320 kB 00:00 2023-07-14T19:05:02.925 INFO:teuthology.orchestra.run.smithi130.stdout:(86/131): python3-prettytable-0.7.2-14.el8.noar 292 kB/s | 44 kB 00:00 2023-07-14T19:05:03.011 INFO:teuthology.orchestra.run.smithi191.stdout:(20/131): python3-cephfs-18.0.0-4940.g81c9387d. 814 kB/s | 197 kB 00:00 2023-07-14T19:05:03.067 INFO:teuthology.orchestra.run.smithi130.stdout:(87/131): python3-numpy-1.14.3-10.el8.x86_64.rp 19 MB/s | 3.7 MB 00:00 2023-07-14T19:05:03.092 INFO:teuthology.orchestra.run.smithi130.stdout:(88/131): python3-numpy-f2py-1.14.3-10.el8.x86_ 1.3 MB/s | 225 kB 00:00 2023-07-14T19:05:03.117 INFO:teuthology.orchestra.run.smithi130.stdout:(89/131): python3-werkzeug-0.12.2-4.el8.noarch. 2.1 MB/s | 457 kB 00:00 2023-07-14T19:05:03.193 INFO:teuthology.orchestra.run.smithi130.stdout:(90/131): uuid-1.6.2-43.el8.x86_64.rpm 509 kB/s | 64 kB 00:00 2023-07-14T19:05:03.310 INFO:teuthology.orchestra.run.smithi130.stdout:(91/131): python3-pyasn1-0.3.7-6.el8.noarch.rpm 582 kB/s | 126 kB 00:00 2023-07-14T19:05:03.377 INFO:teuthology.orchestra.run.smithi130.stdout:(92/131): python3-jsonpointer-1.10-11.el8.noarc 77 kB/s | 20 kB 00:00 2023-07-14T19:05:03.427 INFO:teuthology.orchestra.run.smithi191.stdout:(21/131): python3-rados-18.0.0-4940.g81c9387d.e 891 kB/s | 371 kB 00:00 2023-07-14T19:05:03.435 INFO:teuthology.orchestra.run.smithi130.stdout:(93/131): python3-pyOpenSSL-19.0.0-1.el8.noarch 428 kB/s | 103 kB 00:00 2023-07-14T19:05:03.477 INFO:teuthology.orchestra.run.smithi130.stdout:(94/131): libtool-2.4.6-25.el8.x86_64.rpm 4.1 MB/s | 709 kB 00:00 2023-07-14T19:05:03.511 INFO:teuthology.orchestra.run.smithi130.stdout:(95/131): libpq-13.5-1.el8.x86_64.rpm 2.6 MB/s | 198 kB 00:00 2023-07-14T19:05:03.570 INFO:teuthology.orchestra.run.smithi130.stdout:(96/131): jq-1.6-3.el8.x86_64.rpm 3.4 MB/s | 202 kB 00:00 2023-07-14T19:05:03.645 INFO:teuthology.orchestra.run.smithi130.stdout:(97/131): openblas-threads-0.3.15-3.el8.x86_64. 18 MB/s | 4.9 MB 00:00 2023-07-14T19:05:03.713 INFO:teuthology.orchestra.run.smithi130.stdout:(98/131): librdkafka-0.11.4-3.el8.x86_64.rpm 2.4 MB/s | 354 kB 00:00 2023-07-14T19:05:03.811 INFO:teuthology.orchestra.run.smithi191.stdout:(22/131): python3-rbd-18.0.0-4940.g81c9387d.el8 950 kB/s | 364 kB 00:00 2023-07-14T19:05:03.854 INFO:teuthology.orchestra.run.smithi130.stdout:(99/131): openblas-0.3.15-3.el8.x86_64.rpm 13 MB/s | 4.7 MB 00:00 2023-07-14T19:05:03.900 INFO:teuthology.orchestra.run.smithi040.stdout:(17/131): ceph-common-18.0.0-4940.g81c9387d.el8 1.6 MB/s | 22 MB 00:13 2023-07-14T19:05:03.904 INFO:teuthology.orchestra.run.smithi130.stdout:(100/131): postgresql-contrib-10.21-2.module+el 4.1 MB/s | 808 kB 00:00 2023-07-14T19:05:03.970 INFO:teuthology.orchestra.run.smithi191.stdout:(23/131): python3-rgw-18.0.0-4940.g81c9387d.el8 673 kB/s | 107 kB 00:00 2023-07-14T19:05:04.004 INFO:teuthology.orchestra.run.smithi130.stdout:(101/131): postgresql-server-10.21-2.module+el8 14 MB/s | 5.1 MB 00:00 2023-07-14T19:05:04.030 INFO:teuthology.orchestra.run.smithi130.stdout:(102/131): librabbitmq-0.9.0-3.el8.x86_64.rpm 373 kB/s | 47 kB 00:00 2023-07-14T19:05:04.050 INFO:teuthology.orchestra.run.smithi040.stdout:(18/131): python3-ceph-argparse-18.0.0-4940.g81 299 kB/s | 45 kB 00:00 2023-07-14T19:05:04.088 INFO:teuthology.orchestra.run.smithi130.stdout:(103/131): postgresql-10.21-2.module+el8.6.0+15 6.4 MB/s | 1.5 MB 00:00 2023-07-14T19:05:04.104 INFO:teuthology.orchestra.run.smithi191.stdout:(24/131): rbd-fuse-18.0.0-4940.g81c9387d.el8.x8 702 kB/s | 94 kB 00:00 2023-07-14T19:05:04.147 INFO:teuthology.orchestra.run.smithi130.stdout:(104/131): libselinux-devel-2.9-5.el8.x86_64.rp 1.4 MB/s | 200 kB 00:00 2023-07-14T19:05:04.172 INFO:teuthology.orchestra.run.smithi130.stdout:(105/131): python3-jwt-1.6.1-2.el8.noarch.rpm 300 kB/s | 43 kB 00:00 2023-07-14T19:05:04.218 INFO:teuthology.orchestra.run.smithi040.stdout:(19/131): python3-ceph-common-18.0.0-4940.g81c9 732 kB/s | 122 kB 00:00 2023-07-14T19:05:04.231 INFO:teuthology.orchestra.run.smithi130.stdout:(106/131): keyutils-libs-devel-1.5.10-9.el8.x86 577 kB/s | 48 kB 00:00 2023-07-14T19:05:04.289 INFO:teuthology.orchestra.run.smithi130.stdout:(107/131): pcre2-utf32-10.32-2.el8.x86_64.rpm 1.1 MB/s | 220 kB 00:00 2023-07-14T19:05:04.315 INFO:teuthology.orchestra.run.smithi130.stdout:(108/131): libsepol-devel-2.9-3.el8.x86_64.rpm 611 kB/s | 87 kB 00:00 2023-07-14T19:05:04.340 INFO:teuthology.orchestra.run.smithi130.stdout:(109/131): python3-requests-oauthlib-1.0.0-1.el 392 kB/s | 43 kB 00:00 2023-07-14T19:05:04.343 INFO:teuthology.orchestra.run.smithi040.stdout:(20/131): librgw2-18.0.0-4940.g81c9387d.el8.x86 3.5 MB/s | 6.8 MB 00:01 2023-07-14T19:05:04.368 INFO:teuthology.orchestra.run.smithi040.stdout:(21/131): python3-cephfs-18.0.0-4940.g81c9387d. 1.3 MB/s | 197 kB 00:00 2023-07-14T19:05:04.407 INFO:teuthology.orchestra.run.smithi130.stdout:(110/131): python3-pycparser-2.14-14.el8.noarch 930 kB/s | 109 kB 00:00 2023-07-14T19:05:04.432 INFO:teuthology.orchestra.run.smithi130.stdout:(111/131): pcre2-utf16-10.32-2.el8.x86_64.rpm 1.9 MB/s | 229 kB 00:00 2023-07-14T19:05:04.483 INFO:teuthology.orchestra.run.smithi130.stdout:(112/131): krb5-devel-1.18.2-14.el8.x86_64.rpm 3.8 MB/s | 560 kB 00:00 2023-07-14T19:05:04.502 INFO:teuthology.orchestra.run.smithi040.stdout:(22/131): python3-rados-18.0.0-4940.g81c9387d.e 2.3 MB/s | 371 kB 00:00 2023-07-14T19:05:04.575 INFO:teuthology.orchestra.run.smithi130.stdout:(113/131): libaio-devel-0.3.112-1.el8.x86_64.rp 112 kB/s | 19 kB 00:00 2023-07-14T19:05:04.600 INFO:teuthology.orchestra.run.smithi130.stdout:(114/131): python3-oauthlib-2.1.0-1.el8.noarch. 928 kB/s | 155 kB 00:00 2023-07-14T19:05:04.619 INFO:teuthology.orchestra.run.smithi040.stdout:(23/131): python3-rbd-18.0.0-4940.g81c9387d.el8 1.4 MB/s | 364 kB 00:00 2023-07-14T19:05:04.633 INFO:teuthology.orchestra.run.smithi130.stdout:(115/131): python3-cryptography-3.2.1-5.el8.x86 3.6 MB/s | 559 kB 00:00 2023-07-14T19:05:04.644 INFO:teuthology.orchestra.run.smithi040.stdout:(24/131): python3-rgw-18.0.0-4940.g81c9387d.el8 751 kB/s | 107 kB 00:00 2023-07-14T19:05:04.676 INFO:teuthology.orchestra.run.smithi130.stdout:(116/131): libverto-devel-0.3.0-5.el8.x86_64.rp 178 kB/s | 18 kB 00:00 2023-07-14T19:05:04.767 INFO:teuthology.orchestra.run.smithi130.stdout:(117/131): pcre2-devel-10.32-2.el8.x86_64.rpm 3.5 MB/s | 605 kB 00:00 2023-07-14T19:05:04.769 INFO:teuthology.orchestra.run.smithi040.stdout:(25/131): rbd-fuse-18.0.0-4940.g81c9387d.el8.x8 624 kB/s | 94 kB 00:00 2023-07-14T19:05:04.793 INFO:teuthology.orchestra.run.smithi130.stdout:(118/131): python3-cffi-1.11.5-5.el8.x86_64.rpm 1.5 MB/s | 238 kB 00:00 2023-07-14T19:05:04.860 INFO:teuthology.orchestra.run.smithi130.stdout:(119/131): openssl-devel-1.1.1k-7.el8_6.x86_64. 13 MB/s | 2.3 MB 00:00 2023-07-14T19:05:04.893 INFO:teuthology.orchestra.run.smithi130.stdout:(120/131): xfsprogs-devel-5.0.0-10.el8.x86_64.r 575 kB/s | 72 kB 00:00 2023-07-14T19:05:04.936 INFO:teuthology.orchestra.run.smithi040.stdout:(26/131): rbd-nbd-18.0.0-4940.g81c9387d.el8.x86 1.0 MB/s | 179 kB 00:00 2023-07-14T19:05:05.018 INFO:teuthology.orchestra.run.smithi130.stdout:(121/131): elfutils-libelf-devel-0.186-1.el8.x8 377 kB/s | 60 kB 00:00 2023-07-14T19:05:05.044 INFO:teuthology.orchestra.run.smithi130.stdout:(122/131): libuuid-devel-2.32.1-35.el8.x86_64.r 653 kB/s | 98 kB 00:00 2023-07-14T19:05:05.077 INFO:teuthology.orchestra.run.smithi130.stdout:(123/131): fuse-2.9.7-15.el8.x86_64.rpm 291 kB/s | 83 kB 00:00 2023-07-14T19:05:05.153 INFO:teuthology.orchestra.run.smithi130.stdout:(124/131): zlib-devel-1.2.11-18.el8_5.x86_64.rp 431 kB/s | 58 kB 00:00 2023-07-14T19:05:05.178 INFO:teuthology.orchestra.run.smithi130.stdout:(125/131): libcom_err-devel-1.45.6-4.el8.x86_64 288 kB/s | 39 kB 00:00 2023-07-14T19:05:05.212 INFO:teuthology.orchestra.run.smithi130.stdout:(126/131): libgfortran-8.5.0-10.1.el8_6.x86_64. 4.7 MB/s | 644 kB 00:00 2023-07-14T19:05:05.320 INFO:teuthology.orchestra.run.smithi130.stdout:(127/131): libquadmath-8.5.0-10.1.el8_6.x86_64. 1.0 MB/s | 171 kB 00:00 2023-07-14T19:05:05.389 INFO:teuthology.orchestra.run.smithi130.stdout:(128/131): NetworkManager-initscripts-updown-1. 652 kB/s | 138 kB 00:00 2023-07-14T19:05:05.395 INFO:teuthology.orchestra.run.smithi040.stdout:(27/131): ceph-test-18.0.0-4940.g81c9387d.el8.x 8.3 MB/s | 48 MB 00:05 2023-07-14T19:05:05.420 INFO:teuthology.orchestra.run.smithi040.stdout:(28/131): ceph-grafana-dashboards-18.0.0-4940.g 48 kB/s | 23 kB 00:00 2023-07-14T19:05:05.454 INFO:teuthology.orchestra.run.smithi130.stdout:(129/131): iproute-5.15.0-4.el8_6.1.x86_64.rpm 12 MB/s | 798 kB 00:00 2023-07-14T19:05:05.554 INFO:teuthology.orchestra.run.smithi040.stdout:(29/131): ceph-mgr-cephadm-18.0.0-4940.g81c9387 817 kB/s | 129 kB 00:00 2023-07-14T19:05:05.820 INFO:teuthology.orchestra.run.smithi040.stdout:(30/131): rbd-mirror-18.0.0-4940.g81c9387d.el8. 3.0 MB/s | 3.5 MB 00:01 2023-07-14T19:05:05.987 INFO:teuthology.orchestra.run.smithi040.stdout:(31/131): ceph-mgr-modules-core-18.0.0-4940.g81 1.4 MB/s | 241 kB 00:00 2023-07-14T19:05:06.121 INFO:teuthology.orchestra.run.smithi040.stdout:(32/131): ceph-mgr-rook-18.0.0-4940.g81c9387d.e 355 kB/s | 47 kB 00:00 2023-07-14T19:05:06.296 INFO:teuthology.orchestra.run.smithi040.stdout:(33/131): ceph-prometheus-alerts-18.0.0-4940.g8 83 kB/s | 15 kB 00:00 2023-07-14T19:05:06.338 INFO:teuthology.orchestra.run.smithi040.stdout:(34/131): ceph-mgr-dashboard-18.0.0-4940.g81c93 1.9 MB/s | 1.7 MB 00:00 2023-07-14T19:05:06.472 INFO:teuthology.orchestra.run.smithi040.stdout:(35/131): ceph-volume-18.0.0-4940.g81c9387d.el8 1.4 MB/s | 258 kB 00:00 2023-07-14T19:05:06.522 INFO:teuthology.orchestra.run.smithi040.stdout:(36/131): cephadm-18.0.0-4940.g81c9387d.el8.noa 901 kB/s | 165 kB 00:00 2023-07-14T19:05:06.578 INFO:teuthology.orchestra.run.smithi191.stdout:(25/131): rbd-mirror-18.0.0-4940.g81c9387d.el8. 1.4 MB/s | 3.5 MB 00:02 2023-07-14T19:05:06.596 INFO:teuthology.orchestra.run.smithi130.stdout:(130/131): librados2-18.0.0-4940.g81c9387d.el8. 2.7 MB/s | 3.7 MB 00:01 2023-07-14T19:05:06.616 INFO:teuthology.orchestra.run.smithi040.stdout:(37/131): cephfs-top-18.0.0-4940.g81c9387d.el8. 127 kB/s | 18 kB 00:00 2023-07-14T19:05:06.787 INFO:teuthology.orchestra.run.smithi191.stdout:(26/131): rbd-nbd-18.0.0-4940.g81c9387d.el8.x86 860 kB/s | 179 kB 00:00 2023-07-14T19:05:06.937 INFO:teuthology.orchestra.run.smithi191.stdout:(27/131): ceph-grafana-dashboards-18.0.0-4940.g 154 kB/s | 23 kB 00:00 2023-07-14T19:05:06.948 INFO:teuthology.orchestra.run.smithi040.stdout:(38/131): ceph-mgr-diskprediction-local-18.0.0- 5.3 MB/s | 7.4 MB 00:01 2023-07-14T19:05:07.088 INFO:teuthology.orchestra.run.smithi191.stdout:(28/131): ceph-mgr-cephadm-18.0.0-4940.g81c9387 862 kB/s | 129 kB 00:00 2023-07-14T19:05:07.321 INFO:teuthology.orchestra.run.smithi191.stdout:(29/131): librgw2-18.0.0-4940.g81c9387d.el8.x86 1.3 MB/s | 6.8 MB 00:05 2023-07-14T19:05:07.506 INFO:teuthology.orchestra.run.smithi040.stdout:(39/131): liboath-2.6.2-3.el8.x86_64.rpm 66 kB/s | 59 kB 00:00 2023-07-14T19:05:07.556 INFO:teuthology.orchestra.run.smithi040.stdout:(40/131): libunwind-1.3.1-3.el8.x86_64.rpm 123 kB/s | 75 kB 00:00 2023-07-14T19:05:07.713 INFO:teuthology.orchestra.run.smithi130.stdout:(131/131): librbd1-18.0.0-4940.g81c9387d.el8.x8 1.5 MB/s | 3.6 MB 00:02 2023-07-14T19:05:07.717 INFO:teuthology.orchestra.run.smithi130.stdout:-------------------------------------------------------------------------------- 2023-07-14T19:05:07.718 INFO:teuthology.orchestra.run.smithi130.stdout:Total 7.7 MB/s | 206 MB 00:26 2023-07-14T19:05:07.923 INFO:teuthology.orchestra.run.smithi040.stdout:(41/131): python3-bcrypt-3.1.6-2.el8.1.x86_64.r 119 kB/s | 44 kB 00:00 2023-07-14T19:05:08.238 INFO:teuthology.orchestra.run.smithi191.stdout:(30/131): ceph-mgr-dashboard-18.0.0-4940.g81c93 1.5 MB/s | 1.7 MB 00:01 2023-07-14T19:05:08.522 INFO:teuthology.orchestra.run.smithi191.stdout:(31/131): ceph-mgr-modules-core-18.0.0-4940.g81 850 kB/s | 241 kB 00:00 2023-07-14T19:05:08.543 INFO:teuthology.orchestra.run.smithi130.stdout:Running transaction check 2023-07-14T19:05:08.663 INFO:teuthology.orchestra.run.smithi130.stdout:Transaction check succeeded. 2023-07-14T19:05:08.664 INFO:teuthology.orchestra.run.smithi130.stdout:Running transaction test 2023-07-14T19:05:08.709 INFO:teuthology.orchestra.run.smithi191.stdout:(32/131): ceph-mgr-rook-18.0.0-4940.g81c9387d.e 253 kB/s | 47 kB 00:00 2023-07-14T19:05:08.765 INFO:teuthology.orchestra.run.smithi040.stdout:(42/131): python3-beautifulsoup4-4.6.3-2.el8.1. 220 kB/s | 185 kB 00:00 2023-07-14T19:05:08.847 INFO:teuthology.orchestra.run.smithi191.stdout:(33/131): ceph-prometheus-alerts-18.0.0-4940.g8 105 kB/s | 15 kB 00:00 2023-07-14T19:05:08.907 INFO:teuthology.orchestra.run.smithi040.stdout:(43/131): gperftools-libs-2.7-9.el8.x86_64.rpm 128 kB/s | 306 kB 00:02 2023-07-14T19:05:08.932 INFO:teuthology.orchestra.run.smithi040.stdout:(44/131): python3-cachetools-3.1.1-4.el8.noarch 198 kB/s | 33 kB 00:00 2023-07-14T19:05:09.039 INFO:teuthology.orchestra.run.smithi191.stdout:(34/131): ceph-volume-18.0.0-4940.g81c9387d.el8 1.3 MB/s | 258 kB 00:00 2023-07-14T19:05:09.141 INFO:teuthology.orchestra.run.smithi040.stdout:(45/131): python3-certifi-2018.10.15-7.el8.noar 70 kB/s | 16 kB 00:00 2023-07-14T19:05:09.189 INFO:teuthology.orchestra.run.smithi191.stdout:(35/131): cephadm-18.0.0-4940.g81c9387d.el8.noa 1.1 MB/s | 165 kB 00:00 2023-07-14T19:05:09.323 INFO:teuthology.orchestra.run.smithi191.stdout:(36/131): cephfs-top-18.0.0-4940.g81c9387d.el8. 135 kB/s | 18 kB 00:00 2023-07-14T19:05:09.616 INFO:teuthology.orchestra.run.smithi040.stdout:(46/131): python3-cheroot-8.5.2-1.el8.noarch.rp 252 kB/s | 173 kB 00:00 2023-07-14T19:05:09.781 INFO:teuthology.orchestra.run.smithi130.stdout:Transaction test succeeded. 2023-07-14T19:05:09.785 INFO:teuthology.orchestra.run.smithi130.stdout:Running transaction 2023-07-14T19:05:09.973 INFO:teuthology.orchestra.run.smithi191.stdout:(37/131): gperftools-libs-2.7-9.el8.x86_64.rpm 471 kB/s | 306 kB 00:00 2023-07-14T19:05:10.033 INFO:teuthology.orchestra.run.smithi040.stdout:(47/131): python3-google-auth-1.1.1-10.el8.noar 195 kB/s | 81 kB 00:00 2023-07-14T19:05:10.082 INFO:teuthology.orchestra.run.smithi191.stdout:(38/131): liboath-2.6.2-3.el8.x86_64.rpm 544 kB/s | 59 kB 00:00 2023-07-14T19:05:10.142 INFO:teuthology.orchestra.run.smithi040.stdout:(48/131): python3-jaraco-6.2-6.el8.noarch.rpm 102 kB/s | 11 kB 00:00 2023-07-14T19:05:10.259 INFO:teuthology.orchestra.run.smithi040.stdout:(49/131): python3-jaraco-functools-2.0-4.el8.no 151 kB/s | 18 kB 00:00 2023-07-14T19:05:10.449 INFO:teuthology.orchestra.run.smithi191.stdout:(39/131): libunwind-1.3.1-3.el8.x86_64.rpm 205 kB/s | 75 kB 00:00 2023-07-14T19:05:10.901 INFO:teuthology.orchestra.run.smithi040.stdout:(50/131): python3-cherrypy-18.4.0-1.el8.noarch. 218 kB/s | 384 kB 00:01 2023-07-14T19:05:11.049 INFO:teuthology.orchestra.run.smithi191.stdout:(40/131): python3-asyncssh-2.7.0-2.el8.noarch.r 793 kB/s | 476 kB 00:00 2023-07-14T19:05:11.101 INFO:teuthology.orchestra.run.smithi040.stdout:(51/131): python3-logutils-0.3.5-11.el8.noarch. 244 kB/s | 49 kB 00:00 2023-07-14T19:05:11.133 INFO:teuthology.orchestra.run.smithi191.stdout:(41/131): ceph-mgr-diskprediction-local-18.0.0- 1.9 MB/s | 7.4 MB 00:03 2023-07-14T19:05:11.158 INFO:teuthology.orchestra.run.smithi191.stdout:(42/131): python3-bcrypt-3.1.6-2.el8.1.x86_64.r 402 kB/s | 44 kB 00:00 2023-07-14T19:05:11.267 INFO:teuthology.orchestra.run.smithi191.stdout:(43/131): python3-cachetools-3.1.1-4.el8.noarch 305 kB/s | 33 kB 00:00 2023-07-14T19:05:11.322 INFO:teuthology.orchestra.run.smithi040.stdout:(52/131): python3-more-itertools-7.2.0-3.el8.no 267 kB/s | 59 kB 00:00 2023-07-14T19:05:11.376 INFO:teuthology.orchestra.run.smithi191.stdout:(44/131): python3-certifi-2018.10.15-7.el8.noar 150 kB/s | 16 kB 00:00 2023-07-14T19:05:11.435 INFO:teuthology.orchestra.run.smithi040.stdout:(53/131): python3-asyncssh-2.7.0-2.el8.noarch.r 121 kB/s | 476 kB 00:03 2023-07-14T19:05:11.544 INFO:teuthology.orchestra.run.smithi040.stdout:(54/131): python3-natsort-7.1.1-2.el8.noarch.rp 272 kB/s | 60 kB 00:00 2023-07-14T19:05:11.576 INFO:teuthology.orchestra.run.smithi191.stdout:(45/131): python3-cheroot-8.5.2-1.el8.noarch.rp 864 kB/s | 173 kB 00:00 2023-07-14T19:05:11.652 INFO:teuthology.orchestra.run.smithi040.stdout:(55/131): python3-portend-2.6-1.el8.noarch.rpm 144 kB/s | 16 kB 00:00 2023-07-14T19:05:11.784 INFO:teuthology.orchestra.run.smithi191.stdout:(46/131): python3-beautifulsoup4-4.6.3-2.el8.1. 284 kB/s | 185 kB 00:00 2023-07-14T19:05:11.861 INFO:teuthology.orchestra.run.smithi040.stdout:(56/131): python3-repoze-lru-0.7-6.el8.noarch.r 161 kB/s | 34 kB 00:00 2023-07-14T19:05:11.968 INFO:teuthology.orchestra.run.smithi191.stdout:(47/131): python3-cherrypy-18.4.0-1.el8.noarch. 981 kB/s | 384 kB 00:00 2023-07-14T19:05:11.993 INFO:teuthology.orchestra.run.smithi191.stdout:(48/131): python3-google-auth-1.1.1-10.el8.noar 389 kB/s | 81 kB 00:00 2023-07-14T19:05:12.577 INFO:teuthology.orchestra.run.smithi191.stdout:(49/131): ceph-test-18.0.0-4940.g81c9387d.el8.x 3.6 MB/s | 48 MB 00:13 2023-07-14T19:05:12.602 INFO:teuthology.orchestra.run.smithi191.stdout:(50/131): python3-jaraco-6.2-6.el8.noarch.rpm 17 kB/s | 11 kB 00:00 2023-07-14T19:05:12.627 INFO:teuthology.orchestra.run.smithi191.stdout:(51/131): python3-jaraco-functools-2.0-4.el8.no 28 kB/s | 18 kB 00:00 2023-07-14T19:05:12.836 INFO:teuthology.orchestra.run.smithi191.stdout:(52/131): python3-logutils-0.3.5-11.el8.noarch. 208 kB/s | 49 kB 00:00 2023-07-14T19:05:13.061 INFO:teuthology.orchestra.run.smithi191.stdout:(53/131): python3-more-itertools-7.2.0-3.el8.no 135 kB/s | 59 kB 00:00 2023-07-14T19:05:13.086 INFO:teuthology.orchestra.run.smithi191.stdout:(54/131): python3-natsort-7.1.1-2.el8.noarch.rp 241 kB/s | 60 kB 00:00 2023-07-14T19:05:13.212 INFO:teuthology.orchestra.run.smithi191.stdout:(55/131): python3-portend-2.6-1.el8.noarch.rpm 125 kB/s | 16 kB 00:00 2023-07-14T19:05:13.269 INFO:teuthology.orchestra.run.smithi040.stdout:(57/131): python3-routes-2.4.1-12.el8.noarch.rp 139 kB/s | 196 kB 00:01 2023-07-14T19:05:13.337 INFO:teuthology.orchestra.run.smithi191.stdout:(56/131): python3-repoze-lru-0.7-6.el8.noarch.r 269 kB/s | 34 kB 00:00 2023-07-14T19:05:13.571 INFO:teuthology.orchestra.run.smithi191.stdout:(57/131): python3-routes-2.4.1-12.el8.noarch.rp 842 kB/s | 196 kB 00:00 2023-07-14T19:05:13.713 INFO:teuthology.orchestra.run.smithi191.stdout:(58/131): python3-rsa-4.8-1.el8.noarch.rpm 434 kB/s | 61 kB 00:00 2023-07-14T19:05:13.838 INFO:teuthology.orchestra.run.smithi191.stdout:(59/131): python3-simplegeneric-0.8.1-17.el8.no 150 kB/s | 19 kB 00:00 2023-07-14T19:05:13.938 INFO:teuthology.orchestra.run.smithi191.stdout:(60/131): python3-pecan-1.3.2-9.el8.noarch.rpm 323 kB/s | 283 kB 00:00 2023-07-14T19:05:13.972 INFO:teuthology.orchestra.run.smithi191.stdout:(61/131): python3-singledispatch-3.4.0.3-18.el8 179 kB/s | 24 kB 00:00 2023-07-14T19:05:14.072 INFO:teuthology.orchestra.run.smithi191.stdout:(62/131): python3-tempora-1.14.1-5.el8.noarch.r 219 kB/s | 29 kB 00:00 2023-07-14T19:05:14.135 INFO:teuthology.orchestra.run.smithi191.stdout:(63/131): python3-trustme-0.6.0-4.el8.noarch.rp 168 kB/s | 27 kB 00:00 2023-07-14T19:05:14.184 INFO:teuthology.orchestra.run.smithi040.stdout:(58/131): python3-rsa-4.8-1.el8.noarch.rpm 67 kB/s | 61 kB 00:00 2023-07-14T19:05:14.253 INFO:teuthology.orchestra.run.smithi040.stdout:(59/131): python3-pecan-1.3.2-9.el8.noarch.rpm 100 kB/s | 283 kB 00:02 2023-07-14T19:05:14.437 INFO:teuthology.orchestra.run.smithi040.stdout:(60/131): python3-simplegeneric-0.8.1-17.el8.no 74 kB/s | 19 kB 00:00 2023-07-14T19:05:14.512 INFO:teuthology.orchestra.run.smithi040.stdout:(61/131): python3-singledispatch-3.4.0.3-18.el8 92 kB/s | 24 kB 00:00 2023-07-14T19:05:14.531 INFO:teuthology.orchestra.run.smithi191.stdout:(64/131): python3-webob-1.8.5-1.el8.1.noarch.rp 636 kB/s | 251 kB 00:00 2023-07-14T19:05:14.681 INFO:teuthology.orchestra.run.smithi191.stdout:(65/131): python3-websocket-client-0.56.0-5.el8 408 kB/s | 61 kB 00:00 2023-07-14T19:05:14.821 INFO:teuthology.orchestra.run.smithi040.stdout:(62/131): python3-tempora-1.14.1-5.el8.noarch.r 76 kB/s | 29 kB 00:00 2023-07-14T19:05:14.832 INFO:teuthology.orchestra.run.smithi191.stdout:(66/131): python3-webtest-2.0.33-1.el8.noarch.r 569 kB/s | 85 kB 00:00 2023-07-14T19:05:14.902 INFO:teuthology.orchestra.run.smithi130.stdout: Preparing : 1/1 2023-07-14T19:05:14.973 INFO:teuthology.orchestra.run.smithi191.stdout:(67/131): python3-zc-lockfile-2.0-2.el8.noarch. 162 kB/s | 23 kB 00:00 2023-07-14T19:05:15.124 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : thrift-0.13.0-2.el8.x86_64 1/134 2023-07-14T19:05:15.196 INFO:teuthology.orchestra.run.smithi040.stdout:(63/131): python3-trustme-0.6.0-4.el8.noarch.rp 40 kB/s | 27 kB 00:00 2023-07-14T19:05:15.274 INFO:teuthology.orchestra.run.smithi191.stdout:(68/131): python3-waitress-1.4.3-1.el8.noarch.r 213 kB/s | 255 kB 00:01 2023-07-14T19:05:15.299 INFO:teuthology.orchestra.run.smithi191.stdout:(69/131): xmlstarlet-1.6.1-20.el8.x86_64.rpm 2.7 MB/s | 70 kB 00:00 2023-07-14T19:05:15.303 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : liboath-2.6.2-3.el8.x86_64 2/134 2023-07-14T19:05:15.433 INFO:teuthology.orchestra.run.smithi191.stdout:(70/131): librabbitmq-0.9.0-3.el8.x86_64.rpm 351 kB/s | 47 kB 00:00 2023-07-14T19:05:15.513 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : librabbitmq-0.9.0-3.el8.x86_64 3/134 2023-07-14T19:05:15.525 INFO:teuthology.orchestra.run.smithi191.stdout:(71/131): libselinux-devel-2.9-5.el8.x86_64.rpm 2.1 MB/s | 200 kB 00:00 2023-07-14T19:05:15.681 INFO:teuthology.orchestra.run.smithi191.stdout:(72/131): python3-jwt-1.6.1-2.el8.noarch.rpm 273 kB/s | 43 kB 00:00 2023-07-14T19:05:15.706 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : librdkafka-0.11.4-3.el8.x86_64 4/134 2023-07-14T19:05:15.799 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: librdkafka-0.11.4-3.el8.x86_64 4/134 2023-07-14T19:05:15.834 INFO:teuthology.orchestra.run.smithi191.stdout:(73/131): pcre2-utf32-10.32-2.el8.x86_64.rpm 1.4 MB/s | 220 kB 00:00 2023-07-14T19:05:15.993 INFO:teuthology.orchestra.run.smithi191.stdout:(74/131): python3-kubernetes-11.0.0-6.el8.noarc 349 kB/s | 1.2 MB 00:03 2023-07-14T19:05:16.076 INFO:teuthology.orchestra.run.smithi191.stdout:(75/131): keyutils-libs-devel-1.5.10-9.el8.x86_ 199 kB/s | 48 kB 00:00 2023-07-14T19:05:16.135 INFO:teuthology.orchestra.run.smithi191.stdout:(76/131): libsepol-devel-2.9-3.el8.x86_64.rpm 614 kB/s | 87 kB 00:00 2023-07-14T19:05:16.193 INFO:teuthology.orchestra.run.smithi191.stdout:(77/131): python3-requests-oauthlib-1.0.0-1.el8 365 kB/s | 43 kB 00:00 2023-07-14T19:05:16.239 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 5/134 2023-07-14T19:05:16.310 INFO:teuthology.orchestra.run.smithi191.stdout:(78/131): python3-pycparser-2.14-14.el8.noarch. 623 kB/s | 109 kB 00:00 2023-07-14T19:05:16.335 INFO:teuthology.orchestra.run.smithi191.stdout:(79/131): pcre2-utf16-10.32-2.el8.x86_64.rpm 1.6 MB/s | 229 kB 00:00 2023-07-14T19:05:16.419 INFO:teuthology.orchestra.run.smithi191.stdout:(80/131): krb5-devel-1.18.2-14.el8.x86_64.rpm 5.0 MB/s | 560 kB 00:00 2023-07-14T19:05:16.496 INFO:teuthology.orchestra.run.smithi040.stdout:(64/131): python3-kubernetes-11.0.0-6.el8.noarc 191 kB/s | 1.2 MB 00:06 2023-07-14T19:05:16.519 INFO:teuthology.orchestra.run.smithi191.stdout:(81/131): libaio-devel-0.3.112-1.el8.x86_64.rpm 102 kB/s | 19 kB 00:00 2023-07-14T19:05:16.521 INFO:teuthology.orchestra.run.smithi040.stdout:(65/131): python3-waitress-1.4.3-1.el8.noarch.r 150 kB/s | 255 kB 00:01 2023-07-14T19:05:16.603 INFO:teuthology.orchestra.run.smithi191.stdout:(82/131): python3-oauthlib-2.1.0-1.el8.noarch.r 844 kB/s | 155 kB 00:00 2023-07-14T19:05:16.626 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 5/134 2023-07-14T19:05:16.720 INFO:teuthology.orchestra.run.smithi191.stdout:(83/131): python3-cryptography-3.2.1-5.el8.x86_ 2.7 MB/s | 559 kB 00:00 2023-07-14T19:05:16.745 INFO:teuthology.orchestra.run.smithi191.stdout:(84/131): libverto-devel-0.3.0-5.el8.x86_64.rpm 126 kB/s | 18 kB 00:00 2023-07-14T19:05:16.812 INFO:teuthology.orchestra.run.smithi191.stdout:(85/131): pcre2-devel-10.32-2.el8.x86_64.rpm 6.4 MB/s | 605 kB 00:00 2023-07-14T19:05:16.871 INFO:teuthology.orchestra.run.smithi191.stdout:(86/131): python3-cffi-1.11.5-5.el8.x86_64.rpm 1.9 MB/s | 238 kB 00:00 2023-07-14T19:05:16.872 INFO:teuthology.orchestra.run.smithi130.stdout: Upgrading : librados2-2:18.0.0-4940.g81c9387d.el8.x86_64 6/134 2023-07-14T19:05:16.971 INFO:teuthology.orchestra.run.smithi191.stdout:(87/131): openssl-devel-1.1.1k-7.el8_6.x86_64.r 15 MB/s | 2.3 MB 00:00 2023-07-14T19:05:17.038 INFO:teuthology.orchestra.run.smithi191.stdout:(88/131): xfsprogs-devel-5.0.0-10.el8.x86_64.rp 433 kB/s | 72 kB 00:00 2023-07-14T19:05:17.055 INFO:teuthology.orchestra.run.smithi040.stdout:(66/131): python3-webtest-2.0.33-1.el8.noarch.r 160 kB/s | 85 kB 00:00 2023-07-14T19:05:17.088 INFO:teuthology.orchestra.run.smithi191.stdout:(89/131): fuse-2.9.7-15.el8.x86_64.rpm 707 kB/s | 83 kB 00:00 2023-07-14T19:05:17.155 INFO:teuthology.orchestra.run.smithi040.stdout:(67/131): python3-websocket-client-0.56.0-5.el8 93 kB/s | 61 kB 00:00 2023-07-14T19:05:17.206 INFO:teuthology.orchestra.run.smithi040.stdout:(68/131): python3-zc-lockfile-2.0-2.el8.noarch. 153 kB/s | 23 kB 00:00 2023-07-14T19:05:17.219 INFO:teuthology.orchestra.run.smithi191.stdout:(90/131): libuuid-devel-2.32.1-35.el8.x86_64.rp 753 kB/s | 98 kB 00:00 2023-07-14T19:05:17.231 INFO:teuthology.orchestra.run.smithi040.stdout:(69/131): xmlstarlet-1.6.1-20.el8.x86_64.rpm 2.7 MB/s | 70 kB 00:00 2023-07-14T19:05:17.247 INFO:teuthology.orchestra.run.smithi191.stdout:(91/131): elfutils-libelf-devel-0.186-1.el8.x86 287 kB/s | 60 kB 00:00 2023-07-14T19:05:17.257 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: librados2-2:18.0.0-4940.g81c9387d.el8.x86_64 6/134 2023-07-14T19:05:17.323 INFO:teuthology.orchestra.run.smithi191.stdout:(92/131): zlib-devel-1.2.11-18.el8_5.x86_64.rpm 559 kB/s | 58 kB 00:00 2023-07-14T19:05:17.381 INFO:teuthology.orchestra.run.smithi040.stdout:(70/131): librabbitmq-0.9.0-3.el8.x86_64.rpm 312 kB/s | 47 kB 00:00 2023-07-14T19:05:17.398 INFO:teuthology.orchestra.run.smithi191.stdout:(93/131): libcom_err-devel-1.45.6-4.el8.x86_64. 257 kB/s | 39 kB 00:00 2023-07-14T19:05:17.432 INFO:teuthology.orchestra.run.smithi191.stdout:(94/131): libgfortran-8.5.0-10.1.el8_6.x86_64.r 5.8 MB/s | 644 kB 00:00 2023-07-14T19:05:17.439 INFO:teuthology.orchestra.run.smithi130.stdout: Upgrading : librbd1-2:18.0.0-4940.g81c9387d.el8.x86_64 7/134 2023-07-14T19:05:17.498 INFO:teuthology.orchestra.run.smithi191.stdout:(95/131): NetworkManager-initscripts-updown-1.3 2.0 MB/s | 138 kB 00:00 2023-07-14T19:05:17.562 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: librbd1-2:18.0.0-4940.g81c9387d.el8.x86_64 7/134 2023-07-14T19:05:17.573 INFO:teuthology.orchestra.run.smithi040.stdout:(71/131): libselinux-devel-2.9-5.el8.x86_64.rpm 1.0 MB/s | 200 kB 00:00 2023-07-14T19:05:17.582 INFO:teuthology.orchestra.run.smithi191.stdout:(96/131): libquadmath-8.5.0-10.1.el8_6.x86_64.r 931 kB/s | 171 kB 00:00 2023-07-14T19:05:17.632 INFO:teuthology.orchestra.run.smithi040.stdout:(72/131): python3-jwt-1.6.1-2.el8.noarch.rpm 734 kB/s | 43 kB 00:00 2023-07-14T19:05:17.633 INFO:teuthology.orchestra.run.smithi191.stdout:(97/131): python3-pyasn1-modules-0.3.7-6.el8.no 2.2 MB/s | 110 kB 00:00 2023-07-14T19:05:17.772 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : libcephfs2-2:18.0.0-4940.g81c9387d.el8.x86_64 8/134 2023-07-14T19:05:17.840 INFO:teuthology.orchestra.run.smithi040.stdout:(73/131): pcre2-utf32-10.32-2.el8.x86_64.rpm 1.0 MB/s | 220 kB 00:00 2023-07-14T19:05:17.841 INFO:teuthology.orchestra.run.smithi191.stdout:(98/131): python3-scipy-1.0.0-21.module+el8.5.0 40 MB/s | 14 MB 00:00 2023-07-14T19:05:17.866 INFO:teuthology.orchestra.run.smithi040.stdout:(74/131): python3-webob-1.8.5-1.el8.1.noarch.rp 94 kB/s | 251 kB 00:02 2023-07-14T19:05:17.866 INFO:teuthology.orchestra.run.smithi191.stdout:(99/131): python3-pytz-2017.2-9.el8.noarch.rpm 232 kB/s | 54 kB 00:00 2023-07-14T19:05:17.868 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: libcephfs2-2:18.0.0-4940.g81c9387d.el8.x86_64 8/134 2023-07-14T19:05:17.924 INFO:teuthology.orchestra.run.smithi040.stdout:(75/131): keyutils-libs-devel-1.5.10-9.el8.x86_ 574 kB/s | 48 kB 00:00 2023-07-14T19:05:17.942 INFO:teuthology.orchestra.run.smithi191.stdout:(100/131): lttng-ust-2.8.1-11.el8.x86_64.rpm 2.5 MB/s | 259 kB 00:00 2023-07-14T19:05:18.016 INFO:teuthology.orchestra.run.smithi040.stdout:(76/131): libsepol-devel-2.9-3.el8.x86_64.rpm 578 kB/s | 87 kB 00:00 2023-07-14T19:05:18.042 INFO:teuthology.orchestra.run.smithi191.stdout:(101/131): socat-1.7.4.1-1.el8.x86_64.rpm 1.8 MB/s | 323 kB 00:00 2023-07-14T19:05:18.043 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-rados-2:18.0.0-4940.g81c9387d.el8.x86_64 9/134 2023-07-14T19:05:18.058 INFO:teuthology.orchestra.run.smithi040.stdout:(77/131): python3-requests-oauthlib-1.0.0-1.el8 319 kB/s | 43 kB 00:00 2023-07-14T19:05:18.068 INFO:teuthology.orchestra.run.smithi191.stdout:(102/131): python3-markupsafe-0.23-19.el8.x86_6 309 kB/s | 39 kB 00:00 2023-07-14T19:05:18.109 INFO:teuthology.orchestra.run.smithi040.stdout:(78/131): python3-pycparser-2.14-14.el8.noarch. 1.2 MB/s | 109 kB 00:00 2023-07-14T19:05:18.126 INFO:teuthology.orchestra.run.smithi191.stdout:(103/131): bison-3.0.4-10.el8.x86_64.rpm 8.0 MB/s | 688 kB 00:00 2023-07-14T19:05:18.176 INFO:teuthology.orchestra.run.smithi191.stdout:(104/131): python3-jinja2-2.10.1-3.el8.noarch.r 4.8 MB/s | 538 kB 00:00 2023-07-14T19:05:18.217 INFO:teuthology.orchestra.run.smithi040.stdout:(79/131): pcre2-utf16-10.32-2.el8.x86_64.rpm 1.4 MB/s | 229 kB 00:00 2023-07-14T19:05:18.218 INFO:teuthology.orchestra.run.smithi191.stdout:(105/131): python3-mako-1.0.6-13.el8.noarch.rpm 1.7 MB/s | 157 kB 00:00 2023-07-14T19:05:18.244 INFO:teuthology.orchestra.run.smithi191.stdout:(106/131): oniguruma-6.8.2-2.el8.x86_64.rpm 2.7 MB/s | 187 kB 00:00 2023-07-14T19:05:18.280 INFO:teuthology.orchestra.run.smithi040.stdout:(80/131): krb5-devel-1.18.2-14.el8.x86_64.rpm 3.2 MB/s | 560 kB 00:00 2023-07-14T19:05:18.309 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : libpq-13.5-1.el8.x86_64 10/134 2023-07-14T19:05:18.335 INFO:teuthology.orchestra.run.smithi040.stdout:(81/131): libaio-devel-0.3.112-1.el8.x86_64.rpm 160 kB/s | 19 kB 00:00 2023-07-14T19:05:18.394 INFO:teuthology.orchestra.run.smithi191.stdout:(107/131): automake-1.16.1-7.el8.noarch.rpm 4.0 MB/s | 713 kB 00:00 2023-07-14T19:05:18.410 INFO:teuthology.orchestra.run.smithi040.stdout:(82/131): python3-oauthlib-2.1.0-1.el8.noarch.r 1.2 MB/s | 155 kB 00:00 2023-07-14T19:05:18.460 INFO:teuthology.orchestra.run.smithi040.stdout:(83/131): python3-cryptography-3.2.1-5.el8.x86_ 4.4 MB/s | 559 kB 00:00 2023-07-14T19:05:18.478 INFO:teuthology.orchestra.run.smithi191.stdout:(108/131): python3-babel-2.5.1-7.el8.noarch.rpm 20 MB/s | 4.8 MB 00:00 2023-07-14T19:05:18.503 INFO:teuthology.orchestra.run.smithi191.stdout:(109/131): python3-jsonpatch-1.21-2.el8.noarch. 245 kB/s | 27 kB 00:00 2023-07-14T19:05:18.550 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 11/134 2023-07-14T19:05:18.560 INFO:teuthology.orchestra.run.smithi040.stdout:(84/131): libverto-devel-0.3.0-5.el8.x86_64.rpm 119 kB/s | 18 kB 00:00 2023-07-14T19:05:18.578 INFO:teuthology.orchestra.run.smithi191.stdout:(110/131): flex-2.6.1-9.el8.x86_64.rpm 3.1 MB/s | 320 kB 00:00 2023-07-14T19:05:18.619 INFO:teuthology.orchestra.run.smithi040.stdout:(85/131): pcre2-devel-10.32-2.el8.x86_64.rpm 3.7 MB/s | 605 kB 00:00 2023-07-14T19:05:18.645 INFO:teuthology.orchestra.run.smithi191.stdout:(111/131): python3-prettytable-0.7.2-14.el8.noa 309 kB/s | 44 kB 00:00 2023-07-14T19:05:18.703 INFO:teuthology.orchestra.run.smithi040.stdout:(86/131): python3-cffi-1.11.5-5.el8.x86_64.rpm 1.6 MB/s | 238 kB 00:00 2023-07-14T19:05:18.787 INFO:teuthology.orchestra.run.smithi191.stdout:(112/131): python3-numpy-1.14.3-10.el8.x86_64.r 18 MB/s | 3.7 MB 00:00 2023-07-14T19:05:18.811 INFO:teuthology.orchestra.run.smithi040.stdout:(87/131): openssl-devel-1.1.1k-7.el8_6.x86_64.r 12 MB/s | 2.3 MB 00:00 2023-07-14T19:05:18.821 INFO:teuthology.orchestra.run.smithi191.stdout:(113/131): python3-werkzeug-0.12.2-4.el8.noarch 2.5 MB/s | 457 kB 00:00 2023-07-14T19:05:18.828 INFO:teuthology.orchestra.run.smithi040.stdout:(88/131): xfsprogs-devel-5.0.0-10.el8.x86_64.rp 577 kB/s | 72 kB 00:00 2023-07-14T19:05:18.929 INFO:teuthology.orchestra.run.smithi191.stdout:(114/131): uuid-1.6.2-43.el8.x86_64.rpm 591 kB/s | 64 kB 00:00 2023-07-14T19:05:18.945 INFO:teuthology.orchestra.run.smithi040.stdout:(89/131): elfutils-libelf-devel-0.186-1.el8.x86 513 kB/s | 60 kB 00:00 2023-07-14T19:05:18.949 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 12/134 2023-07-14T19:05:18.963 INFO:teuthology.orchestra.run.smithi191.stdout:(115/131): python3-numpy-f2py-1.14.3-10.el8.x86 1.3 MB/s | 225 kB 00:00 2023-07-14T19:05:19.004 INFO:teuthology.orchestra.run.smithi040.stdout:(90/131): fuse-2.9.7-15.el8.x86_64.rpm 430 kB/s | 83 kB 00:00 2023-07-14T19:05:19.054 INFO:teuthology.orchestra.run.smithi040.stdout:(91/131): zlib-devel-1.2.11-18.el8_5.x86_64.rpm 1.1 MB/s | 58 kB 00:00 2023-07-14T19:05:19.055 INFO:teuthology.orchestra.run.smithi191.stdout:(116/131): python3-pyasn1-0.3.7-6.el8.noarch.rp 1.0 MB/s | 126 kB 00:00 2023-07-14T19:05:19.080 INFO:teuthology.orchestra.run.smithi191.stdout:(117/131): python3-jsonpointer-1.10-11.el8.noar 171 kB/s | 20 kB 00:00 2023-07-14T19:05:19.114 INFO:teuthology.orchestra.run.smithi040.stdout:(92/131): libuuid-devel-2.32.1-35.el8.x86_64.rp 587 kB/s | 98 kB 00:00 2023-07-14T19:05:19.159 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : postgresql-10.21-2.module+el8.6.0+15342+53518fac 13/134 2023-07-14T19:05:19.197 INFO:teuthology.orchestra.run.smithi191.stdout:(118/131): python3-pyOpenSSL-19.0.0-1.el8.noarc 730 kB/s | 103 kB 00:00 2023-07-14T19:05:19.239 INFO:teuthology.orchestra.run.smithi191.stdout:(119/131): libtool-2.4.6-25.el8.x86_64.rpm 4.4 MB/s | 709 kB 00:00 2023-07-14T19:05:19.246 INFO:teuthology.orchestra.run.smithi040.stdout:(93/131): libcom_err-devel-1.45.6-4.el8.x86_64. 201 kB/s | 39 kB 00:00 2023-07-14T19:05:19.280 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : libcephsqlite-2:18.0.0-4940.g81c9387d.el8.x86_64 14/134 2023-07-14T19:05:19.297 INFO:teuthology.orchestra.run.smithi040.stdout:(94/131): libgfortran-8.5.0-10.1.el8_6.x86_64.r 3.4 MB/s | 644 kB 00:00 2023-07-14T19:05:19.330 INFO:teuthology.orchestra.run.smithi040.stdout:(95/131): libquadmath-8.5.0-10.1.el8_6.x86_64.r 2.0 MB/s | 171 kB 00:00 2023-07-14T19:05:19.374 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: libcephsqlite-2:18.0.0-4940.g81c9387d.el8.x86_64 14/134 2023-07-14T19:05:19.406 INFO:teuthology.orchestra.run.smithi191.stdout:(120/131): libpq-13.5-1.el8.x86_64.rpm 1.2 MB/s | 198 kB 00:00 2023-07-14T19:05:19.464 INFO:teuthology.orchestra.run.smithi040.stdout:(96/131): NetworkManager-initscripts-updown-1.3 825 kB/s | 138 kB 00:00 2023-07-14T19:05:19.473 INFO:teuthology.orchestra.run.smithi191.stdout:(121/131): openblas-threads-0.3.15-3.el8.x86_64 18 MB/s | 4.9 MB 00:00 2023-07-14T19:05:19.522 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : libradosstriper1-2:18.0.0-4940.g81c9387d.el8.x86 15/134 2023-07-14T19:05:19.579 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: libradosstriper1-2:18.0.0-4940.g81c9387d.el8.x86 15/134 2023-07-14T19:05:19.598 INFO:teuthology.orchestra.run.smithi040.stdout:(97/131): python3-pyasn1-modules-0.3.7-6.el8.no 827 kB/s | 110 kB 00:00 2023-07-14T19:05:19.607 INFO:teuthology.orchestra.run.smithi191.stdout:(122/131): openblas-0.3.15-3.el8.x86_64.rpm 24 MB/s | 4.7 MB 00:00 2023-07-14T19:05:19.648 INFO:teuthology.orchestra.run.smithi191.stdout:(123/131): thrift-0.13.0-2.el8.x86_64.rpm 380 kB/s | 1.7 MB 00:04 2023-07-14T19:05:19.674 INFO:teuthology.orchestra.run.smithi191.stdout:(124/131): jq-1.6-3.el8.x86_64.rpm 1.0 MB/s | 202 kB 00:00 2023-07-14T19:05:19.722 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : libquadmath-8.5.0-10.1.el8_6.x86_64 16/134 2023-07-14T19:05:19.732 INFO:teuthology.orchestra.run.smithi191.stdout:(125/131): librdkafka-0.11.4-3.el8.x86_64.rpm 2.8 MB/s | 354 kB 00:00 2023-07-14T19:05:19.798 INFO:teuthology.orchestra.run.smithi040.stdout:(98/131): python3-scipy-1.0.0-21.module+el8.5.0 29 MB/s | 14 MB 00:00 2023-07-14T19:05:19.815 INFO:teuthology.orchestra.run.smithi040.stdout:(99/131): python3-pytz-2017.2-9.el8.noarch.rpm 250 kB/s | 54 kB 00:00 2023-07-14T19:05:19.833 INFO:teuthology.orchestra.run.smithi191.stdout:(126/131): postgresql-contrib-10.21-2.module+el 5.1 MB/s | 808 kB 00:00 2023-07-14T19:05:19.875 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: libquadmath-8.5.0-10.1.el8_6.x86_64 16/134 2023-07-14T19:05:19.907 INFO:teuthology.orchestra.run.smithi040.stdout:(100/131): lttng-ust-2.8.1-11.el8.x86_64.rpm 2.3 MB/s | 259 kB 00:00 2023-07-14T19:05:19.925 INFO:teuthology.orchestra.run.smithi191.stdout:(127/131): postgresql-10.21-2.module+el8.6.0+15 7.8 MB/s | 1.5 MB 00:00 2023-07-14T19:05:19.932 INFO:teuthology.orchestra.run.smithi040.stdout:(101/131): socat-1.7.4.1-1.el8.x86_64.rpm 2.7 MB/s | 323 kB 00:00 2023-07-14T19:05:20.000 INFO:teuthology.orchestra.run.smithi191.stdout:(128/131): postgresql-server-10.21-2.module+el8 14 MB/s | 5.1 MB 00:00 2023-07-14T19:05:20.013 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : libgfortran-8.5.0-10.1.el8_6.x86_64 17/134 2023-07-14T19:05:20.049 INFO:teuthology.orchestra.run.smithi040.stdout:(102/131): python3-markupsafe-0.23-19.el8.x86_6 273 kB/s | 39 kB 00:00 2023-07-14T19:05:20.082 INFO:teuthology.orchestra.run.smithi040.stdout:(103/131): bison-3.0.4-10.el8.x86_64.rpm 4.5 MB/s | 688 kB 00:00 2023-07-14T19:05:20.167 INFO:teuthology.orchestra.run.smithi191.stdout:(129/131): iproute-5.15.0-4.el8_6.1.x86_64.rpm 4.7 MB/s | 798 kB 00:00 2023-07-14T19:05:20.208 INFO:teuthology.orchestra.run.smithi040.stdout:(104/131): python3-mako-1.0.6-13.el8.noarch.rpm 1.2 MB/s | 157 kB 00:00 2023-07-14T19:05:20.242 INFO:teuthology.orchestra.run.smithi040.stdout:(105/131): python3-jinja2-2.10.1-3.el8.noarch.r 2.7 MB/s | 538 kB 00:00 2023-07-14T19:05:20.317 INFO:teuthology.orchestra.run.smithi040.stdout:(106/131): oniguruma-6.8.2-2.el8.x86_64.rpm 1.7 MB/s | 187 kB 00:00 2023-07-14T19:05:20.392 INFO:teuthology.orchestra.run.smithi040.stdout:(107/131): automake-1.16.1-7.el8.noarch.rpm 4.6 MB/s | 713 kB 00:00 2023-07-14T19:05:20.492 INFO:teuthology.orchestra.run.smithi040.stdout:(108/131): python3-jsonpatch-1.21-2.el8.noarch. 266 kB/s | 27 kB 00:00 2023-07-14T19:05:20.582 INFO:teuthology.orchestra.run.smithi040.stdout:(109/131): python3-babel-2.5.1-7.el8.noarch.rpm 18 MB/s | 4.8 MB 00:00 2023-07-14T19:05:20.612 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: libgfortran-8.5.0-10.1.el8_6.x86_64 17/134 2023-07-14T19:05:20.651 INFO:teuthology.orchestra.run.smithi040.stdout:(110/131): flex-2.6.1-9.el8.x86_64.rpm 2.0 MB/s | 320 kB 00:00 2023-07-14T19:05:20.702 INFO:teuthology.orchestra.run.smithi040.stdout:(111/131): python3-prettytable-0.7.2-14.el8.noa 368 kB/s | 44 kB 00:00 2023-07-14T19:05:20.780 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : openblas-threads-0.3.15-3.el8.x86_64 18/134 2023-07-14T19:05:20.819 INFO:teuthology.orchestra.run.smithi040.stdout:(112/131): python3-werkzeug-0.12.2-4.el8.noarch 3.8 MB/s | 457 kB 00:00 2023-07-14T19:05:20.935 INFO:teuthology.orchestra.run.smithi040.stdout:(113/131): python3-numpy-1.14.3-10.el8.x86_64.r 13 MB/s | 3.7 MB 00:00 2023-07-14T19:05:20.961 INFO:teuthology.orchestra.run.smithi040.stdout:(114/131): python3-numpy-f2py-1.14.3-10.el8.x86 1.5 MB/s | 225 kB 00:00 2023-07-14T19:05:20.975 INFO:teuthology.orchestra.run.smithi191.stdout:(130/131): librados2-18.0.0-4940.g81c9387d.el8. 3.2 MB/s | 3.7 MB 00:01 2023-07-14T19:05:21.003 INFO:teuthology.orchestra.run.smithi040.stdout:(115/131): uuid-1.6.2-43.el8.x86_64.rpm 956 kB/s | 64 kB 00:00 2023-07-14T19:05:21.052 INFO:teuthology.orchestra.run.smithi040.stdout:(116/131): python3-pyasn1-0.3.7-6.el8.noarch.rp 1.4 MB/s | 126 kB 00:00 2023-07-14T19:05:21.120 INFO:teuthology.orchestra.run.smithi040.stdout:(117/131): python3-jsonpointer-1.10-11.el8.noar 171 kB/s | 20 kB 00:00 2023-07-14T19:05:21.145 INFO:teuthology.orchestra.run.smithi040.stdout:(118/131): python3-pyOpenSSL-19.0.0-1.el8.noarc 1.1 MB/s | 103 kB 00:00 2023-07-14T19:05:21.196 INFO:teuthology.orchestra.run.smithi040.stdout:(119/131): libtool-2.4.6-25.el8.x86_64.rpm 9.2 MB/s | 709 kB 00:00 2023-07-14T19:05:21.321 INFO:teuthology.orchestra.run.smithi040.stdout:(120/131): openblas-threads-0.3.15-3.el8.x86_64 28 MB/s | 4.9 MB 00:00 2023-07-14T19:05:21.346 INFO:teuthology.orchestra.run.smithi040.stdout:(121/131): libpq-13.5-1.el8.x86_64.rpm 1.3 MB/s | 198 kB 00:00 2023-07-14T19:05:21.371 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: openblas-threads-0.3.15-3.el8.x86_64 18/134 2023-07-14T19:05:21.463 INFO:teuthology.orchestra.run.smithi040.stdout:(122/131): jq-1.6-3.el8.x86_64.rpm 1.7 MB/s | 202 kB 00:00 2023-07-14T19:05:21.563 INFO:teuthology.orchestra.run.smithi040.stdout:(123/131): openblas-0.3.15-3.el8.x86_64.rpm 20 MB/s | 4.7 MB 00:00 2023-07-14T19:05:21.588 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : openblas-0.3.15-3.el8.x86_64 19/134 2023-07-14T19:05:21.597 INFO:teuthology.orchestra.run.smithi040.stdout:(124/131): librdkafka-0.11.4-3.el8.x86_64.rpm 2.6 MB/s | 354 kB 00:00 2023-07-14T19:05:21.697 INFO:teuthology.orchestra.run.smithi040.stdout:(125/131): postgresql-contrib-10.21-2.module+el 7.9 MB/s | 808 kB 00:00 2023-07-14T19:05:21.797 INFO:teuthology.orchestra.run.smithi040.stdout:(126/131): postgresql-server-10.21-2.module+el8 22 MB/s | 5.1 MB 00:00 2023-07-14T19:05:21.848 INFO:teuthology.orchestra.run.smithi040.stdout:(127/131): postgresql-10.21-2.module+el8.6.0+15 10 MB/s | 1.5 MB 00:00 2023-07-14T19:05:21.859 INFO:teuthology.orchestra.run.smithi191.stdout:(131/131): librbd1-18.0.0-4940.g81c9387d.el8.x8 1.9 MB/s | 3.6 MB 00:01 2023-07-14T19:05:21.862 INFO:teuthology.orchestra.run.smithi191.stdout:-------------------------------------------------------------------------------- 2023-07-14T19:05:21.862 INFO:teuthology.orchestra.run.smithi191.stdout:Total 5.1 MB/s | 206 MB 00:39 2023-07-14T19:05:22.295 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: openblas-0.3.15-3.el8.x86_64 19/134 2023-07-14T19:05:22.533 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 20/134 2023-07-14T19:05:22.675 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction check 2023-07-14T19:05:22.739 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : zlib-devel-1.2.11-18.el8_5.x86_64 21/134 2023-07-14T19:05:22.795 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction check succeeded. 2023-07-14T19:05:22.795 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction test 2023-07-14T19:05:22.906 INFO:teuthology.orchestra.run.smithi040.stdout:(128/131): librados2-18.0.0-4940.g81c9387d.el8. 3.3 MB/s | 3.7 MB 00:01 2023-07-14T19:05:22.975 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 22/134 2023-07-14T19:05:23.073 INFO:teuthology.orchestra.run.smithi040.stdout:(129/131): iproute-5.15.0-4.el8_6.1.x86_64.rpm 4.7 MB/s | 798 kB 00:00 2023-07-14T19:05:23.182 INFO:teuthology.orchestra.run.smithi040.stdout:(130/131): librbd1-18.0.0-4940.g81c9387d.el8.x8 2.7 MB/s | 3.6 MB 00:01 2023-07-14T19:05:23.186 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 23/134 2023-07-14T19:05:23.395 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-pytz-2017.2-9.el8.noarch 24/134 2023-07-14T19:05:23.659 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 25/134 2023-07-14T19:05:23.812 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-ceph-common-2:18.0.0-4940.g81c9387d.el8. 26/134 2023-07-14T19:05:23.897 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction test succeeded. 2023-07-14T19:05:23.901 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction 2023-07-14T19:05:23.932 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-ceph-argparse-2:18.0.0-4940.g81c9387d.el 27/134 2023-07-14T19:05:24.879 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-cephfs-2:18.0.0-4940.g81c9387d.el8.x86_6 28/134 2023-07-14T19:05:25.176 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 29/134 2023-07-14T19:05:25.482 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-jinja2-2.10.1-3.el8.noarch 30/134 2023-07-14T19:05:25.786 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-mako-1.0.6-13.el8.noarch 31/134 2023-07-14T19:05:25.979 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 32/134 2023-07-14T19:05:27.755 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 32/134 2023-07-14T19:05:28.081 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-scipy-1.0.0-21.module+el8.5.0+10916+41bd 33/134 2023-07-14T19:05:28.316 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-rsa-4.8-1.el8.noarch 34/134 2023-07-14T19:05:28.514 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 35/134 2023-07-14T19:05:28.703 INFO:teuthology.orchestra.run.smithi191.stdout: Preparing : 1/1 2023-07-14T19:05:28.743 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-rbd-2:18.0.0-4940.g81c9387d.el8.x86_64 36/134 2023-07-14T19:05:28.896 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : thrift-0.13.0-2.el8.x86_64 1/134 2023-07-14T19:05:28.899 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : librados-devel-2:18.0.0-4940.g81c9387d.el8.x86_6 37/134 2023-07-14T19:05:29.078 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : liboath-2.6.2-3.el8.x86_64 2/134 2023-07-14T19:05:29.094 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : libcom_err-devel-1.45.6-4.el8.x86_64 38/134 2023-07-14T19:05:29.248 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : librdkafka-0.11.4-3.el8.x86_64 3/134 2023-07-14T19:05:29.287 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : libuuid-devel-2.32.1-35.el8.x86_64 39/134 2023-07-14T19:05:29.355 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: librdkafka-0.11.4-3.el8.x86_64 3/134 2023-07-14T19:05:29.506 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : fuse-2.9.7-15.el8.x86_64 40/134 2023-07-14T19:05:29.606 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 4/134 2023-07-14T19:05:29.632 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : libverto-devel-0.3.0-5.el8.x86_64 41/134 2023-07-14T19:05:29.859 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : pcre2-utf16-10.32-2.el8.x86_64 42/134 2023-07-14T19:05:29.988 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 4/134 2023-07-14T19:05:30.074 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 43/134 2023-07-14T19:05:30.197 INFO:teuthology.orchestra.run.smithi191.stdout: Upgrading : librados2-2:18.0.0-4940.g81c9387d.el8.x86_64 5/134 2023-07-14T19:05:30.429 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-cffi-1.11.5-5.el8.x86_64 44/134 2023-07-14T19:05:30.595 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: librados2-2:18.0.0-4940.g81c9387d.el8.x86_64 5/134 2023-07-14T19:05:30.698 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-cryptography-3.2.1-5.el8.x86_64 45/134 2023-07-14T19:05:30.772 INFO:teuthology.orchestra.run.smithi191.stdout: Upgrading : librbd1-2:18.0.0-4940.g81c9387d.el8.x86_64 6/134 2023-07-14T19:05:30.899 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: librbd1-2:18.0.0-4940.g81c9387d.el8.x86_64 6/134 2023-07-14T19:05:30.938 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 46/134 2023-07-14T19:05:31.105 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : libcephfs2-2:18.0.0-4940.g81c9387d.el8.x86_64 7/134 2023-07-14T19:05:31.167 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 47/134 2023-07-14T19:05:31.211 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: libcephfs2-2:18.0.0-4940.g81c9387d.el8.x86_64 7/134 2023-07-14T19:05:31.377 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-rados-2:18.0.0-4940.g81c9387d.el8.x86_64 8/134 2023-07-14T19:05:31.425 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 48/134 2023-07-14T19:05:31.547 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : librabbitmq-0.9.0-3.el8.x86_64 9/134 2023-07-14T19:05:31.761 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 49/134 2023-07-14T19:05:31.984 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : libpq-13.5-1.el8.x86_64 10/134 2023-07-14T19:05:32.011 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-asyncssh-2.7.0-2.el8.noarch 50/134 2023-07-14T19:05:32.191 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 11/134 2023-07-14T19:05:32.204 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 51/134 2023-07-14T19:05:32.450 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 52/134 2023-07-14T19:05:32.582 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 12/134 2023-07-14T19:05:32.607 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : libsepol-devel-2.9-3.el8.x86_64 53/134 2023-07-14T19:05:32.788 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : keyutils-libs-devel-1.5.10-9.el8.x86_64 54/134 2023-07-14T19:05:32.850 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : postgresql-10.21-2.module+el8.6.0+15342+53518fac 13/134 2023-07-14T19:05:32.988 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : libcephsqlite-2:18.0.0-4940.g81c9387d.el8.x86_64 14/134 2023-07-14T19:05:33.101 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: libcephsqlite-2:18.0.0-4940.g81c9387d.el8.x86_64 14/134 2023-07-14T19:05:33.110 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : pcre2-utf32-10.32-2.el8.x86_64 55/134 2023-07-14T19:05:33.255 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : libradosstriper1-2:18.0.0-4940.g81c9387d.el8.x86 15/134 2023-07-14T19:05:33.403 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: libradosstriper1-2:18.0.0-4940.g81c9387d.el8.x86 15/134 2023-07-14T19:05:33.409 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : pcre2-devel-10.32-2.el8.x86_64 56/134 2023-07-14T19:05:33.617 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 16/134 2023-07-14T19:05:33.633 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : libselinux-devel-2.9-5.el8.x86_64 57/134 2023-07-14T19:05:33.797 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 17/134 2023-07-14T19:05:33.868 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : krb5-devel-1.18.2-14.el8.x86_64 58/134 2023-07-14T19:05:33.971 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-pytz-2017.2-9.el8.noarch 18/134 2023-07-14T19:05:34.044 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 59/134 2023-07-14T19:05:34.088 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : libquadmath-8.5.0-10.1.el8_6.x86_64 19/134 2023-07-14T19:05:34.226 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 60/134 2023-07-14T19:05:34.259 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: libquadmath-8.5.0-10.1.el8_6.x86_64 19/134 2023-07-14T19:05:34.386 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : uuid-1.6.2-43.el8.x86_64 61/134 2023-07-14T19:05:34.413 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : libgfortran-8.5.0-10.1.el8_6.x86_64 20/134 2023-07-14T19:05:34.439 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: uuid-1.6.2-43.el8.x86_64 61/134 2023-07-14T19:05:34.723 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 62/134 2023-07-14T19:05:34.906 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : automake-1.16.1-7.el8.noarch 63/134 2023-07-14T19:05:35.009 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: libgfortran-8.5.0-10.1.el8_6.x86_64 20/134 2023-07-14T19:05:35.078 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : oniguruma-6.8.2-2.el8.x86_64 64/134 2023-07-14T19:05:35.160 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: oniguruma-6.8.2-2.el8.x86_64 64/134 2023-07-14T19:05:35.205 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : openblas-threads-0.3.15-3.el8.x86_64 21/134 2023-07-14T19:05:35.407 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : jq-1.6-3.el8.x86_64 65/134 2023-07-14T19:05:35.408 INFO:teuthology.orchestra.run.smithi040.stdout:(131/131): thrift-0.13.0-2.el8.x86_64.rpm 97 kB/s | 1.7 MB 00:18 2023-07-14T19:05:35.411 INFO:teuthology.orchestra.run.smithi040.stdout:-------------------------------------------------------------------------------- 2023-07-14T19:05:35.411 INFO:teuthology.orchestra.run.smithi040.stdout:Total 4.5 MB/s | 206 MB 00:45 2023-07-14T19:05:35.553 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 66/134 2023-07-14T19:05:35.685 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 67/134 2023-07-14T19:05:35.800 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: openblas-threads-0.3.15-3.el8.x86_64 21/134 2023-07-14T19:05:35.897 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 68/134 2023-07-14T19:05:35.995 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : openblas-0.3.15-3.el8.x86_64 22/134 2023-07-14T19:05:36.145 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 69/134 2023-07-14T19:05:36.249 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction check 2023-07-14T19:05:36.383 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction check succeeded. 2023-07-14T19:05:36.383 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction test 2023-07-14T19:05:36.393 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 70/134 2023-07-14T19:05:36.540 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 71/134 2023-07-14T19:05:36.687 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: openblas-0.3.15-3.el8.x86_64 22/134 2023-07-14T19:05:36.721 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 72/134 2023-07-14T19:05:36.903 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 73/134 2023-07-14T19:05:36.919 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 23/134 2023-07-14T19:05:37.101 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : zlib-devel-1.2.11-18.el8_5.x86_64 24/134 2023-07-14T19:05:37.114 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 74/134 2023-07-14T19:05:37.288 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-natsort-7.1.1-2.el8.noarch 75/134 2023-07-14T19:05:37.356 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 25/134 2023-07-14T19:05:37.447 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 76/134 2023-07-14T19:05:37.511 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-ceph-common-2:18.0.0-4940.g81c9387d.el8. 26/134 2023-07-14T19:05:37.574 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction test succeeded. 2023-07-14T19:05:37.579 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction 2023-07-14T19:05:37.617 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 77/134 2023-07-14T19:05:37.623 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-ceph-argparse-2:18.0.0-4940.g81c9387d.el 27/134 2023-07-14T19:05:37.827 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 78/134 2023-07-14T19:05:37.876 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-cephfs-2:18.0.0-4940.g81c9387d.el8.x86_6 28/134 2023-07-14T19:05:38.082 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 79/134 2023-07-14T19:05:38.104 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 29/134 2023-07-14T19:05:38.259 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 80/134 2023-07-14T19:05:38.508 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-portend-2.6-1.el8.noarch 81/134 2023-07-14T19:05:38.693 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 82/134 2023-07-14T19:05:38.893 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 83/134 2023-07-14T19:05:39.119 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 84/134 2023-07-14T19:05:39.866 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 29/134 2023-07-14T19:05:40.249 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 85/134 2023-07-14T19:05:40.564 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 86/134 2023-07-14T19:05:40.810 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 87/134 2023-07-14T19:05:40.863 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-scipy-1.0.0-21.module+el8.5.0+10916+41bd 30/134 2023-07-14T19:05:41.162 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 88/134 2023-07-14T19:05:41.175 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 31/134 2023-07-14T19:05:41.430 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-jinja2-2.10.1-3.el8.noarch 32/134 2023-07-14T19:05:41.453 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 89/134 2023-07-14T19:05:41.643 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 90/134 2023-07-14T19:05:41.692 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-mako-1.0.6-13.el8.noarch 33/134 2023-07-14T19:05:41.914 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-rsa-4.8-1.el8.noarch 34/134 2023-07-14T19:05:42.105 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 35/134 2023-07-14T19:05:42.284 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-rbd-2:18.0.0-4940.g81c9387d.el8.x86_64 36/134 2023-07-14T19:05:42.451 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : librados-devel-2:18.0.0-4940.g81c9387d.el8.x86_6 37/134 2023-07-14T19:05:42.473 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 91/134 2023-07-14T19:05:42.539 INFO:teuthology.orchestra.run.smithi040.stdout: Preparing : 1/1 2023-07-14T19:05:42.610 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 38/134 2023-07-14T19:05:42.668 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : librgw2-2:18.0.0-4940.g81c9387d.el8.x86_64 92/134 2023-07-14T19:05:42.728 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : thrift-0.13.0-2.el8.x86_64 1/134 2023-07-14T19:05:42.741 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: librgw2-2:18.0.0-4940.g81c9387d.el8.x86_64 92/134 2023-07-14T19:05:42.796 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 39/134 2023-07-14T19:05:42.893 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-rgw-2:18.0.0-4940.g81c9387d.el8.x86_64 93/134 2023-07-14T19:05:42.905 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : liboath-2.6.2-3.el8.x86_64 2/134 2023-07-14T19:05:42.986 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : uuid-1.6.2-43.el8.x86_64 40/134 2023-07-14T19:05:43.053 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: uuid-1.6.2-43.el8.x86_64 40/134 2023-07-14T19:05:43.121 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : librdkafka-0.11.4-3.el8.x86_64 3/134 2023-07-14T19:05:43.288 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 41/134 2023-07-14T19:05:43.356 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: librdkafka-0.11.4-3.el8.x86_64 3/134 2023-07-14T19:05:43.532 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : automake-1.16.1-7.el8.noarch 42/134 2023-07-14T19:05:43.588 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 4/134 2023-07-14T19:05:43.695 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : oniguruma-6.8.2-2.el8.x86_64 43/134 2023-07-14T19:05:43.774 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: oniguruma-6.8.2-2.el8.x86_64 43/134 2023-07-14T19:05:43.982 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : jq-1.6-3.el8.x86_64 44/134 2023-07-14T19:05:44.018 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 4/134 2023-07-14T19:05:44.164 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 45/134 2023-07-14T19:05:44.221 INFO:teuthology.orchestra.run.smithi040.stdout: Upgrading : librados2-2:18.0.0-4940.g81c9387d.el8.x86_64 5/134 2023-07-14T19:05:44.368 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : libcom_err-devel-1.45.6-4.el8.x86_64 46/134 2023-07-14T19:05:44.521 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : libuuid-devel-2.32.1-35.el8.x86_64 47/134 2023-07-14T19:05:44.642 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: librados2-2:18.0.0-4940.g81c9387d.el8.x86_64 5/134 2023-07-14T19:05:44.680 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : fuse-2.9.7-15.el8.x86_64 48/134 2023-07-14T19:05:44.821 INFO:teuthology.orchestra.run.smithi040.stdout: Upgrading : librbd1-2:18.0.0-4940.g81c9387d.el8.x86_64 6/134 2023-07-14T19:05:44.831 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : libverto-devel-0.3.0-5.el8.x86_64 49/134 2023-07-14T19:05:44.954 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: librbd1-2:18.0.0-4940.g81c9387d.el8.x86_64 6/134 2023-07-14T19:05:45.094 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : pcre2-utf16-10.32-2.el8.x86_64 50/134 2023-07-14T19:05:45.121 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : libcephfs2-2:18.0.0-4940.g81c9387d.el8.x86_64 7/134 2023-07-14T19:05:45.223 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: libcephfs2-2:18.0.0-4940.g81c9387d.el8.x86_64 7/134 2023-07-14T19:05:45.301 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 51/134 2023-07-14T19:05:45.426 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-rados-2:18.0.0-4940.g81c9387d.el8.x86_64 8/134 2023-07-14T19:05:45.668 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-cffi-1.11.5-5.el8.x86_64 52/134 2023-07-14T19:05:45.713 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : librabbitmq-0.9.0-3.el8.x86_64 9/134 2023-07-14T19:05:45.956 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : libpq-13.5-1.el8.x86_64 10/134 2023-07-14T19:05:46.021 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-cryptography-3.2.1-5.el8.x86_64 53/134 2023-07-14T19:05:46.142 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 11/134 2023-07-14T19:05:46.238 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 54/134 2023-07-14T19:05:46.466 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 55/134 2023-07-14T19:05:46.560 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: ceph-common-2:18.0.0-4940.g81c9387d.el8.x86_64 94/134 2023-07-14T19:05:46.590 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 12/134 2023-07-14T19:05:46.715 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 56/134 2023-07-14T19:05:46.768 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : ceph-common-2:18.0.0-4940.g81c9387d.el8.x86_64 94/134 2023-07-14T19:05:46.892 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : postgresql-10.21-2.module+el8.6.0+15342+53518fac 13/134 2023-07-14T19:05:47.029 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : libcephsqlite-2:18.0.0-4940.g81c9387d.el8.x86_64 14/134 2023-07-14T19:05:47.035 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 57/134 2023-07-14T19:05:47.140 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: libcephsqlite-2:18.0.0-4940.g81c9387d.el8.x86_64 14/134 2023-07-14T19:05:47.235 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-asyncssh-2.7.0-2.el8.noarch 58/134 2023-07-14T19:05:47.338 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : libradosstriper1-2:18.0.0-4940.g81c9387d.el8.x86 15/134 2023-07-14T19:05:47.348 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: ceph-common-2:18.0.0-4940.g81c9387d.el8.x86_64 94/134 2023-07-14T19:05:47.436 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 59/134 2023-07-14T19:05:47.495 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: libradosstriper1-2:18.0.0-4940.g81c9387d.el8.x86 15/134 2023-07-14T19:05:47.640 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 60/134 2023-07-14T19:05:47.746 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 16/134 2023-07-14T19:05:47.806 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : libsepol-devel-2.9-3.el8.x86_64 61/134 2023-07-14T19:05:47.937 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 17/134 2023-07-14T19:05:47.989 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : keyutils-libs-devel-1.5.10-9.el8.x86_64 62/134 2023-07-14T19:05:48.111 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-pytz-2017.2-9.el8.noarch 18/134 2023-07-14T19:05:48.270 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : libquadmath-8.5.0-10.1.el8_6.x86_64 19/134 2023-07-14T19:05:48.321 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : ceph-base-2:18.0.0-4940.g81c9387d.el8.x86_64 95/134 2023-07-14T19:05:48.328 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : pcre2-utf32-10.32-2.el8.x86_64 63/134 2023-07-14T19:05:48.398 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: ceph-base-2:18.0.0-4940.g81c9387d.el8.x86_64 95/134 2023-07-14T19:05:48.408 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: libquadmath-8.5.0-10.1.el8_6.x86_64 19/134 2023-07-14T19:05:48.534 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : ceph-selinux-2:18.0.0-4940.g81c9387d.el8.x86_64 96/134 2023-07-14T19:05:48.571 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : libgfortran-8.5.0-10.1.el8_6.x86_64 20/134 2023-07-14T19:05:48.589 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : pcre2-devel-10.32-2.el8.x86_64 64/134 2023-07-14T19:05:48.823 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : libselinux-devel-2.9-5.el8.x86_64 65/134 2023-07-14T19:05:49.070 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : krb5-devel-1.18.2-14.el8.x86_64 66/134 2023-07-14T19:05:49.218 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 67/134 2023-07-14T19:05:49.259 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: libgfortran-8.5.0-10.1.el8_6.x86_64 20/134 2023-07-14T19:05:49.387 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 68/134 2023-07-14T19:05:49.479 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : openblas-threads-0.3.15-3.el8.x86_64 21/134 2023-07-14T19:05:49.651 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 69/134 2023-07-14T19:05:49.867 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 70/134 2023-07-14T19:05:50.014 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 71/134 2023-07-14T19:05:50.142 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: openblas-threads-0.3.15-3.el8.x86_64 21/134 2023-07-14T19:05:50.154 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 72/134 2023-07-14T19:05:50.321 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : openblas-0.3.15-3.el8.x86_64 22/134 2023-07-14T19:05:50.343 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 73/134 2023-07-14T19:05:50.579 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 74/134 2023-07-14T19:05:50.745 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-natsort-7.1.1-2.el8.noarch 75/134 2023-07-14T19:05:50.904 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 76/134 2023-07-14T19:05:51.040 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 77/134 2023-07-14T19:05:51.060 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: openblas-0.3.15-3.el8.x86_64 22/134 2023-07-14T19:05:51.285 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 23/134 2023-07-14T19:05:51.303 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 78/134 2023-07-14T19:05:51.499 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : zlib-devel-1.2.11-18.el8_5.x86_64 24/134 2023-07-14T19:05:51.523 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 79/134 2023-07-14T19:05:51.716 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 80/134 2023-07-14T19:05:51.853 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 25/134 2023-07-14T19:05:51.967 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-portend-2.6-1.el8.noarch 81/134 2023-07-14T19:05:51.986 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-ceph-common-2:18.0.0-4940.g81c9387d.el8. 26/134 2023-07-14T19:05:52.091 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-ceph-argparse-2:18.0.0-4940.g81c9387d.el 27/134 2023-07-14T19:05:52.184 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 82/134 2023-07-14T19:05:52.344 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-cephfs-2:18.0.0-4940.g81c9387d.el8.x86_6 28/134 2023-07-14T19:05:52.350 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 83/134 2023-07-14T19:05:52.545 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 29/134 2023-07-14T19:05:52.567 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 84/134 2023-07-14T19:05:53.745 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 85/134 2023-07-14T19:05:54.021 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 86/134 2023-07-14T19:05:54.309 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 87/134 2023-07-14T19:05:54.425 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 29/134 2023-07-14T19:05:54.626 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 88/134 2023-07-14T19:05:54.934 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 89/134 2023-07-14T19:05:55.187 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 90/134 2023-07-14T19:05:55.500 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-scipy-1.0.0-21.module+el8.5.0+10916+41bd 30/134 2023-07-14T19:05:55.818 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 31/134 2023-07-14T19:05:55.960 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 91/134 2023-07-14T19:05:56.076 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-jinja2-2.10.1-3.el8.noarch 32/134 2023-07-14T19:05:56.134 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : librgw2-2:18.0.0-4940.g81c9387d.el8.x86_64 92/134 2023-07-14T19:05:56.213 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: librgw2-2:18.0.0-4940.g81c9387d.el8.x86_64 92/134 2023-07-14T19:05:56.304 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-mako-1.0.6-13.el8.noarch 33/134 2023-07-14T19:05:56.401 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-rgw-2:18.0.0-4940.g81c9387d.el8.x86_64 93/134 2023-07-14T19:05:56.527 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-rsa-4.8-1.el8.noarch 34/134 2023-07-14T19:05:56.716 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 35/134 2023-07-14T19:05:56.901 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-rbd-2:18.0.0-4940.g81c9387d.el8.x86_64 36/134 2023-07-14T19:05:57.228 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : librados-devel-2:18.0.0-4940.g81c9387d.el8.x86_6 37/134 2023-07-14T19:05:57.412 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 38/134 2023-07-14T19:05:57.564 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 39/134 2023-07-14T19:05:57.736 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : uuid-1.6.2-43.el8.x86_64 40/134 2023-07-14T19:05:57.813 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: uuid-1.6.2-43.el8.x86_64 40/134 2023-07-14T19:05:58.079 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 41/134 2023-07-14T19:05:58.275 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : automake-1.16.1-7.el8.noarch 42/134 2023-07-14T19:05:58.444 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : oniguruma-6.8.2-2.el8.x86_64 43/134 2023-07-14T19:05:58.508 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: oniguruma-6.8.2-2.el8.x86_64 43/134 2023-07-14T19:05:58.695 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : jq-1.6-3.el8.x86_64 44/134 2023-07-14T19:05:58.899 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 45/134 2023-07-14T19:05:59.061 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : libcom_err-devel-1.45.6-4.el8.x86_64 46/134 2023-07-14T19:05:59.281 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : libuuid-devel-2.32.1-35.el8.x86_64 47/134 2023-07-14T19:05:59.464 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : fuse-2.9.7-15.el8.x86_64 48/134 2023-07-14T19:05:59.616 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : libverto-devel-0.3.0-5.el8.x86_64 49/134 2023-07-14T19:05:59.888 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : pcre2-utf16-10.32-2.el8.x86_64 50/134 2023-07-14T19:06:00.052 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: ceph-common-2:18.0.0-4940.g81c9387d.el8.x86_64 94/134 2023-07-14T19:06:00.097 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 51/134 2023-07-14T19:06:00.259 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : ceph-common-2:18.0.0-4940.g81c9387d.el8.x86_64 94/134 2023-07-14T19:06:00.542 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-cffi-1.11.5-5.el8.x86_64 52/134 2023-07-14T19:06:00.843 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: ceph-common-2:18.0.0-4940.g81c9387d.el8.x86_64 94/134 2023-07-14T19:06:00.872 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-cryptography-3.2.1-5.el8.x86_64 53/134 2023-07-14T19:06:01.121 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : ceph-base-2:18.0.0-4940.g81c9387d.el8.x86_64 95/134 2023-07-14T19:06:01.142 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 54/134 2023-07-14T19:06:01.204 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: ceph-base-2:18.0.0-4940.g81c9387d.el8.x86_64 95/134 2023-07-14T19:06:01.364 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 55/134 2023-07-14T19:06:01.392 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : ceph-selinux-2:18.0.0-4940.g81c9387d.el8.x86_64 96/134 2023-07-14T19:06:01.627 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 56/134 2023-07-14T19:06:01.966 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 57/134 2023-07-14T19:06:02.211 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-asyncssh-2.7.0-2.el8.noarch 58/134 2023-07-14T19:06:02.438 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 59/134 2023-07-14T19:06:02.705 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 60/134 2023-07-14T19:06:02.860 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : libsepol-devel-2.9-3.el8.x86_64 61/134 2023-07-14T19:06:03.049 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : keyutils-libs-devel-1.5.10-9.el8.x86_64 62/134 2023-07-14T19:06:03.361 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : pcre2-utf32-10.32-2.el8.x86_64 63/134 2023-07-14T19:06:03.674 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : pcre2-devel-10.32-2.el8.x86_64 64/134 2023-07-14T19:06:03.910 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : libselinux-devel-2.9-5.el8.x86_64 65/134 2023-07-14T19:06:04.129 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : krb5-devel-1.18.2-14.el8.x86_64 66/134 2023-07-14T19:06:04.294 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 67/134 2023-07-14T19:06:04.507 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 68/134 2023-07-14T19:06:04.744 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 69/134 2023-07-14T19:06:04.968 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 70/134 2023-07-14T19:06:05.115 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 71/134 2023-07-14T19:06:05.264 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 72/134 2023-07-14T19:06:05.482 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 73/134 2023-07-14T19:06:06.067 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 74/134 2023-07-14T19:06:06.232 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-natsort-7.1.1-2.el8.noarch 75/134 2023-07-14T19:06:06.397 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 76/134 2023-07-14T19:06:06.533 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 77/134 2023-07-14T19:06:06.783 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 78/134 2023-07-14T19:06:06.999 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 79/134 2023-07-14T19:06:07.201 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 80/134 2023-07-14T19:06:07.439 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-portend-2.6-1.el8.noarch 81/134 2023-07-14T19:06:07.468 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: ceph-selinux-2:18.0.0-4940.g81c9387d.el8.x86_64 96/134 2023-07-14T19:06:07.468 INFO:teuthology.orchestra.run.smithi130.stdout:skipping the directory /sys 2023-07-14T19:06:07.468 INFO:teuthology.orchestra.run.smithi130.stdout:skipping the directory /proc 2023-07-14T19:06:07.469 INFO:teuthology.orchestra.run.smithi130.stdout:skipping the directory /dev 2023-07-14T19:06:07.469 INFO:teuthology.orchestra.run.smithi130.stdout:skipping the directory /run 2023-07-14T19:06:07.469 INFO:teuthology.orchestra.run.smithi130.stdout:skipping the directory /mnt 2023-07-14T19:06:07.469 INFO:teuthology.orchestra.run.smithi130.stdout:skipping the directory /var/tmp 2023-07-14T19:06:07.469 INFO:teuthology.orchestra.run.smithi130.stdout:skipping the directory /home 2023-07-14T19:06:07.469 INFO:teuthology.orchestra.run.smithi130.stdout:skipping the directory /tmp 2023-07-14T19:06:07.469 INFO:teuthology.orchestra.run.smithi130.stdout:skipping the directory /dev 2023-07-14T19:06:07.470 INFO:teuthology.orchestra.run.smithi130.stdout: 2023-07-14T19:06:07.677 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 82/134 2023-07-14T19:06:07.809 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : ceph-volume-2:18.0.0-4940.g81c9387d.el8.noarch 97/134 2023-07-14T19:06:07.836 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 83/134 2023-07-14T19:06:08.065 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 84/134 2023-07-14T19:06:09.267 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 85/134 2023-07-14T19:06:09.388 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: ceph-volume-2:18.0.0-4940.g81c9387d.el8.noarch 97/134 2023-07-14T19:06:09.558 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 86/134 2023-07-14T19:06:09.575 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : ceph-osd-2:18.0.0-4940.g81c9387d.el8.x86_64 98/134 2023-07-14T19:06:09.770 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 87/134 2023-07-14T19:06:09.794 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: ceph-osd-2:18.0.0-4940.g81c9387d.el8.x86_64 98/134 2023-07-14T19:06:09.964 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : ceph-mds-2:18.0.0-4940.g81c9387d.el8.x86_64 99/134 2023-07-14T19:06:10.063 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 88/134 2023-07-14T19:06:10.335 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 89/134 2023-07-14T19:06:10.418 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: ceph-mds-2:18.0.0-4940.g81c9387d.el8.x86_64 99/134 2023-07-14T19:06:10.586 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 90/134 2023-07-14T19:06:10.614 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : ceph-mon-2:18.0.0-4940.g81c9387d.el8.x86_64 100/134 2023-07-14T19:06:10.631 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: ceph-mon-2:18.0.0-4940.g81c9387d.el8.x86_64 100/134 2023-07-14T19:06:11.429 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 91/134 2023-07-14T19:06:11.626 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : librgw2-2:18.0.0-4940.g81c9387d.el8.x86_64 92/134 2023-07-14T19:06:11.696 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: librgw2-2:18.0.0-4940.g81c9387d.el8.x86_64 92/134 2023-07-14T19:06:11.909 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-rgw-2:18.0.0-4940.g81c9387d.el8.x86_64 93/134 2023-07-14T19:06:12.377 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: cephadm-2:18.0.0-4940.g81c9387d.el8.noarch 101/134 2023-07-14T19:06:12.536 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : cephadm-2:18.0.0-4940.g81c9387d.el8.noarch 101/134 2023-07-14T19:06:12.695 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : ceph-prometheus-alerts-2:18.0.0-4940.g81c9387d.e 102/134 2023-07-14T19:06:12.896 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : ceph-grafana-dashboards-2:18.0.0-4940.g81c9387d. 103/134 2023-07-14T19:06:13.113 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : ceph-mgr-cephadm-2:18.0.0-4940.g81c9387d.el8.noa 104/134 2023-07-14T19:06:13.325 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: ceph-mgr-cephadm-2:18.0.0-4940.g81c9387d.el8.noa 104/134 2023-07-14T19:06:13.566 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : ceph-mgr-dashboard-2:18.0.0-4940.g81c9387d.el8.n 105/134 2023-07-14T19:06:14.525 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: ceph-mgr-dashboard-2:18.0.0-4940.g81c9387d.el8.n 105/134 2023-07-14T19:06:14.705 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : ceph-mgr-diskprediction-local-2:18.0.0-4940.g81c 106/134 2023-07-14T19:06:14.796 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:18.0.0-4940.g81c 106/134 2023-07-14T19:06:15.176 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : ceph-mgr-modules-core-2:18.0.0-4940.g81c9387d.el 107/134 2023-07-14T19:06:15.398 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : ceph-mgr-2:18.0.0-4940.g81c9387d.el8.x86_64 108/134 2023-07-14T19:06:15.433 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: ceph-mgr-2:18.0.0-4940.g81c9387d.el8.x86_64 108/134 2023-07-14T19:06:15.580 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : ceph-mgr-rook-2:18.0.0-4940.g81c9387d.el8.noarch 109/134 2023-07-14T19:06:15.591 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: ceph-mgr-rook-2:18.0.0-4940.g81c9387d.el8.noarch 109/134 2023-07-14T19:06:15.844 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: ceph-common-2:18.0.0-4940.g81c9387d.el8.x86_64 94/134 2023-07-14T19:06:16.092 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : ceph-common-2:18.0.0-4940.g81c9387d.el8.x86_64 94/134 2023-07-14T19:06:16.759 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: ceph-common-2:18.0.0-4940.g81c9387d.el8.x86_64 94/134 2023-07-14T19:06:16.945 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : ceph-2:18.0.0-4940.g81c9387d.el8.x86_64 110/134 2023-07-14T19:06:17.031 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : ceph-base-2:18.0.0-4940.g81c9387d.el8.x86_64 95/134 2023-07-14T19:06:17.103 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: ceph-base-2:18.0.0-4940.g81c9387d.el8.x86_64 95/134 2023-07-14T19:06:17.121 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : ceph-radosgw-2:18.0.0-4940.g81c9387d.el8.x86_64 111/134 2023-07-14T19:06:17.156 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: ceph-radosgw-2:18.0.0-4940.g81c9387d.el8.x86_64 111/134 2023-07-14T19:06:17.216 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : ceph-selinux-2:18.0.0-4940.g81c9387d.el8.x86_64 96/134 2023-07-14T19:06:17.338 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : ceph-immutable-object-cache-2:18.0.0-4940.g81c93 112/134 2023-07-14T19:06:17.380 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: ceph-immutable-object-cache-2:18.0.0-4940.g81c93 112/134 2023-07-14T19:06:17.521 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : cephfs-mirror-2:18.0.0-4940.g81c9387d.el8.x86_64 113/134 2023-07-14T19:06:17.873 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: cephfs-mirror-2:18.0.0-4940.g81c9387d.el8.x86_64 113/134 2023-07-14T19:06:18.030 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : rbd-mirror-2:18.0.0-4940.g81c9387d.el8.x86_64 114/134 2023-07-14T19:06:20.239 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: ceph-selinux-2:18.0.0-4940.g81c9387d.el8.x86_64 96/134 2023-07-14T19:06:20.239 INFO:teuthology.orchestra.run.smithi191.stdout:skipping the directory /sys 2023-07-14T19:06:20.239 INFO:teuthology.orchestra.run.smithi191.stdout:skipping the directory /proc 2023-07-14T19:06:20.239 INFO:teuthology.orchestra.run.smithi191.stdout:skipping the directory /dev 2023-07-14T19:06:20.239 INFO:teuthology.orchestra.run.smithi191.stdout:skipping the directory /run 2023-07-14T19:06:20.239 INFO:teuthology.orchestra.run.smithi191.stdout:skipping the directory /mnt 2023-07-14T19:06:20.240 INFO:teuthology.orchestra.run.smithi191.stdout:skipping the directory /var/tmp 2023-07-14T19:06:20.240 INFO:teuthology.orchestra.run.smithi191.stdout:skipping the directory /home 2023-07-14T19:06:20.240 INFO:teuthology.orchestra.run.smithi191.stdout:skipping the directory /tmp 2023-07-14T19:06:20.240 INFO:teuthology.orchestra.run.smithi191.stdout:skipping the directory /dev 2023-07-14T19:06:20.240 INFO:teuthology.orchestra.run.smithi191.stdout: 2023-07-14T19:06:20.588 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : ceph-volume-2:18.0.0-4940.g81c9387d.el8.noarch 97/134 2023-07-14T19:06:22.164 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: ceph-volume-2:18.0.0-4940.g81c9387d.el8.noarch 97/134 2023-07-14T19:06:22.375 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : ceph-osd-2:18.0.0-4940.g81c9387d.el8.x86_64 98/134 2023-07-14T19:06:22.581 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: rbd-mirror-2:18.0.0-4940.g81c9387d.el8.x86_64 114/134 2023-07-14T19:06:22.594 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: ceph-osd-2:18.0.0-4940.g81c9387d.el8.x86_64 98/134 2023-07-14T19:06:22.812 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : ceph-mds-2:18.0.0-4940.g81c9387d.el8.x86_64 99/134 2023-07-14T19:06:22.874 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : ceph-test-2:18.0.0-4940.g81c9387d.el8.x86_64 115/134 2023-07-14T19:06:23.037 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : libtool-2.4.6-25.el8.x86_64 116/134 2023-07-14T19:06:23.235 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: libtool-2.4.6-25.el8.x86_64 116/134 2023-07-14T19:06:23.265 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: ceph-mds-2:18.0.0-4940.g81c9387d.el8.x86_64 99/134 2023-07-14T19:06:23.446 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : ceph-mon-2:18.0.0-4940.g81c9387d.el8.x86_64 100/134 2023-07-14T19:06:23.464 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: ceph-mon-2:18.0.0-4940.g81c9387d.el8.x86_64 100/134 2023-07-14T19:06:24.103 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : postgresql-contrib-10.21-2.module+el8.6.0+15342+ 117/134 2023-07-14T19:06:24.427 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : openssl-devel-1:1.1.1k-7.el8_6.x86_64 118/134 2023-07-14T19:06:24.598 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : ceph-fuse-2:18.0.0-4940.g81c9387d.el8.x86_64 119/134 2023-07-14T19:06:24.791 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : xfsprogs-devel-5.0.0-10.el8.x86_64 120/134 2023-07-14T19:06:24.982 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : libcephfs-devel-2:18.0.0-4940.g81c9387d.el8.x86_ 121/134 2023-07-14T19:06:25.121 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : elfutils-libelf-devel-0.186-1.el8.x86_64 122/134 2023-07-14T19:06:25.151 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: cephadm-2:18.0.0-4940.g81c9387d.el8.noarch 101/134 2023-07-14T19:06:25.336 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : cephadm-2:18.0.0-4940.g81c9387d.el8.noarch 101/134 2023-07-14T19:06:25.468 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : ceph-prometheus-alerts-2:18.0.0-4940.g81c9387d.e 102/134 2023-07-14T19:06:25.709 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : ceph-grafana-dashboards-2:18.0.0-4940.g81c9387d. 103/134 2023-07-14T19:06:25.911 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : ceph-mgr-cephadm-2:18.0.0-4940.g81c9387d.el8.noa 104/134 2023-07-14T19:06:26.127 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: ceph-mgr-cephadm-2:18.0.0-4940.g81c9387d.el8.noa 104/134 2023-07-14T19:06:26.399 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : ceph-mgr-dashboard-2:18.0.0-4940.g81c9387d.el8.n 105/134 2023-07-14T19:06:27.308 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: postgresql-server-10.21-2.module+el8.6.0+15342+5 123/134 2023-07-14T19:06:27.357 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: ceph-mgr-dashboard-2:18.0.0-4940.g81c9387d.el8.n 105/134 2023-07-14T19:06:27.528 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : ceph-mgr-diskprediction-local-2:18.0.0-4940.g81c 106/134 2023-07-14T19:06:27.587 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : postgresql-server-10.21-2.module+el8.6.0+15342+5 123/134 2023-07-14T19:06:27.604 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: postgresql-server-10.21-2.module+el8.6.0+15342+5 123/134 2023-07-14T19:06:27.619 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:18.0.0-4940.g81c 106/134 2023-07-14T19:06:27.762 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : cephfs-top-2:18.0.0-4940.g81c9387d.el8.noarch 124/134 2023-07-14T19:06:28.036 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : ceph-mgr-modules-core-2:18.0.0-4940.g81c9387d.el 107/134 2023-07-14T19:06:28.052 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : rbd-fuse-2:18.0.0-4940.g81c9387d.el8.x86_64 125/134 2023-07-14T19:06:28.249 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : ceph-mgr-2:18.0.0-4940.g81c9387d.el8.x86_64 108/134 2023-07-14T19:06:28.284 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: ceph-mgr-2:18.0.0-4940.g81c9387d.el8.x86_64 108/134 2023-07-14T19:06:28.368 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : rbd-nbd-2:18.0.0-4940.g81c9387d.el8.x86_64 126/134 2023-07-14T19:06:28.445 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : ceph-mgr-rook-2:18.0.0-4940.g81c9387d.el8.noarch 109/134 2023-07-14T19:06:28.456 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: ceph-mgr-rook-2:18.0.0-4940.g81c9387d.el8.noarch 109/134 2023-07-14T19:06:28.730 INFO:teuthology.orchestra.run.smithi130.stdout: Upgrading : iproute-5.15.0-4.el8_6.1.x86_64 127/134 2023-07-14T19:06:28.878 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : NetworkManager-initscripts-updown-1:1.36.0-9.el8 128/134 2023-07-14T19:06:28.886 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: NetworkManager-initscripts-updown-1:1.36.0-9.el8 128/134 2023-07-14T19:06:29.081 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : libaio-devel-0.3.112-1.el8.x86_64 129/134 2023-07-14T19:06:29.286 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : flex-2.6.1-9.el8.x86_64 130/134 2023-07-14T19:06:29.367 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: flex-2.6.1-9.el8.x86_64 130/134 2023-07-14T19:06:29.594 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : bison-3.0.4-10.el8.x86_64 131/134 2023-07-14T19:06:29.607 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: bison-3.0.4-10.el8.x86_64 131/134 2023-07-14T19:06:29.607 INFO:teuthology.orchestra.run.smithi130.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 132/134 2023-07-14T19:06:29.765 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 132/134 2023-07-14T19:06:29.765 INFO:teuthology.orchestra.run.smithi130.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 133/134 2023-07-14T19:06:29.774 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : ceph-2:18.0.0-4940.g81c9387d.el8.x86_64 110/134 2023-07-14T19:06:29.970 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : ceph-radosgw-2:18.0.0-4940.g81c9387d.el8.x86_64 111/134 2023-07-14T19:06:30.005 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: ceph-radosgw-2:18.0.0-4940.g81c9387d.el8.x86_64 111/134 2023-07-14T19:06:30.008 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 133/134 2023-07-14T19:06:30.128 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : ceph-immutable-object-cache-2:18.0.0-4940.g81c93 112/134 2023-07-14T19:06:30.170 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: ceph-immutable-object-cache-2:18.0.0-4940.g81c93 112/134 2023-07-14T19:06:30.185 INFO:teuthology.orchestra.run.smithi130.stdout: Cleanup : iproute-5.15.0-4.el8.x86_64 134/134 2023-07-14T19:06:30.328 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : cephfs-mirror-2:18.0.0-4940.g81c9387d.el8.x86_64 113/134 2023-07-14T19:06:30.679 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: cephfs-mirror-2:18.0.0-4940.g81c9387d.el8.x86_64 113/134 2023-07-14T19:06:30.691 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: iproute-5.15.0-4.el8.x86_64 134/134 2023-07-14T19:06:30.692 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : ceph-2:18.0.0-4940.g81c9387d.el8.x86_64 1/134 2023-07-14T19:06:30.692 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : ceph-base-2:18.0.0-4940.g81c9387d.el8.x86_64 2/134 2023-07-14T19:06:30.692 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : ceph-common-2:18.0.0-4940.g81c9387d.el8.x86_64 3/134 2023-07-14T19:06:30.692 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : ceph-fuse-2:18.0.0-4940.g81c9387d.el8.x86_64 4/134 2023-07-14T19:06:30.692 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : ceph-immutable-object-cache-2:18.0.0-4940.g81c93 5/134 2023-07-14T19:06:30.692 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : ceph-mds-2:18.0.0-4940.g81c9387d.el8.x86_64 6/134 2023-07-14T19:06:30.693 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : ceph-mgr-2:18.0.0-4940.g81c9387d.el8.x86_64 7/134 2023-07-14T19:06:30.693 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : ceph-mon-2:18.0.0-4940.g81c9387d.el8.x86_64 8/134 2023-07-14T19:06:30.693 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : ceph-osd-2:18.0.0-4940.g81c9387d.el8.x86_64 9/134 2023-07-14T19:06:30.693 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : ceph-radosgw-2:18.0.0-4940.g81c9387d.el8.x86_64 10/134 2023-07-14T19:06:30.693 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : ceph-selinux-2:18.0.0-4940.g81c9387d.el8.x86_64 11/134 2023-07-14T19:06:30.693 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : ceph-test-2:18.0.0-4940.g81c9387d.el8.x86_64 12/134 2023-07-14T19:06:30.694 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : cephfs-mirror-2:18.0.0-4940.g81c9387d.el8.x86_64 13/134 2023-07-14T19:06:30.695 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : libcephfs-devel-2:18.0.0-4940.g81c9387d.el8.x86_ 14/134 2023-07-14T19:06:30.695 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : libcephfs2-2:18.0.0-4940.g81c9387d.el8.x86_64 15/134 2023-07-14T19:06:30.695 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : libcephsqlite-2:18.0.0-4940.g81c9387d.el8.x86_64 16/134 2023-07-14T19:06:30.695 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : librados-devel-2:18.0.0-4940.g81c9387d.el8.x86_6 17/134 2023-07-14T19:06:30.695 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : libradosstriper1-2:18.0.0-4940.g81c9387d.el8.x86 18/134 2023-07-14T19:06:30.695 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : librgw2-2:18.0.0-4940.g81c9387d.el8.x86_64 19/134 2023-07-14T19:06:30.696 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-ceph-argparse-2:18.0.0-4940.g81c9387d.el 20/134 2023-07-14T19:06:30.696 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-ceph-common-2:18.0.0-4940.g81c9387d.el8. 21/134 2023-07-14T19:06:30.696 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-cephfs-2:18.0.0-4940.g81c9387d.el8.x86_6 22/134 2023-07-14T19:06:30.696 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-rados-2:18.0.0-4940.g81c9387d.el8.x86_64 23/134 2023-07-14T19:06:30.696 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-rbd-2:18.0.0-4940.g81c9387d.el8.x86_64 24/134 2023-07-14T19:06:30.696 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-rgw-2:18.0.0-4940.g81c9387d.el8.x86_64 25/134 2023-07-14T19:06:30.696 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : rbd-fuse-2:18.0.0-4940.g81c9387d.el8.x86_64 26/134 2023-07-14T19:06:30.703 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : rbd-mirror-2:18.0.0-4940.g81c9387d.el8.x86_64 27/134 2023-07-14T19:06:30.703 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : rbd-nbd-2:18.0.0-4940.g81c9387d.el8.x86_64 28/134 2023-07-14T19:06:30.703 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : ceph-grafana-dashboards-2:18.0.0-4940.g81c9387d. 29/134 2023-07-14T19:06:30.704 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : ceph-mgr-cephadm-2:18.0.0-4940.g81c9387d.el8.noa 30/134 2023-07-14T19:06:30.704 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : ceph-mgr-dashboard-2:18.0.0-4940.g81c9387d.el8.n 31/134 2023-07-14T19:06:30.704 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : ceph-mgr-diskprediction-local-2:18.0.0-4940.g81c 32/134 2023-07-14T19:06:30.704 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : ceph-mgr-modules-core-2:18.0.0-4940.g81c9387d.el 33/134 2023-07-14T19:06:30.704 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : ceph-mgr-rook-2:18.0.0-4940.g81c9387d.el8.noarch 34/134 2023-07-14T19:06:30.704 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : ceph-prometheus-alerts-2:18.0.0-4940.g81c9387d.e 35/134 2023-07-14T19:06:30.704 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : ceph-volume-2:18.0.0-4940.g81c9387d.el8.noarch 36/134 2023-07-14T19:06:30.705 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : cephadm-2:18.0.0-4940.g81c9387d.el8.noarch 37/134 2023-07-14T19:06:30.705 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : cephfs-top-2:18.0.0-4940.g81c9387d.el8.noarch 38/134 2023-07-14T19:06:30.705 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 39/134 2023-07-14T19:06:30.705 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 40/134 2023-07-14T19:06:30.705 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 41/134 2023-07-14T19:06:30.705 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-asyncssh-2.7.0-2.el8.noarch 42/134 2023-07-14T19:06:30.705 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 43/134 2023-07-14T19:06:30.706 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 44/134 2023-07-14T19:06:30.706 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 45/134 2023-07-14T19:06:30.706 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 46/134 2023-07-14T19:06:30.706 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 47/134 2023-07-14T19:06:30.706 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 48/134 2023-07-14T19:06:30.706 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 49/134 2023-07-14T19:06:30.706 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 50/134 2023-07-14T19:06:30.706 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 51/134 2023-07-14T19:06:30.707 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 52/134 2023-07-14T19:06:30.707 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 53/134 2023-07-14T19:06:30.707 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 54/134 2023-07-14T19:06:30.707 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-natsort-7.1.1-2.el8.noarch 55/134 2023-07-14T19:06:30.707 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 56/134 2023-07-14T19:06:30.707 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-portend-2.6-1.el8.noarch 57/134 2023-07-14T19:06:30.707 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 58/134 2023-07-14T19:06:30.707 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 59/134 2023-07-14T19:06:30.708 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-rsa-4.8-1.el8.noarch 60/134 2023-07-14T19:06:30.708 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 61/134 2023-07-14T19:06:30.708 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 62/134 2023-07-14T19:06:30.708 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 63/134 2023-07-14T19:06:30.708 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 64/134 2023-07-14T19:06:30.708 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 65/134 2023-07-14T19:06:30.708 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 66/134 2023-07-14T19:06:30.709 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 67/134 2023-07-14T19:06:30.709 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 68/134 2023-07-14T19:06:30.709 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 69/134 2023-07-14T19:06:30.709 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : thrift-0.13.0-2.el8.x86_64 70/134 2023-07-14T19:06:30.709 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 71/134 2023-07-14T19:06:30.709 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-scipy-1.0.0-21.module+el8.5.0+10916+41bd 72/134 2023-07-14T19:06:30.709 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 73/134 2023-07-14T19:06:30.709 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-pytz-2017.2-9.el8.noarch 74/134 2023-07-14T19:06:30.710 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 75/134 2023-07-14T19:06:30.710 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 76/134 2023-07-14T19:06:30.710 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 77/134 2023-07-14T19:06:30.710 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : bison-3.0.4-10.el8.x86_64 78/134 2023-07-14T19:06:30.710 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-jinja2-2.10.1-3.el8.noarch 79/134 2023-07-14T19:06:30.710 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-mako-1.0.6-13.el8.noarch 80/134 2023-07-14T19:06:30.710 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : oniguruma-6.8.2-2.el8.x86_64 81/134 2023-07-14T19:06:30.710 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : automake-1.16.1-7.el8.noarch 82/134 2023-07-14T19:06:30.711 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 83/134 2023-07-14T19:06:30.711 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 84/134 2023-07-14T19:06:30.711 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : flex-2.6.1-9.el8.x86_64 85/134 2023-07-14T19:06:30.711 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 86/134 2023-07-14T19:06:30.711 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 87/134 2023-07-14T19:06:30.711 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 88/134 2023-07-14T19:06:30.711 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 89/134 2023-07-14T19:06:30.712 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : uuid-1.6.2-43.el8.x86_64 90/134 2023-07-14T19:06:30.712 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 91/134 2023-07-14T19:06:30.712 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 92/134 2023-07-14T19:06:30.712 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 93/134 2023-07-14T19:06:30.712 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : libtool-2.4.6-25.el8.x86_64 94/134 2023-07-14T19:06:30.712 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : openblas-threads-0.3.15-3.el8.x86_64 95/134 2023-07-14T19:06:30.712 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : libpq-13.5-1.el8.x86_64 96/134 2023-07-14T19:06:30.712 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : openblas-0.3.15-3.el8.x86_64 97/134 2023-07-14T19:06:30.713 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : jq-1.6-3.el8.x86_64 98/134 2023-07-14T19:06:30.713 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : librdkafka-0.11.4-3.el8.x86_64 99/134 2023-07-14T19:06:30.713 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : postgresql-server-10.21-2.module+el8.6.0+15342+5 100/134 2023-07-14T19:06:30.713 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : postgresql-contrib-10.21-2.module+el8.6.0+15342+ 101/134 2023-07-14T19:06:30.713 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : postgresql-10.21-2.module+el8.6.0+15342+53518fac 102/134 2023-07-14T19:06:30.713 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : librabbitmq-0.9.0-3.el8.x86_64 103/134 2023-07-14T19:06:30.713 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : libselinux-devel-2.9-5.el8.x86_64 104/134 2023-07-14T19:06:30.713 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 105/134 2023-07-14T19:06:30.714 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : pcre2-utf32-10.32-2.el8.x86_64 106/134 2023-07-14T19:06:30.714 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : keyutils-libs-devel-1.5.10-9.el8.x86_64 107/134 2023-07-14T19:06:30.714 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : libsepol-devel-2.9-3.el8.x86_64 108/134 2023-07-14T19:06:30.714 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 109/134 2023-07-14T19:06:30.714 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 110/134 2023-07-14T19:06:30.714 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : pcre2-utf16-10.32-2.el8.x86_64 111/134 2023-07-14T19:06:30.714 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : krb5-devel-1.18.2-14.el8.x86_64 112/134 2023-07-14T19:06:30.715 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : libaio-devel-0.3.112-1.el8.x86_64 113/134 2023-07-14T19:06:30.715 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 114/134 2023-07-14T19:06:30.715 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-cryptography-3.2.1-5.el8.x86_64 115/134 2023-07-14T19:06:30.715 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : libverto-devel-0.3.0-5.el8.x86_64 116/134 2023-07-14T19:06:30.715 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : pcre2-devel-10.32-2.el8.x86_64 117/134 2023-07-14T19:06:30.715 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-cffi-1.11.5-5.el8.x86_64 118/134 2023-07-14T19:06:30.715 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : openssl-devel-1:1.1.1k-7.el8_6.x86_64 119/134 2023-07-14T19:06:30.715 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : xfsprogs-devel-5.0.0-10.el8.x86_64 120/134 2023-07-14T19:06:30.716 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : fuse-2.9.7-15.el8.x86_64 121/134 2023-07-14T19:06:30.716 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : elfutils-libelf-devel-0.186-1.el8.x86_64 122/134 2023-07-14T19:06:30.716 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : libuuid-devel-2.32.1-35.el8.x86_64 123/134 2023-07-14T19:06:30.716 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : zlib-devel-1.2.11-18.el8_5.x86_64 124/134 2023-07-14T19:06:30.716 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : libcom_err-devel-1.45.6-4.el8.x86_64 125/134 2023-07-14T19:06:30.716 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : libgfortran-8.5.0-10.1.el8_6.x86_64 126/134 2023-07-14T19:06:30.716 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : libquadmath-8.5.0-10.1.el8_6.x86_64 127/134 2023-07-14T19:06:30.717 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : NetworkManager-initscripts-updown-1:1.36.0-9.el8 128/134 2023-07-14T19:06:30.717 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : librados2-2:18.0.0-4940.g81c9387d.el8.x86_64 129/134 2023-07-14T19:06:30.717 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 130/134 2023-07-14T19:06:30.717 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : librbd1-2:18.0.0-4940.g81c9387d.el8.x86_64 131/134 2023-07-14T19:06:30.717 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 132/134 2023-07-14T19:06:30.717 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : iproute-5.15.0-4.el8_6.1.x86_64 133/134 2023-07-14T19:06:30.836 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : rbd-mirror-2:18.0.0-4940.g81c9387d.el8.x86_64 114/134 2023-07-14T19:06:33.212 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : iproute-5.15.0-4.el8.x86_64 134/134 2023-07-14T19:06:33.213 INFO:teuthology.orchestra.run.smithi130.stdout:Last metadata expiration check: 0:02:04 ago on Fri 14 Jul 2023 07:04:29 PM UTC. 2023-07-14T19:06:35.397 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: rbd-mirror-2:18.0.0-4940.g81c9387d.el8.x86_64 114/134 2023-07-14T19:06:36.289 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : ceph-test-2:18.0.0-4940.g81c9387d.el8.x86_64 115/134 2023-07-14T19:06:36.650 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : openssl-devel-1:1.1.1k-7.el8_6.x86_64 116/134 2023-07-14T19:06:36.830 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : ceph-fuse-2:18.0.0-4940.g81c9387d.el8.x86_64 117/134 2023-07-14T19:06:37.073 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : xfsprogs-devel-5.0.0-10.el8.x86_64 118/134 2023-07-14T19:06:37.251 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : libtool-2.4.6-25.el8.x86_64 119/134 2023-07-14T19:06:37.400 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: ceph-selinux-2:18.0.0-4940.g81c9387d.el8.x86_64 96/134 2023-07-14T19:06:37.401 INFO:teuthology.orchestra.run.smithi040.stdout:skipping the directory /sys 2023-07-14T19:06:37.401 INFO:teuthology.orchestra.run.smithi040.stdout:skipping the directory /proc 2023-07-14T19:06:37.401 INFO:teuthology.orchestra.run.smithi040.stdout:skipping the directory /dev 2023-07-14T19:06:37.401 INFO:teuthology.orchestra.run.smithi040.stdout:skipping the directory /run 2023-07-14T19:06:37.401 INFO:teuthology.orchestra.run.smithi040.stdout:skipping the directory /mnt 2023-07-14T19:06:37.402 INFO:teuthology.orchestra.run.smithi040.stdout:skipping the directory /var/tmp 2023-07-14T19:06:37.402 INFO:teuthology.orchestra.run.smithi040.stdout:skipping the directory /home 2023-07-14T19:06:37.402 INFO:teuthology.orchestra.run.smithi040.stdout:skipping the directory /tmp 2023-07-14T19:06:37.402 INFO:teuthology.orchestra.run.smithi040.stdout:skipping the directory /dev 2023-07-14T19:06:37.403 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-07-14T19:06:37.445 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: libtool-2.4.6-25.el8.x86_64 119/134 2023-07-14T19:06:37.689 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : postgresql-contrib-10.21-2.module+el8.6.0+15342+ 120/134 2023-07-14T19:06:37.756 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : ceph-volume-2:18.0.0-4940.g81c9387d.el8.noarch 97/134 2023-07-14T19:06:37.880 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : libcephfs-devel-2:18.0.0-4940.g81c9387d.el8.x86_ 121/134 2023-07-14T19:06:38.019 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : elfutils-libelf-devel-0.186-1.el8.x86_64 122/134 2023-07-14T19:06:39.449 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: ceph-volume-2:18.0.0-4940.g81c9387d.el8.noarch 97/134 2023-07-14T19:06:39.594 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : ceph-osd-2:18.0.0-4940.g81c9387d.el8.x86_64 98/134 2023-07-14T19:06:39.825 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: ceph-osd-2:18.0.0-4940.g81c9387d.el8.x86_64 98/134 2023-07-14T19:06:40.030 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : ceph-mds-2:18.0.0-4940.g81c9387d.el8.x86_64 99/134 2023-07-14T19:06:40.222 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: postgresql-server-10.21-2.module+el8.6.0+15342+5 123/134 2023-07-14T19:06:40.476 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : postgresql-server-10.21-2.module+el8.6.0+15342+5 123/134 2023-07-14T19:06:40.493 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: postgresql-server-10.21-2.module+el8.6.0+15342+5 123/134 2023-07-14T19:06:40.513 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: ceph-mds-2:18.0.0-4940.g81c9387d.el8.x86_64 99/134 2023-07-14T19:06:40.660 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : cephfs-top-2:18.0.0-4940.g81c9387d.el8.noarch 124/134 2023-07-14T19:06:40.680 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : ceph-mon-2:18.0.0-4940.g81c9387d.el8.x86_64 100/134 2023-07-14T19:06:40.698 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: ceph-mon-2:18.0.0-4940.g81c9387d.el8.x86_64 100/134 2023-07-14T19:06:40.950 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : rbd-fuse-2:18.0.0-4940.g81c9387d.el8.x86_64 125/134 2023-07-14T19:06:41.233 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : rbd-nbd-2:18.0.0-4940.g81c9387d.el8.x86_64 126/134 2023-07-14T19:06:41.614 INFO:teuthology.orchestra.run.smithi191.stdout: Upgrading : iproute-5.15.0-4.el8_6.1.x86_64 127/134 2023-07-14T19:06:41.817 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : flex-2.6.1-9.el8.x86_64 128/134 2023-07-14T19:06:41.899 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: flex-2.6.1-9.el8.x86_64 128/134 2023-07-14T19:06:42.092 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : bison-3.0.4-10.el8.x86_64 129/134 2023-07-14T19:06:42.105 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: bison-3.0.4-10.el8.x86_64 129/134 2023-07-14T19:06:42.149 INFO:teuthology.orchestra.run.smithi130.stdout:Installed products updated. 2023-07-14T19:06:42.234 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : NetworkManager-initscripts-updown-1:1.36.0-9.el8 130/134 2023-07-14T19:06:42.242 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: NetworkManager-initscripts-updown-1:1.36.0-9.el8 130/134 2023-07-14T19:06:42.361 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : libaio-devel-0.3.112-1.el8.x86_64 131/134 2023-07-14T19:06:42.361 INFO:teuthology.orchestra.run.smithi191.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 132/134 2023-07-14T19:06:42.363 INFO:teuthology.orchestra.run.smithi130.stdout: 2023-07-14T19:06:42.363 INFO:teuthology.orchestra.run.smithi130.stdout:Upgraded: 2023-07-14T19:06:42.364 INFO:teuthology.orchestra.run.smithi130.stdout: iproute-5.15.0-4.el8_6.1.x86_64 2023-07-14T19:06:42.364 INFO:teuthology.orchestra.run.smithi130.stdout: librados2-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T19:06:42.364 INFO:teuthology.orchestra.run.smithi130.stdout: librbd1-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T19:06:42.364 INFO:teuthology.orchestra.run.smithi130.stdout:Installed: 2023-07-14T19:06:42.364 INFO:teuthology.orchestra.run.smithi130.stdout: NetworkManager-initscripts-updown-1:1.36.0-9.el8_6.noarch 2023-07-14T19:06:42.364 INFO:teuthology.orchestra.run.smithi130.stdout: automake-1.16.1-7.el8.noarch 2023-07-14T19:06:42.365 INFO:teuthology.orchestra.run.smithi130.stdout: bison-3.0.4-10.el8.x86_64 2023-07-14T19:06:42.365 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T19:06:42.365 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-base-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T19:06:42.365 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-common-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T19:06:42.365 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-fuse-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T19:06:42.365 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-grafana-dashboards-2:18.0.0-4940.g81c9387d.el8.noarch 2023-07-14T19:06:42.365 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-immutable-object-cache-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T19:06:42.365 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-mds-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T19:06:42.366 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-mgr-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T19:06:42.366 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-mgr-cephadm-2:18.0.0-4940.g81c9387d.el8.noarch 2023-07-14T19:06:42.366 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-mgr-dashboard-2:18.0.0-4940.g81c9387d.el8.noarch 2023-07-14T19:06:42.366 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-mgr-diskprediction-local-2:18.0.0-4940.g81c9387d.el8.noarch 2023-07-14T19:06:42.366 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-mgr-modules-core-2:18.0.0-4940.g81c9387d.el8.noarch 2023-07-14T19:06:42.366 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-mgr-rook-2:18.0.0-4940.g81c9387d.el8.noarch 2023-07-14T19:06:42.366 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-mon-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T19:06:42.367 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-osd-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T19:06:42.367 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-prometheus-alerts-2:18.0.0-4940.g81c9387d.el8.noarch 2023-07-14T19:06:42.367 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-radosgw-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T19:06:42.367 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-selinux-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T19:06:42.367 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-test-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T19:06:42.367 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-volume-2:18.0.0-4940.g81c9387d.el8.noarch 2023-07-14T19:06:42.367 INFO:teuthology.orchestra.run.smithi130.stdout: cephadm-2:18.0.0-4940.g81c9387d.el8.noarch 2023-07-14T19:06:42.367 INFO:teuthology.orchestra.run.smithi130.stdout: cephfs-mirror-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T19:06:42.368 INFO:teuthology.orchestra.run.smithi130.stdout: cephfs-top-2:18.0.0-4940.g81c9387d.el8.noarch 2023-07-14T19:06:42.368 INFO:teuthology.orchestra.run.smithi130.stdout: elfutils-libelf-devel-0.186-1.el8.x86_64 2023-07-14T19:06:42.368 INFO:teuthology.orchestra.run.smithi130.stdout: flex-2.6.1-9.el8.x86_64 2023-07-14T19:06:42.368 INFO:teuthology.orchestra.run.smithi130.stdout: fuse-2.9.7-15.el8.x86_64 2023-07-14T19:06:42.368 INFO:teuthology.orchestra.run.smithi130.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2023-07-14T19:06:42.368 INFO:teuthology.orchestra.run.smithi130.stdout: jq-1.6-3.el8.x86_64 2023-07-14T19:06:42.368 INFO:teuthology.orchestra.run.smithi130.stdout: keyutils-libs-devel-1.5.10-9.el8.x86_64 2023-07-14T19:06:42.369 INFO:teuthology.orchestra.run.smithi130.stdout: krb5-devel-1.18.2-14.el8.x86_64 2023-07-14T19:06:42.369 INFO:teuthology.orchestra.run.smithi130.stdout: libaio-devel-0.3.112-1.el8.x86_64 2023-07-14T19:06:42.369 INFO:teuthology.orchestra.run.smithi130.stdout: libcephfs-devel-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T19:06:42.369 INFO:teuthology.orchestra.run.smithi130.stdout: libcephfs2-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T19:06:42.369 INFO:teuthology.orchestra.run.smithi130.stdout: libcephsqlite-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T19:06:42.369 INFO:teuthology.orchestra.run.smithi130.stdout: libcom_err-devel-1.45.6-4.el8.x86_64 2023-07-14T19:06:42.369 INFO:teuthology.orchestra.run.smithi130.stdout: libgfortran-8.5.0-10.1.el8_6.x86_64 2023-07-14T19:06:42.370 INFO:teuthology.orchestra.run.smithi130.stdout: liboath-2.6.2-3.el8.x86_64 2023-07-14T19:06:42.370 INFO:teuthology.orchestra.run.smithi130.stdout: libpq-13.5-1.el8.x86_64 2023-07-14T19:06:42.370 INFO:teuthology.orchestra.run.smithi130.stdout: libquadmath-8.5.0-10.1.el8_6.x86_64 2023-07-14T19:06:42.370 INFO:teuthology.orchestra.run.smithi130.stdout: librabbitmq-0.9.0-3.el8.x86_64 2023-07-14T19:06:42.370 INFO:teuthology.orchestra.run.smithi130.stdout: librados-devel-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T19:06:42.370 INFO:teuthology.orchestra.run.smithi130.stdout: libradosstriper1-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T19:06:42.370 INFO:teuthology.orchestra.run.smithi130.stdout: librdkafka-0.11.4-3.el8.x86_64 2023-07-14T19:06:42.370 INFO:teuthology.orchestra.run.smithi130.stdout: librgw2-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T19:06:42.371 INFO:teuthology.orchestra.run.smithi130.stdout: libselinux-devel-2.9-5.el8.x86_64 2023-07-14T19:06:42.371 INFO:teuthology.orchestra.run.smithi130.stdout: libsepol-devel-2.9-3.el8.x86_64 2023-07-14T19:06:42.371 INFO:teuthology.orchestra.run.smithi130.stdout: libtool-2.4.6-25.el8.x86_64 2023-07-14T19:06:42.371 INFO:teuthology.orchestra.run.smithi130.stdout: libunwind-1.3.1-3.el8.x86_64 2023-07-14T19:06:42.371 INFO:teuthology.orchestra.run.smithi130.stdout: libuuid-devel-2.32.1-35.el8.x86_64 2023-07-14T19:06:42.371 INFO:teuthology.orchestra.run.smithi130.stdout: libverto-devel-0.3.0-5.el8.x86_64 2023-07-14T19:06:42.371 INFO:teuthology.orchestra.run.smithi130.stdout: lttng-ust-2.8.1-11.el8.x86_64 2023-07-14T19:06:42.372 INFO:teuthology.orchestra.run.smithi130.stdout: oniguruma-6.8.2-2.el8.x86_64 2023-07-14T19:06:42.372 INFO:teuthology.orchestra.run.smithi130.stdout: openblas-0.3.15-3.el8.x86_64 2023-07-14T19:06:42.372 INFO:teuthology.orchestra.run.smithi130.stdout: openblas-threads-0.3.15-3.el8.x86_64 2023-07-14T19:06:42.372 INFO:teuthology.orchestra.run.smithi130.stdout: openssl-devel-1:1.1.1k-7.el8_6.x86_64 2023-07-14T19:06:42.372 INFO:teuthology.orchestra.run.smithi130.stdout: pcre2-devel-10.32-2.el8.x86_64 2023-07-14T19:06:42.372 INFO:teuthology.orchestra.run.smithi130.stdout: pcre2-utf16-10.32-2.el8.x86_64 2023-07-14T19:06:42.372 INFO:teuthology.orchestra.run.smithi130.stdout: pcre2-utf32-10.32-2.el8.x86_64 2023-07-14T19:06:42.372 INFO:teuthology.orchestra.run.smithi130.stdout: postgresql-10.21-2.module+el8.6.0+15342+53518fac.x86_64 2023-07-14T19:06:42.373 INFO:teuthology.orchestra.run.smithi130.stdout: postgresql-contrib-10.21-2.module+el8.6.0+15342+53518fac.x86_64 2023-07-14T19:06:42.373 INFO:teuthology.orchestra.run.smithi130.stdout: postgresql-server-10.21-2.module+el8.6.0+15342+53518fac.x86_64 2023-07-14T19:06:42.373 INFO:teuthology.orchestra.run.smithi130.stdout: python3-asyncssh-2.7.0-2.el8.noarch 2023-07-14T19:06:42.373 INFO:teuthology.orchestra.run.smithi130.stdout: python3-babel-2.5.1-7.el8.noarch 2023-07-14T19:06:42.373 INFO:teuthology.orchestra.run.smithi130.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2023-07-14T19:06:42.373 INFO:teuthology.orchestra.run.smithi130.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2023-07-14T19:06:42.373 INFO:teuthology.orchestra.run.smithi130.stdout: python3-cachetools-3.1.1-4.el8.noarch 2023-07-14T19:06:42.374 INFO:teuthology.orchestra.run.smithi130.stdout: python3-ceph-argparse-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T19:06:42.374 INFO:teuthology.orchestra.run.smithi130.stdout: python3-ceph-common-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T19:06:42.374 INFO:teuthology.orchestra.run.smithi130.stdout: python3-cephfs-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T19:06:42.374 INFO:teuthology.orchestra.run.smithi130.stdout: python3-certifi-2018.10.15-7.el8.noarch 2023-07-14T19:06:42.374 INFO:teuthology.orchestra.run.smithi130.stdout: python3-cffi-1.11.5-5.el8.x86_64 2023-07-14T19:06:42.374 INFO:teuthology.orchestra.run.smithi130.stdout: python3-cheroot-8.5.2-1.el8.noarch 2023-07-14T19:06:42.374 INFO:teuthology.orchestra.run.smithi130.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2023-07-14T19:06:42.374 INFO:teuthology.orchestra.run.smithi130.stdout: python3-cryptography-3.2.1-5.el8.x86_64 2023-07-14T19:06:42.375 INFO:teuthology.orchestra.run.smithi130.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2023-07-14T19:06:42.375 INFO:teuthology.orchestra.run.smithi130.stdout: python3-jaraco-6.2-6.el8.noarch 2023-07-14T19:06:42.375 INFO:teuthology.orchestra.run.smithi130.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2023-07-14T19:06:42.375 INFO:teuthology.orchestra.run.smithi130.stdout: python3-jinja2-2.10.1-3.el8.noarch 2023-07-14T19:06:42.375 INFO:teuthology.orchestra.run.smithi130.stdout: python3-jsonpatch-1.21-2.el8.noarch 2023-07-14T19:06:42.375 INFO:teuthology.orchestra.run.smithi130.stdout: python3-jsonpointer-1.10-11.el8.noarch 2023-07-14T19:06:42.376 INFO:teuthology.orchestra.run.smithi130.stdout: python3-jwt-1.6.1-2.el8.noarch 2023-07-14T19:06:42.376 INFO:teuthology.orchestra.run.smithi130.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2023-07-14T19:06:42.376 INFO:teuthology.orchestra.run.smithi130.stdout: python3-logutils-0.3.5-11.el8.noarch 2023-07-14T19:06:42.376 INFO:teuthology.orchestra.run.smithi130.stdout: python3-mako-1.0.6-13.el8.noarch 2023-07-14T19:06:42.376 INFO:teuthology.orchestra.run.smithi130.stdout: python3-markupsafe-0.23-19.el8.x86_64 2023-07-14T19:06:42.376 INFO:teuthology.orchestra.run.smithi130.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2023-07-14T19:06:42.377 INFO:teuthology.orchestra.run.smithi130.stdout: python3-natsort-7.1.1-2.el8.noarch 2023-07-14T19:06:42.377 INFO:teuthology.orchestra.run.smithi130.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2023-07-14T19:06:42.377 INFO:teuthology.orchestra.run.smithi130.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2023-07-14T19:06:42.377 INFO:teuthology.orchestra.run.smithi130.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2023-07-14T19:06:42.377 INFO:teuthology.orchestra.run.smithi130.stdout: python3-pecan-1.3.2-9.el8.noarch 2023-07-14T19:06:42.378 INFO:teuthology.orchestra.run.smithi130.stdout: python3-portend-2.6-1.el8.noarch 2023-07-14T19:06:42.378 INFO:teuthology.orchestra.run.smithi130.stdout: python3-prettytable-0.7.2-14.el8.noarch 2023-07-14T19:06:42.378 INFO:teuthology.orchestra.run.smithi130.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2023-07-14T19:06:42.378 INFO:teuthology.orchestra.run.smithi130.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2023-07-14T19:06:42.378 INFO:teuthology.orchestra.run.smithi130.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2023-07-14T19:06:42.379 INFO:teuthology.orchestra.run.smithi130.stdout: python3-pycparser-2.14-14.el8.noarch 2023-07-14T19:06:42.379 INFO:teuthology.orchestra.run.smithi130.stdout: python3-pytz-2017.2-9.el8.noarch 2023-07-14T19:06:42.379 INFO:teuthology.orchestra.run.smithi130.stdout: python3-rados-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T19:06:42.379 INFO:teuthology.orchestra.run.smithi130.stdout: python3-rbd-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T19:06:42.379 INFO:teuthology.orchestra.run.smithi130.stdout: python3-repoze-lru-0.7-6.el8.noarch 2023-07-14T19:06:42.379 INFO:teuthology.orchestra.run.smithi130.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2023-07-14T19:06:42.380 INFO:teuthology.orchestra.run.smithi130.stdout: python3-rgw-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T19:06:42.380 INFO:teuthology.orchestra.run.smithi130.stdout: python3-routes-2.4.1-12.el8.noarch 2023-07-14T19:06:42.380 INFO:teuthology.orchestra.run.smithi130.stdout: python3-rsa-4.8-1.el8.noarch 2023-07-14T19:06:42.380 INFO:teuthology.orchestra.run.smithi130.stdout: python3-scipy-1.0.0-21.module+el8.5.0+10916+41bd434d.x86_64 2023-07-14T19:06:42.380 INFO:teuthology.orchestra.run.smithi130.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2023-07-14T19:06:42.381 INFO:teuthology.orchestra.run.smithi130.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2023-07-14T19:06:42.381 INFO:teuthology.orchestra.run.smithi130.stdout: python3-tempora-1.14.1-5.el8.noarch 2023-07-14T19:06:42.381 INFO:teuthology.orchestra.run.smithi130.stdout: python3-trustme-0.6.0-4.el8.noarch 2023-07-14T19:06:42.381 INFO:teuthology.orchestra.run.smithi130.stdout: python3-waitress-1.4.3-1.el8.noarch 2023-07-14T19:06:42.381 INFO:teuthology.orchestra.run.smithi130.stdout: python3-webob-1.8.5-1.el8.1.noarch 2023-07-14T19:06:42.381 INFO:teuthology.orchestra.run.smithi130.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2023-07-14T19:06:42.381 INFO:teuthology.orchestra.run.smithi130.stdout: python3-webtest-2.0.33-1.el8.noarch 2023-07-14T19:06:42.381 INFO:teuthology.orchestra.run.smithi130.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2023-07-14T19:06:42.382 INFO:teuthology.orchestra.run.smithi130.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2023-07-14T19:06:42.382 INFO:teuthology.orchestra.run.smithi130.stdout: rbd-fuse-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T19:06:42.382 INFO:teuthology.orchestra.run.smithi130.stdout: rbd-mirror-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T19:06:42.382 INFO:teuthology.orchestra.run.smithi130.stdout: rbd-nbd-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T19:06:42.382 INFO:teuthology.orchestra.run.smithi130.stdout: socat-1.7.4.1-1.el8.x86_64 2023-07-14T19:06:42.382 INFO:teuthology.orchestra.run.smithi130.stdout: thrift-0.13.0-2.el8.x86_64 2023-07-14T19:06:42.382 INFO:teuthology.orchestra.run.smithi130.stdout: uuid-1.6.2-43.el8.x86_64 2023-07-14T19:06:42.383 INFO:teuthology.orchestra.run.smithi130.stdout: xfsprogs-devel-5.0.0-10.el8.x86_64 2023-07-14T19:06:42.383 INFO:teuthology.orchestra.run.smithi130.stdout: xmlstarlet-1.6.1-20.el8.x86_64 2023-07-14T19:06:42.383 INFO:teuthology.orchestra.run.smithi130.stdout: zlib-devel-1.2.11-18.el8_5.x86_64 2023-07-14T19:06:42.383 INFO:teuthology.orchestra.run.smithi130.stdout: 2023-07-14T19:06:42.384 INFO:teuthology.orchestra.run.smithi130.stdout:Complete! 2023-07-14T19:06:42.549 DEBUG:teuthology.parallel:result is None 2023-07-14T19:06:42.579 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: cephadm-2:18.0.0-4940.g81c9387d.el8.noarch 101/134 2023-07-14T19:06:42.587 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 132/134 2023-07-14T19:06:42.588 INFO:teuthology.orchestra.run.smithi191.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 133/134 2023-07-14T19:06:42.788 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : cephadm-2:18.0.0-4940.g81c9387d.el8.noarch 101/134 2023-07-14T19:06:42.857 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 133/134 2023-07-14T19:06:42.911 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : ceph-prometheus-alerts-2:18.0.0-4940.g81c9387d.e 102/134 2023-07-14T19:06:43.074 INFO:teuthology.orchestra.run.smithi191.stdout: Cleanup : iproute-5.15.0-4.el8.x86_64 134/134 2023-07-14T19:06:43.105 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : ceph-grafana-dashboards-2:18.0.0-4940.g81c9387d. 103/134 2023-07-14T19:06:43.288 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : ceph-mgr-cephadm-2:18.0.0-4940.g81c9387d.el8.noa 104/134 2023-07-14T19:06:43.527 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: ceph-mgr-cephadm-2:18.0.0-4940.g81c9387d.el8.noa 104/134 2023-07-14T19:06:43.595 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: iproute-5.15.0-4.el8.x86_64 134/134 2023-07-14T19:06:43.596 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : ceph-2:18.0.0-4940.g81c9387d.el8.x86_64 1/134 2023-07-14T19:06:43.596 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : ceph-base-2:18.0.0-4940.g81c9387d.el8.x86_64 2/134 2023-07-14T19:06:43.596 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : ceph-common-2:18.0.0-4940.g81c9387d.el8.x86_64 3/134 2023-07-14T19:06:43.596 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : ceph-fuse-2:18.0.0-4940.g81c9387d.el8.x86_64 4/134 2023-07-14T19:06:43.596 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : ceph-immutable-object-cache-2:18.0.0-4940.g81c93 5/134 2023-07-14T19:06:43.596 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : ceph-mds-2:18.0.0-4940.g81c9387d.el8.x86_64 6/134 2023-07-14T19:06:43.597 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : ceph-mgr-2:18.0.0-4940.g81c9387d.el8.x86_64 7/134 2023-07-14T19:06:43.597 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : ceph-mon-2:18.0.0-4940.g81c9387d.el8.x86_64 8/134 2023-07-14T19:06:43.597 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : ceph-osd-2:18.0.0-4940.g81c9387d.el8.x86_64 9/134 2023-07-14T19:06:43.597 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : ceph-radosgw-2:18.0.0-4940.g81c9387d.el8.x86_64 10/134 2023-07-14T19:06:43.597 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : ceph-selinux-2:18.0.0-4940.g81c9387d.el8.x86_64 11/134 2023-07-14T19:06:43.597 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : ceph-test-2:18.0.0-4940.g81c9387d.el8.x86_64 12/134 2023-07-14T19:06:43.598 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : cephfs-mirror-2:18.0.0-4940.g81c9387d.el8.x86_64 13/134 2023-07-14T19:06:43.598 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : libcephfs-devel-2:18.0.0-4940.g81c9387d.el8.x86_ 14/134 2023-07-14T19:06:43.598 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : libcephfs2-2:18.0.0-4940.g81c9387d.el8.x86_64 15/134 2023-07-14T19:06:43.598 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : libcephsqlite-2:18.0.0-4940.g81c9387d.el8.x86_64 16/134 2023-07-14T19:06:43.598 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : librados-devel-2:18.0.0-4940.g81c9387d.el8.x86_6 17/134 2023-07-14T19:06:43.598 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : libradosstriper1-2:18.0.0-4940.g81c9387d.el8.x86 18/134 2023-07-14T19:06:43.598 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : librgw2-2:18.0.0-4940.g81c9387d.el8.x86_64 19/134 2023-07-14T19:06:43.598 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-ceph-argparse-2:18.0.0-4940.g81c9387d.el 20/134 2023-07-14T19:06:43.599 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-ceph-common-2:18.0.0-4940.g81c9387d.el8. 21/134 2023-07-14T19:06:43.599 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-cephfs-2:18.0.0-4940.g81c9387d.el8.x86_6 22/134 2023-07-14T19:06:43.599 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-rados-2:18.0.0-4940.g81c9387d.el8.x86_64 23/134 2023-07-14T19:06:43.599 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-rbd-2:18.0.0-4940.g81c9387d.el8.x86_64 24/134 2023-07-14T19:06:43.599 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-rgw-2:18.0.0-4940.g81c9387d.el8.x86_64 25/134 2023-07-14T19:06:43.599 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : rbd-fuse-2:18.0.0-4940.g81c9387d.el8.x86_64 26/134 2023-07-14T19:06:43.599 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : rbd-mirror-2:18.0.0-4940.g81c9387d.el8.x86_64 27/134 2023-07-14T19:06:43.613 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : rbd-nbd-2:18.0.0-4940.g81c9387d.el8.x86_64 28/134 2023-07-14T19:06:43.613 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : ceph-grafana-dashboards-2:18.0.0-4940.g81c9387d. 29/134 2023-07-14T19:06:43.614 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : ceph-mgr-cephadm-2:18.0.0-4940.g81c9387d.el8.noa 30/134 2023-07-14T19:06:43.614 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : ceph-mgr-dashboard-2:18.0.0-4940.g81c9387d.el8.n 31/134 2023-07-14T19:06:43.614 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : ceph-mgr-diskprediction-local-2:18.0.0-4940.g81c 32/134 2023-07-14T19:06:43.615 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : ceph-mgr-modules-core-2:18.0.0-4940.g81c9387d.el 33/134 2023-07-14T19:06:43.615 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : ceph-mgr-rook-2:18.0.0-4940.g81c9387d.el8.noarch 34/134 2023-07-14T19:06:43.615 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : ceph-prometheus-alerts-2:18.0.0-4940.g81c9387d.e 35/134 2023-07-14T19:06:43.615 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : ceph-volume-2:18.0.0-4940.g81c9387d.el8.noarch 36/134 2023-07-14T19:06:43.616 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : cephadm-2:18.0.0-4940.g81c9387d.el8.noarch 37/134 2023-07-14T19:06:43.616 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : cephfs-top-2:18.0.0-4940.g81c9387d.el8.noarch 38/134 2023-07-14T19:06:43.616 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 39/134 2023-07-14T19:06:43.617 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 40/134 2023-07-14T19:06:43.617 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 41/134 2023-07-14T19:06:43.617 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-asyncssh-2.7.0-2.el8.noarch 42/134 2023-07-14T19:06:43.617 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 43/134 2023-07-14T19:06:43.618 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 44/134 2023-07-14T19:06:43.618 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 45/134 2023-07-14T19:06:43.618 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 46/134 2023-07-14T19:06:43.618 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 47/134 2023-07-14T19:06:43.618 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 48/134 2023-07-14T19:06:43.619 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 49/134 2023-07-14T19:06:43.619 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 50/134 2023-07-14T19:06:43.619 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 51/134 2023-07-14T19:06:43.619 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 52/134 2023-07-14T19:06:43.619 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 53/134 2023-07-14T19:06:43.619 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 54/134 2023-07-14T19:06:43.619 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-natsort-7.1.1-2.el8.noarch 55/134 2023-07-14T19:06:43.620 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 56/134 2023-07-14T19:06:43.620 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-portend-2.6-1.el8.noarch 57/134 2023-07-14T19:06:43.620 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 58/134 2023-07-14T19:06:43.620 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 59/134 2023-07-14T19:06:43.620 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-rsa-4.8-1.el8.noarch 60/134 2023-07-14T19:06:43.620 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 61/134 2023-07-14T19:06:43.620 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 62/134 2023-07-14T19:06:43.621 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 63/134 2023-07-14T19:06:43.621 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 64/134 2023-07-14T19:06:43.621 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 65/134 2023-07-14T19:06:43.621 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 66/134 2023-07-14T19:06:43.621 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 67/134 2023-07-14T19:06:43.621 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 68/134 2023-07-14T19:06:43.621 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 69/134 2023-07-14T19:06:43.621 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : thrift-0.13.0-2.el8.x86_64 70/134 2023-07-14T19:06:43.622 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 71/134 2023-07-14T19:06:43.622 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : librabbitmq-0.9.0-3.el8.x86_64 72/134 2023-07-14T19:06:43.622 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : libselinux-devel-2.9-5.el8.x86_64 73/134 2023-07-14T19:06:43.622 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 74/134 2023-07-14T19:06:43.622 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : pcre2-utf32-10.32-2.el8.x86_64 75/134 2023-07-14T19:06:43.622 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : keyutils-libs-devel-1.5.10-9.el8.x86_64 76/134 2023-07-14T19:06:43.622 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : libsepol-devel-2.9-3.el8.x86_64 77/134 2023-07-14T19:06:43.623 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 78/134 2023-07-14T19:06:43.623 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 79/134 2023-07-14T19:06:43.623 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : pcre2-utf16-10.32-2.el8.x86_64 80/134 2023-07-14T19:06:43.623 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : krb5-devel-1.18.2-14.el8.x86_64 81/134 2023-07-14T19:06:43.623 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : libaio-devel-0.3.112-1.el8.x86_64 82/134 2023-07-14T19:06:43.623 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 83/134 2023-07-14T19:06:43.623 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-cryptography-3.2.1-5.el8.x86_64 84/134 2023-07-14T19:06:43.623 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : libverto-devel-0.3.0-5.el8.x86_64 85/134 2023-07-14T19:06:43.624 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : pcre2-devel-10.32-2.el8.x86_64 86/134 2023-07-14T19:06:43.624 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-cffi-1.11.5-5.el8.x86_64 87/134 2023-07-14T19:06:43.624 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : openssl-devel-1:1.1.1k-7.el8_6.x86_64 88/134 2023-07-14T19:06:43.624 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : xfsprogs-devel-5.0.0-10.el8.x86_64 89/134 2023-07-14T19:06:43.624 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : fuse-2.9.7-15.el8.x86_64 90/134 2023-07-14T19:06:43.624 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : elfutils-libelf-devel-0.186-1.el8.x86_64 91/134 2023-07-14T19:06:43.624 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : libuuid-devel-2.32.1-35.el8.x86_64 92/134 2023-07-14T19:06:43.625 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : zlib-devel-1.2.11-18.el8_5.x86_64 93/134 2023-07-14T19:06:43.625 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : libcom_err-devel-1.45.6-4.el8.x86_64 94/134 2023-07-14T19:06:43.625 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : libgfortran-8.5.0-10.1.el8_6.x86_64 95/134 2023-07-14T19:06:43.625 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : libquadmath-8.5.0-10.1.el8_6.x86_64 96/134 2023-07-14T19:06:43.625 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : NetworkManager-initscripts-updown-1:1.36.0-9.el8 97/134 2023-07-14T19:06:43.625 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-scipy-1.0.0-21.module+el8.5.0+10916+41bd 98/134 2023-07-14T19:06:43.625 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 99/134 2023-07-14T19:06:43.626 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-pytz-2017.2-9.el8.noarch 100/134 2023-07-14T19:06:43.626 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 101/134 2023-07-14T19:06:43.626 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 102/134 2023-07-14T19:06:43.626 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 103/134 2023-07-14T19:06:43.626 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : bison-3.0.4-10.el8.x86_64 104/134 2023-07-14T19:06:43.626 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-jinja2-2.10.1-3.el8.noarch 105/134 2023-07-14T19:06:43.627 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-mako-1.0.6-13.el8.noarch 106/134 2023-07-14T19:06:43.627 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : oniguruma-6.8.2-2.el8.x86_64 107/134 2023-07-14T19:06:43.627 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : automake-1.16.1-7.el8.noarch 108/134 2023-07-14T19:06:43.627 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 109/134 2023-07-14T19:06:43.627 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 110/134 2023-07-14T19:06:43.627 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : flex-2.6.1-9.el8.x86_64 111/134 2023-07-14T19:06:43.627 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 112/134 2023-07-14T19:06:43.628 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 113/134 2023-07-14T19:06:43.628 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 114/134 2023-07-14T19:06:43.628 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 115/134 2023-07-14T19:06:43.628 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : uuid-1.6.2-43.el8.x86_64 116/134 2023-07-14T19:06:43.628 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 117/134 2023-07-14T19:06:43.628 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 118/134 2023-07-14T19:06:43.628 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 119/134 2023-07-14T19:06:43.629 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : libtool-2.4.6-25.el8.x86_64 120/134 2023-07-14T19:06:43.629 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : openblas-threads-0.3.15-3.el8.x86_64 121/134 2023-07-14T19:06:43.629 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : libpq-13.5-1.el8.x86_64 122/134 2023-07-14T19:06:43.629 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : openblas-0.3.15-3.el8.x86_64 123/134 2023-07-14T19:06:43.629 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : jq-1.6-3.el8.x86_64 124/134 2023-07-14T19:06:43.629 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : librdkafka-0.11.4-3.el8.x86_64 125/134 2023-07-14T19:06:43.629 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : postgresql-server-10.21-2.module+el8.6.0+15342+5 126/134 2023-07-14T19:06:43.629 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : postgresql-contrib-10.21-2.module+el8.6.0+15342+ 127/134 2023-07-14T19:06:43.630 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : postgresql-10.21-2.module+el8.6.0+15342+53518fac 128/134 2023-07-14T19:06:43.630 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : librados2-2:18.0.0-4940.g81c9387d.el8.x86_64 129/134 2023-07-14T19:06:43.630 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 130/134 2023-07-14T19:06:43.630 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : librbd1-2:18.0.0-4940.g81c9387d.el8.x86_64 131/134 2023-07-14T19:06:43.630 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 132/134 2023-07-14T19:06:43.630 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : iproute-5.15.0-4.el8_6.1.x86_64 133/134 2023-07-14T19:06:43.793 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : ceph-mgr-dashboard-2:18.0.0-4940.g81c9387d.el8.n 105/134 2023-07-14T19:06:44.815 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: ceph-mgr-dashboard-2:18.0.0-4940.g81c9387d.el8.n 105/134 2023-07-14T19:06:44.954 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : ceph-mgr-diskprediction-local-2:18.0.0-4940.g81c 106/134 2023-07-14T19:06:45.051 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:18.0.0-4940.g81c 106/134 2023-07-14T19:06:45.414 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : ceph-mgr-modules-core-2:18.0.0-4940.g81c9387d.el 107/134 2023-07-14T19:06:45.659 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : ceph-mgr-2:18.0.0-4940.g81c9387d.el8.x86_64 108/134 2023-07-14T19:06:45.696 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: ceph-mgr-2:18.0.0-4940.g81c9387d.el8.x86_64 108/134 2023-07-14T19:06:45.837 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : ceph-mgr-rook-2:18.0.0-4940.g81c9387d.el8.noarch 109/134 2023-07-14T19:06:45.849 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: ceph-mgr-rook-2:18.0.0-4940.g81c9387d.el8.noarch 109/134 2023-07-14T19:06:46.051 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : iproute-5.15.0-4.el8.x86_64 134/134 2023-07-14T19:06:46.052 INFO:teuthology.orchestra.run.smithi191.stdout:Last metadata expiration check: 0:02:16 ago on Fri 14 Jul 2023 07:04:30 PM UTC. 2023-07-14T19:06:47.267 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : ceph-2:18.0.0-4940.g81c9387d.el8.x86_64 110/134 2023-07-14T19:06:47.467 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : ceph-radosgw-2:18.0.0-4940.g81c9387d.el8.x86_64 111/134 2023-07-14T19:06:47.502 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: ceph-radosgw-2:18.0.0-4940.g81c9387d.el8.x86_64 111/134 2023-07-14T19:06:47.630 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : ceph-immutable-object-cache-2:18.0.0-4940.g81c93 112/134 2023-07-14T19:06:47.672 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: ceph-immutable-object-cache-2:18.0.0-4940.g81c93 112/134 2023-07-14T19:06:47.854 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : cephfs-mirror-2:18.0.0-4940.g81c9387d.el8.x86_64 113/134 2023-07-14T19:06:48.228 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: cephfs-mirror-2:18.0.0-4940.g81c9387d.el8.x86_64 113/134 2023-07-14T19:06:48.404 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : rbd-mirror-2:18.0.0-4940.g81c9387d.el8.x86_64 114/134 2023-07-14T19:06:53.267 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: rbd-mirror-2:18.0.0-4940.g81c9387d.el8.x86_64 114/134 2023-07-14T19:06:54.233 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : ceph-test-2:18.0.0-4940.g81c9387d.el8.x86_64 115/134 2023-07-14T19:06:54.540 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : openssl-devel-1:1.1.1k-7.el8_6.x86_64 116/134 2023-07-14T19:06:54.757 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : ceph-fuse-2:18.0.0-4940.g81c9387d.el8.x86_64 117/134 2023-07-14T19:06:54.991 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : xfsprogs-devel-5.0.0-10.el8.x86_64 118/134 2023-07-14T19:06:55.219 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : libtool-2.4.6-25.el8.x86_64 119/134 2023-07-14T19:06:55.440 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: libtool-2.4.6-25.el8.x86_64 119/134 2023-07-14T19:06:55.733 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : postgresql-contrib-10.21-2.module+el8.6.0+15342+ 120/134 2023-07-14T19:06:55.916 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : libcephfs-devel-2:18.0.0-4940.g81c9387d.el8.x86_ 121/134 2023-07-14T19:06:56.104 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : elfutils-libelf-devel-0.186-1.el8.x86_64 122/134 2023-07-14T19:06:56.206 INFO:teuthology.orchestra.run.smithi191.stdout:Installed products updated. 2023-07-14T19:06:56.419 INFO:teuthology.orchestra.run.smithi191.stdout: 2023-07-14T19:06:56.419 INFO:teuthology.orchestra.run.smithi191.stdout:Upgraded: 2023-07-14T19:06:56.419 INFO:teuthology.orchestra.run.smithi191.stdout: iproute-5.15.0-4.el8_6.1.x86_64 2023-07-14T19:06:56.420 INFO:teuthology.orchestra.run.smithi191.stdout: librados2-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T19:06:56.420 INFO:teuthology.orchestra.run.smithi191.stdout: librbd1-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T19:06:56.420 INFO:teuthology.orchestra.run.smithi191.stdout:Installed: 2023-07-14T19:06:56.421 INFO:teuthology.orchestra.run.smithi191.stdout: NetworkManager-initscripts-updown-1:1.36.0-9.el8_6.noarch 2023-07-14T19:06:56.421 INFO:teuthology.orchestra.run.smithi191.stdout: automake-1.16.1-7.el8.noarch 2023-07-14T19:06:56.421 INFO:teuthology.orchestra.run.smithi191.stdout: bison-3.0.4-10.el8.x86_64 2023-07-14T19:06:56.422 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T19:06:56.422 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-base-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T19:06:56.422 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-common-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T19:06:56.422 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-fuse-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T19:06:56.423 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-grafana-dashboards-2:18.0.0-4940.g81c9387d.el8.noarch 2023-07-14T19:06:56.423 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-immutable-object-cache-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T19:06:56.423 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-mds-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T19:06:56.424 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-mgr-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T19:06:56.424 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-mgr-cephadm-2:18.0.0-4940.g81c9387d.el8.noarch 2023-07-14T19:06:56.424 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-mgr-dashboard-2:18.0.0-4940.g81c9387d.el8.noarch 2023-07-14T19:06:56.424 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-mgr-diskprediction-local-2:18.0.0-4940.g81c9387d.el8.noarch 2023-07-14T19:06:56.425 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-mgr-modules-core-2:18.0.0-4940.g81c9387d.el8.noarch 2023-07-14T19:06:56.425 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-mgr-rook-2:18.0.0-4940.g81c9387d.el8.noarch 2023-07-14T19:06:56.425 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-mon-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T19:06:56.425 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-osd-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T19:06:56.425 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-prometheus-alerts-2:18.0.0-4940.g81c9387d.el8.noarch 2023-07-14T19:06:56.426 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-radosgw-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T19:06:56.426 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-selinux-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T19:06:56.426 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-test-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T19:06:56.426 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-volume-2:18.0.0-4940.g81c9387d.el8.noarch 2023-07-14T19:06:56.426 INFO:teuthology.orchestra.run.smithi191.stdout: cephadm-2:18.0.0-4940.g81c9387d.el8.noarch 2023-07-14T19:06:56.427 INFO:teuthology.orchestra.run.smithi191.stdout: cephfs-mirror-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T19:06:56.427 INFO:teuthology.orchestra.run.smithi191.stdout: cephfs-top-2:18.0.0-4940.g81c9387d.el8.noarch 2023-07-14T19:06:56.427 INFO:teuthology.orchestra.run.smithi191.stdout: elfutils-libelf-devel-0.186-1.el8.x86_64 2023-07-14T19:06:56.427 INFO:teuthology.orchestra.run.smithi191.stdout: flex-2.6.1-9.el8.x86_64 2023-07-14T19:06:56.427 INFO:teuthology.orchestra.run.smithi191.stdout: fuse-2.9.7-15.el8.x86_64 2023-07-14T19:06:56.427 INFO:teuthology.orchestra.run.smithi191.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2023-07-14T19:06:56.428 INFO:teuthology.orchestra.run.smithi191.stdout: jq-1.6-3.el8.x86_64 2023-07-14T19:06:56.428 INFO:teuthology.orchestra.run.smithi191.stdout: keyutils-libs-devel-1.5.10-9.el8.x86_64 2023-07-14T19:06:56.428 INFO:teuthology.orchestra.run.smithi191.stdout: krb5-devel-1.18.2-14.el8.x86_64 2023-07-14T19:06:56.428 INFO:teuthology.orchestra.run.smithi191.stdout: libaio-devel-0.3.112-1.el8.x86_64 2023-07-14T19:06:56.428 INFO:teuthology.orchestra.run.smithi191.stdout: libcephfs-devel-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T19:06:56.428 INFO:teuthology.orchestra.run.smithi191.stdout: libcephfs2-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T19:06:56.429 INFO:teuthology.orchestra.run.smithi191.stdout: libcephsqlite-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T19:06:56.429 INFO:teuthology.orchestra.run.smithi191.stdout: libcom_err-devel-1.45.6-4.el8.x86_64 2023-07-14T19:06:56.429 INFO:teuthology.orchestra.run.smithi191.stdout: libgfortran-8.5.0-10.1.el8_6.x86_64 2023-07-14T19:06:56.429 INFO:teuthology.orchestra.run.smithi191.stdout: liboath-2.6.2-3.el8.x86_64 2023-07-14T19:06:56.429 INFO:teuthology.orchestra.run.smithi191.stdout: libpq-13.5-1.el8.x86_64 2023-07-14T19:06:56.430 INFO:teuthology.orchestra.run.smithi191.stdout: libquadmath-8.5.0-10.1.el8_6.x86_64 2023-07-14T19:06:56.430 INFO:teuthology.orchestra.run.smithi191.stdout: librabbitmq-0.9.0-3.el8.x86_64 2023-07-14T19:06:56.430 INFO:teuthology.orchestra.run.smithi191.stdout: librados-devel-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T19:06:56.430 INFO:teuthology.orchestra.run.smithi191.stdout: libradosstriper1-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T19:06:56.431 INFO:teuthology.orchestra.run.smithi191.stdout: librdkafka-0.11.4-3.el8.x86_64 2023-07-14T19:06:56.431 INFO:teuthology.orchestra.run.smithi191.stdout: librgw2-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T19:06:56.431 INFO:teuthology.orchestra.run.smithi191.stdout: libselinux-devel-2.9-5.el8.x86_64 2023-07-14T19:06:56.431 INFO:teuthology.orchestra.run.smithi191.stdout: libsepol-devel-2.9-3.el8.x86_64 2023-07-14T19:06:56.431 INFO:teuthology.orchestra.run.smithi191.stdout: libtool-2.4.6-25.el8.x86_64 2023-07-14T19:06:56.431 INFO:teuthology.orchestra.run.smithi191.stdout: libunwind-1.3.1-3.el8.x86_64 2023-07-14T19:06:56.432 INFO:teuthology.orchestra.run.smithi191.stdout: libuuid-devel-2.32.1-35.el8.x86_64 2023-07-14T19:06:56.432 INFO:teuthology.orchestra.run.smithi191.stdout: libverto-devel-0.3.0-5.el8.x86_64 2023-07-14T19:06:56.432 INFO:teuthology.orchestra.run.smithi191.stdout: lttng-ust-2.8.1-11.el8.x86_64 2023-07-14T19:06:56.432 INFO:teuthology.orchestra.run.smithi191.stdout: oniguruma-6.8.2-2.el8.x86_64 2023-07-14T19:06:56.432 INFO:teuthology.orchestra.run.smithi191.stdout: openblas-0.3.15-3.el8.x86_64 2023-07-14T19:06:56.432 INFO:teuthology.orchestra.run.smithi191.stdout: openblas-threads-0.3.15-3.el8.x86_64 2023-07-14T19:06:56.432 INFO:teuthology.orchestra.run.smithi191.stdout: openssl-devel-1:1.1.1k-7.el8_6.x86_64 2023-07-14T19:06:56.432 INFO:teuthology.orchestra.run.smithi191.stdout: pcre2-devel-10.32-2.el8.x86_64 2023-07-14T19:06:56.433 INFO:teuthology.orchestra.run.smithi191.stdout: pcre2-utf16-10.32-2.el8.x86_64 2023-07-14T19:06:56.433 INFO:teuthology.orchestra.run.smithi191.stdout: pcre2-utf32-10.32-2.el8.x86_64 2023-07-14T19:06:56.433 INFO:teuthology.orchestra.run.smithi191.stdout: postgresql-10.21-2.module+el8.6.0+15342+53518fac.x86_64 2023-07-14T19:06:56.433 INFO:teuthology.orchestra.run.smithi191.stdout: postgresql-contrib-10.21-2.module+el8.6.0+15342+53518fac.x86_64 2023-07-14T19:06:56.433 INFO:teuthology.orchestra.run.smithi191.stdout: postgresql-server-10.21-2.module+el8.6.0+15342+53518fac.x86_64 2023-07-14T19:06:56.433 INFO:teuthology.orchestra.run.smithi191.stdout: python3-asyncssh-2.7.0-2.el8.noarch 2023-07-14T19:06:56.433 INFO:teuthology.orchestra.run.smithi191.stdout: python3-babel-2.5.1-7.el8.noarch 2023-07-14T19:06:56.434 INFO:teuthology.orchestra.run.smithi191.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2023-07-14T19:06:56.434 INFO:teuthology.orchestra.run.smithi191.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2023-07-14T19:06:56.434 INFO:teuthology.orchestra.run.smithi191.stdout: python3-cachetools-3.1.1-4.el8.noarch 2023-07-14T19:06:56.434 INFO:teuthology.orchestra.run.smithi191.stdout: python3-ceph-argparse-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T19:06:56.435 INFO:teuthology.orchestra.run.smithi191.stdout: python3-ceph-common-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T19:06:56.435 INFO:teuthology.orchestra.run.smithi191.stdout: python3-cephfs-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T19:06:56.435 INFO:teuthology.orchestra.run.smithi191.stdout: python3-certifi-2018.10.15-7.el8.noarch 2023-07-14T19:06:56.435 INFO:teuthology.orchestra.run.smithi191.stdout: python3-cffi-1.11.5-5.el8.x86_64 2023-07-14T19:06:56.435 INFO:teuthology.orchestra.run.smithi191.stdout: python3-cheroot-8.5.2-1.el8.noarch 2023-07-14T19:06:56.435 INFO:teuthology.orchestra.run.smithi191.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2023-07-14T19:06:56.436 INFO:teuthology.orchestra.run.smithi191.stdout: python3-cryptography-3.2.1-5.el8.x86_64 2023-07-14T19:06:56.436 INFO:teuthology.orchestra.run.smithi191.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2023-07-14T19:06:56.436 INFO:teuthology.orchestra.run.smithi191.stdout: python3-jaraco-6.2-6.el8.noarch 2023-07-14T19:06:56.436 INFO:teuthology.orchestra.run.smithi191.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2023-07-14T19:06:56.436 INFO:teuthology.orchestra.run.smithi191.stdout: python3-jinja2-2.10.1-3.el8.noarch 2023-07-14T19:06:56.436 INFO:teuthology.orchestra.run.smithi191.stdout: python3-jsonpatch-1.21-2.el8.noarch 2023-07-14T19:06:56.436 INFO:teuthology.orchestra.run.smithi191.stdout: python3-jsonpointer-1.10-11.el8.noarch 2023-07-14T19:06:56.437 INFO:teuthology.orchestra.run.smithi191.stdout: python3-jwt-1.6.1-2.el8.noarch 2023-07-14T19:06:56.437 INFO:teuthology.orchestra.run.smithi191.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2023-07-14T19:06:56.437 INFO:teuthology.orchestra.run.smithi191.stdout: python3-logutils-0.3.5-11.el8.noarch 2023-07-14T19:06:56.437 INFO:teuthology.orchestra.run.smithi191.stdout: python3-mako-1.0.6-13.el8.noarch 2023-07-14T19:06:56.437 INFO:teuthology.orchestra.run.smithi191.stdout: python3-markupsafe-0.23-19.el8.x86_64 2023-07-14T19:06:56.437 INFO:teuthology.orchestra.run.smithi191.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2023-07-14T19:06:56.437 INFO:teuthology.orchestra.run.smithi191.stdout: python3-natsort-7.1.1-2.el8.noarch 2023-07-14T19:06:56.437 INFO:teuthology.orchestra.run.smithi191.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2023-07-14T19:06:56.438 INFO:teuthology.orchestra.run.smithi191.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2023-07-14T19:06:56.438 INFO:teuthology.orchestra.run.smithi191.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2023-07-14T19:06:56.438 INFO:teuthology.orchestra.run.smithi191.stdout: python3-pecan-1.3.2-9.el8.noarch 2023-07-14T19:06:56.438 INFO:teuthology.orchestra.run.smithi191.stdout: python3-portend-2.6-1.el8.noarch 2023-07-14T19:06:56.438 INFO:teuthology.orchestra.run.smithi191.stdout: python3-prettytable-0.7.2-14.el8.noarch 2023-07-14T19:06:56.438 INFO:teuthology.orchestra.run.smithi191.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2023-07-14T19:06:56.438 INFO:teuthology.orchestra.run.smithi191.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2023-07-14T19:06:56.438 INFO:teuthology.orchestra.run.smithi191.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2023-07-14T19:06:56.439 INFO:teuthology.orchestra.run.smithi191.stdout: python3-pycparser-2.14-14.el8.noarch 2023-07-14T19:06:56.439 INFO:teuthology.orchestra.run.smithi191.stdout: python3-pytz-2017.2-9.el8.noarch 2023-07-14T19:06:56.439 INFO:teuthology.orchestra.run.smithi191.stdout: python3-rados-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T19:06:56.439 INFO:teuthology.orchestra.run.smithi191.stdout: python3-rbd-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T19:06:56.439 INFO:teuthology.orchestra.run.smithi191.stdout: python3-repoze-lru-0.7-6.el8.noarch 2023-07-14T19:06:56.439 INFO:teuthology.orchestra.run.smithi191.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2023-07-14T19:06:56.439 INFO:teuthology.orchestra.run.smithi191.stdout: python3-rgw-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T19:06:56.440 INFO:teuthology.orchestra.run.smithi191.stdout: python3-routes-2.4.1-12.el8.noarch 2023-07-14T19:06:56.440 INFO:teuthology.orchestra.run.smithi191.stdout: python3-rsa-4.8-1.el8.noarch 2023-07-14T19:06:56.440 INFO:teuthology.orchestra.run.smithi191.stdout: python3-scipy-1.0.0-21.module+el8.5.0+10916+41bd434d.x86_64 2023-07-14T19:06:56.440 INFO:teuthology.orchestra.run.smithi191.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2023-07-14T19:06:56.440 INFO:teuthology.orchestra.run.smithi191.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2023-07-14T19:06:56.440 INFO:teuthology.orchestra.run.smithi191.stdout: python3-tempora-1.14.1-5.el8.noarch 2023-07-14T19:06:56.440 INFO:teuthology.orchestra.run.smithi191.stdout: python3-trustme-0.6.0-4.el8.noarch 2023-07-14T19:06:56.441 INFO:teuthology.orchestra.run.smithi191.stdout: python3-waitress-1.4.3-1.el8.noarch 2023-07-14T19:06:56.441 INFO:teuthology.orchestra.run.smithi191.stdout: python3-webob-1.8.5-1.el8.1.noarch 2023-07-14T19:06:56.441 INFO:teuthology.orchestra.run.smithi191.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2023-07-14T19:06:56.441 INFO:teuthology.orchestra.run.smithi191.stdout: python3-webtest-2.0.33-1.el8.noarch 2023-07-14T19:06:56.441 INFO:teuthology.orchestra.run.smithi191.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2023-07-14T19:06:56.441 INFO:teuthology.orchestra.run.smithi191.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2023-07-14T19:06:56.441 INFO:teuthology.orchestra.run.smithi191.stdout: rbd-fuse-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T19:06:56.442 INFO:teuthology.orchestra.run.smithi191.stdout: rbd-mirror-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T19:06:56.442 INFO:teuthology.orchestra.run.smithi191.stdout: rbd-nbd-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T19:06:56.442 INFO:teuthology.orchestra.run.smithi191.stdout: socat-1.7.4.1-1.el8.x86_64 2023-07-14T19:06:56.442 INFO:teuthology.orchestra.run.smithi191.stdout: thrift-0.13.0-2.el8.x86_64 2023-07-14T19:06:56.442 INFO:teuthology.orchestra.run.smithi191.stdout: uuid-1.6.2-43.el8.x86_64 2023-07-14T19:06:56.442 INFO:teuthology.orchestra.run.smithi191.stdout: xfsprogs-devel-5.0.0-10.el8.x86_64 2023-07-14T19:06:56.442 INFO:teuthology.orchestra.run.smithi191.stdout: xmlstarlet-1.6.1-20.el8.x86_64 2023-07-14T19:06:56.443 INFO:teuthology.orchestra.run.smithi191.stdout: zlib-devel-1.2.11-18.el8_5.x86_64 2023-07-14T19:06:56.443 INFO:teuthology.orchestra.run.smithi191.stdout: 2023-07-14T19:06:56.443 INFO:teuthology.orchestra.run.smithi191.stdout:Complete! 2023-07-14T19:06:56.604 DEBUG:teuthology.parallel:result is None 2023-07-14T19:06:58.470 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: postgresql-server-10.21-2.module+el8.6.0+15342+5 123/134 2023-07-14T19:06:58.744 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : postgresql-server-10.21-2.module+el8.6.0+15342+5 123/134 2023-07-14T19:06:58.763 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: postgresql-server-10.21-2.module+el8.6.0+15342+5 123/134 2023-07-14T19:06:59.038 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : cephfs-top-2:18.0.0-4940.g81c9387d.el8.noarch 124/134 2023-07-14T19:06:59.261 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : rbd-fuse-2:18.0.0-4940.g81c9387d.el8.x86_64 125/134 2023-07-14T19:06:59.660 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : rbd-nbd-2:18.0.0-4940.g81c9387d.el8.x86_64 126/134 2023-07-14T19:07:00.024 INFO:teuthology.orchestra.run.smithi040.stdout: Upgrading : iproute-5.15.0-4.el8_6.1.x86_64 127/134 2023-07-14T19:07:00.187 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : flex-2.6.1-9.el8.x86_64 128/134 2023-07-14T19:07:00.278 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: flex-2.6.1-9.el8.x86_64 128/134 2023-07-14T19:07:00.519 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : bison-3.0.4-10.el8.x86_64 129/134 2023-07-14T19:07:00.533 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: bison-3.0.4-10.el8.x86_64 129/134 2023-07-14T19:07:00.677 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : NetworkManager-initscripts-updown-1:1.36.0-9.el8 130/134 2023-07-14T19:07:00.687 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: NetworkManager-initscripts-updown-1:1.36.0-9.el8 130/134 2023-07-14T19:07:00.855 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : libaio-devel-0.3.112-1.el8.x86_64 131/134 2023-07-14T19:07:00.855 INFO:teuthology.orchestra.run.smithi040.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 132/134 2023-07-14T19:07:01.064 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 132/134 2023-07-14T19:07:01.065 INFO:teuthology.orchestra.run.smithi040.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 133/134 2023-07-14T19:07:01.327 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 133/134 2023-07-14T19:07:01.544 INFO:teuthology.orchestra.run.smithi040.stdout: Cleanup : iproute-5.15.0-4.el8.x86_64 134/134 2023-07-14T19:07:02.094 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: iproute-5.15.0-4.el8.x86_64 134/134 2023-07-14T19:07:02.095 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : ceph-2:18.0.0-4940.g81c9387d.el8.x86_64 1/134 2023-07-14T19:07:02.095 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : ceph-base-2:18.0.0-4940.g81c9387d.el8.x86_64 2/134 2023-07-14T19:07:02.095 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : ceph-common-2:18.0.0-4940.g81c9387d.el8.x86_64 3/134 2023-07-14T19:07:02.096 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : ceph-fuse-2:18.0.0-4940.g81c9387d.el8.x86_64 4/134 2023-07-14T19:07:02.096 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : ceph-immutable-object-cache-2:18.0.0-4940.g81c93 5/134 2023-07-14T19:07:02.096 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : ceph-mds-2:18.0.0-4940.g81c9387d.el8.x86_64 6/134 2023-07-14T19:07:02.096 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : ceph-mgr-2:18.0.0-4940.g81c9387d.el8.x86_64 7/134 2023-07-14T19:07:02.097 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : ceph-mon-2:18.0.0-4940.g81c9387d.el8.x86_64 8/134 2023-07-14T19:07:02.097 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : ceph-osd-2:18.0.0-4940.g81c9387d.el8.x86_64 9/134 2023-07-14T19:07:02.097 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : ceph-radosgw-2:18.0.0-4940.g81c9387d.el8.x86_64 10/134 2023-07-14T19:07:02.097 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : ceph-selinux-2:18.0.0-4940.g81c9387d.el8.x86_64 11/134 2023-07-14T19:07:02.098 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : ceph-test-2:18.0.0-4940.g81c9387d.el8.x86_64 12/134 2023-07-14T19:07:02.098 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : cephfs-mirror-2:18.0.0-4940.g81c9387d.el8.x86_64 13/134 2023-07-14T19:07:02.098 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : libcephfs-devel-2:18.0.0-4940.g81c9387d.el8.x86_ 14/134 2023-07-14T19:07:02.099 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : libcephfs2-2:18.0.0-4940.g81c9387d.el8.x86_64 15/134 2023-07-14T19:07:02.099 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : libcephsqlite-2:18.0.0-4940.g81c9387d.el8.x86_64 16/134 2023-07-14T19:07:02.099 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : librados-devel-2:18.0.0-4940.g81c9387d.el8.x86_6 17/134 2023-07-14T19:07:02.099 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : libradosstriper1-2:18.0.0-4940.g81c9387d.el8.x86 18/134 2023-07-14T19:07:02.100 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : librgw2-2:18.0.0-4940.g81c9387d.el8.x86_64 19/134 2023-07-14T19:07:02.100 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-ceph-argparse-2:18.0.0-4940.g81c9387d.el 20/134 2023-07-14T19:07:02.100 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-ceph-common-2:18.0.0-4940.g81c9387d.el8. 21/134 2023-07-14T19:07:02.101 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-cephfs-2:18.0.0-4940.g81c9387d.el8.x86_6 22/134 2023-07-14T19:07:02.101 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-rados-2:18.0.0-4940.g81c9387d.el8.x86_64 23/134 2023-07-14T19:07:02.101 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-rbd-2:18.0.0-4940.g81c9387d.el8.x86_64 24/134 2023-07-14T19:07:02.102 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-rgw-2:18.0.0-4940.g81c9387d.el8.x86_64 25/134 2023-07-14T19:07:02.102 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : rbd-fuse-2:18.0.0-4940.g81c9387d.el8.x86_64 26/134 2023-07-14T19:07:02.102 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : rbd-mirror-2:18.0.0-4940.g81c9387d.el8.x86_64 27/134 2023-07-14T19:07:02.103 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : rbd-nbd-2:18.0.0-4940.g81c9387d.el8.x86_64 28/134 2023-07-14T19:07:02.103 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : ceph-grafana-dashboards-2:18.0.0-4940.g81c9387d. 29/134 2023-07-14T19:07:02.104 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : ceph-mgr-cephadm-2:18.0.0-4940.g81c9387d.el8.noa 30/134 2023-07-14T19:07:02.104 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : ceph-mgr-dashboard-2:18.0.0-4940.g81c9387d.el8.n 31/134 2023-07-14T19:07:02.104 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : ceph-mgr-diskprediction-local-2:18.0.0-4940.g81c 32/134 2023-07-14T19:07:02.104 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : ceph-mgr-modules-core-2:18.0.0-4940.g81c9387d.el 33/134 2023-07-14T19:07:02.105 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : ceph-mgr-rook-2:18.0.0-4940.g81c9387d.el8.noarch 34/134 2023-07-14T19:07:02.105 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : ceph-prometheus-alerts-2:18.0.0-4940.g81c9387d.e 35/134 2023-07-14T19:07:02.105 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : ceph-volume-2:18.0.0-4940.g81c9387d.el8.noarch 36/134 2023-07-14T19:07:02.106 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : cephadm-2:18.0.0-4940.g81c9387d.el8.noarch 37/134 2023-07-14T19:07:02.106 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : cephfs-top-2:18.0.0-4940.g81c9387d.el8.noarch 38/134 2023-07-14T19:07:02.106 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 39/134 2023-07-14T19:07:02.107 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 40/134 2023-07-14T19:07:02.107 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 41/134 2023-07-14T19:07:02.107 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-asyncssh-2.7.0-2.el8.noarch 42/134 2023-07-14T19:07:02.107 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 43/134 2023-07-14T19:07:02.107 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 44/134 2023-07-14T19:07:02.108 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 45/134 2023-07-14T19:07:02.108 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 46/134 2023-07-14T19:07:02.108 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 47/134 2023-07-14T19:07:02.109 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 48/134 2023-07-14T19:07:02.109 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 49/134 2023-07-14T19:07:02.109 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 50/134 2023-07-14T19:07:02.109 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 51/134 2023-07-14T19:07:02.109 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 52/134 2023-07-14T19:07:02.110 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 53/134 2023-07-14T19:07:02.110 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 54/134 2023-07-14T19:07:02.110 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-natsort-7.1.1-2.el8.noarch 55/134 2023-07-14T19:07:02.110 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 56/134 2023-07-14T19:07:02.110 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-portend-2.6-1.el8.noarch 57/134 2023-07-14T19:07:02.111 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 58/134 2023-07-14T19:07:02.111 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 59/134 2023-07-14T19:07:02.111 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-rsa-4.8-1.el8.noarch 60/134 2023-07-14T19:07:02.111 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 61/134 2023-07-14T19:07:02.112 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 62/134 2023-07-14T19:07:02.112 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 63/134 2023-07-14T19:07:02.112 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 64/134 2023-07-14T19:07:02.112 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 65/134 2023-07-14T19:07:02.113 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 66/134 2023-07-14T19:07:02.113 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 67/134 2023-07-14T19:07:02.113 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 68/134 2023-07-14T19:07:02.113 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 69/134 2023-07-14T19:07:02.113 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : thrift-0.13.0-2.el8.x86_64 70/134 2023-07-14T19:07:02.114 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 71/134 2023-07-14T19:07:02.114 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : librabbitmq-0.9.0-3.el8.x86_64 72/134 2023-07-14T19:07:02.114 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : libselinux-devel-2.9-5.el8.x86_64 73/134 2023-07-14T19:07:02.114 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 74/134 2023-07-14T19:07:02.115 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : pcre2-utf32-10.32-2.el8.x86_64 75/134 2023-07-14T19:07:02.115 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : keyutils-libs-devel-1.5.10-9.el8.x86_64 76/134 2023-07-14T19:07:02.115 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : libsepol-devel-2.9-3.el8.x86_64 77/134 2023-07-14T19:07:02.115 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 78/134 2023-07-14T19:07:02.116 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 79/134 2023-07-14T19:07:02.116 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : pcre2-utf16-10.32-2.el8.x86_64 80/134 2023-07-14T19:07:02.116 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : krb5-devel-1.18.2-14.el8.x86_64 81/134 2023-07-14T19:07:02.116 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : libaio-devel-0.3.112-1.el8.x86_64 82/134 2023-07-14T19:07:02.116 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 83/134 2023-07-14T19:07:02.117 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-cryptography-3.2.1-5.el8.x86_64 84/134 2023-07-14T19:07:02.117 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : libverto-devel-0.3.0-5.el8.x86_64 85/134 2023-07-14T19:07:02.117 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : pcre2-devel-10.32-2.el8.x86_64 86/134 2023-07-14T19:07:02.117 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-cffi-1.11.5-5.el8.x86_64 87/134 2023-07-14T19:07:02.117 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : openssl-devel-1:1.1.1k-7.el8_6.x86_64 88/134 2023-07-14T19:07:02.117 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : xfsprogs-devel-5.0.0-10.el8.x86_64 89/134 2023-07-14T19:07:02.117 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : fuse-2.9.7-15.el8.x86_64 90/134 2023-07-14T19:07:02.118 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : elfutils-libelf-devel-0.186-1.el8.x86_64 91/134 2023-07-14T19:07:02.118 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : libuuid-devel-2.32.1-35.el8.x86_64 92/134 2023-07-14T19:07:02.118 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : zlib-devel-1.2.11-18.el8_5.x86_64 93/134 2023-07-14T19:07:02.118 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : libcom_err-devel-1.45.6-4.el8.x86_64 94/134 2023-07-14T19:07:02.118 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : libgfortran-8.5.0-10.1.el8_6.x86_64 95/134 2023-07-14T19:07:02.118 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : libquadmath-8.5.0-10.1.el8_6.x86_64 96/134 2023-07-14T19:07:02.119 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : NetworkManager-initscripts-updown-1:1.36.0-9.el8 97/134 2023-07-14T19:07:02.119 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-scipy-1.0.0-21.module+el8.5.0+10916+41bd 98/134 2023-07-14T19:07:02.119 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 99/134 2023-07-14T19:07:02.119 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-pytz-2017.2-9.el8.noarch 100/134 2023-07-14T19:07:02.119 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 101/134 2023-07-14T19:07:02.120 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 102/134 2023-07-14T19:07:02.120 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 103/134 2023-07-14T19:07:02.120 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : bison-3.0.4-10.el8.x86_64 104/134 2023-07-14T19:07:02.120 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-jinja2-2.10.1-3.el8.noarch 105/134 2023-07-14T19:07:02.120 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-mako-1.0.6-13.el8.noarch 106/134 2023-07-14T19:07:02.121 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : oniguruma-6.8.2-2.el8.x86_64 107/134 2023-07-14T19:07:02.121 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : automake-1.16.1-7.el8.noarch 108/134 2023-07-14T19:07:02.121 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 109/134 2023-07-14T19:07:02.121 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 110/134 2023-07-14T19:07:02.121 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : flex-2.6.1-9.el8.x86_64 111/134 2023-07-14T19:07:02.122 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 112/134 2023-07-14T19:07:02.122 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 113/134 2023-07-14T19:07:02.122 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 114/134 2023-07-14T19:07:02.122 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 115/134 2023-07-14T19:07:02.123 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : uuid-1.6.2-43.el8.x86_64 116/134 2023-07-14T19:07:02.123 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 117/134 2023-07-14T19:07:02.123 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 118/134 2023-07-14T19:07:02.124 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 119/134 2023-07-14T19:07:02.124 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : libtool-2.4.6-25.el8.x86_64 120/134 2023-07-14T19:07:02.124 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : openblas-threads-0.3.15-3.el8.x86_64 121/134 2023-07-14T19:07:02.124 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : libpq-13.5-1.el8.x86_64 122/134 2023-07-14T19:07:02.125 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : openblas-0.3.15-3.el8.x86_64 123/134 2023-07-14T19:07:02.125 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : jq-1.6-3.el8.x86_64 124/134 2023-07-14T19:07:02.125 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : librdkafka-0.11.4-3.el8.x86_64 125/134 2023-07-14T19:07:02.126 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : postgresql-server-10.21-2.module+el8.6.0+15342+5 126/134 2023-07-14T19:07:02.126 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : postgresql-contrib-10.21-2.module+el8.6.0+15342+ 127/134 2023-07-14T19:07:02.126 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : postgresql-10.21-2.module+el8.6.0+15342+53518fac 128/134 2023-07-14T19:07:02.126 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : librados2-2:18.0.0-4940.g81c9387d.el8.x86_64 129/134 2023-07-14T19:07:02.127 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 130/134 2023-07-14T19:07:02.127 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : librbd1-2:18.0.0-4940.g81c9387d.el8.x86_64 131/134 2023-07-14T19:07:02.127 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 132/134 2023-07-14T19:07:02.128 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : iproute-5.15.0-4.el8_6.1.x86_64 133/134 2023-07-14T19:07:04.572 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : iproute-5.15.0-4.el8.x86_64 134/134 2023-07-14T19:07:04.573 INFO:teuthology.orchestra.run.smithi040.stdout:Last metadata expiration check: 0:02:27 ago on Fri 14 Jul 2023 07:04:37 PM UTC. 2023-07-14T19:07:11.441 INFO:teuthology.orchestra.run.smithi040.stdout:Installed products updated. 2023-07-14T19:07:11.675 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-07-14T19:07:11.675 INFO:teuthology.orchestra.run.smithi040.stdout:Upgraded: 2023-07-14T19:07:11.675 INFO:teuthology.orchestra.run.smithi040.stdout: iproute-5.15.0-4.el8_6.1.x86_64 2023-07-14T19:07:11.675 INFO:teuthology.orchestra.run.smithi040.stdout: librados2-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T19:07:11.676 INFO:teuthology.orchestra.run.smithi040.stdout: librbd1-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T19:07:11.676 INFO:teuthology.orchestra.run.smithi040.stdout:Installed: 2023-07-14T19:07:11.676 INFO:teuthology.orchestra.run.smithi040.stdout: NetworkManager-initscripts-updown-1:1.36.0-9.el8_6.noarch 2023-07-14T19:07:11.676 INFO:teuthology.orchestra.run.smithi040.stdout: automake-1.16.1-7.el8.noarch 2023-07-14T19:07:11.676 INFO:teuthology.orchestra.run.smithi040.stdout: bison-3.0.4-10.el8.x86_64 2023-07-14T19:07:11.676 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T19:07:11.676 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-base-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T19:07:11.677 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-common-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T19:07:11.677 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-fuse-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T19:07:11.677 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-grafana-dashboards-2:18.0.0-4940.g81c9387d.el8.noarch 2023-07-14T19:07:11.677 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-immutable-object-cache-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T19:07:11.677 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-mds-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T19:07:11.677 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-mgr-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T19:07:11.677 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-mgr-cephadm-2:18.0.0-4940.g81c9387d.el8.noarch 2023-07-14T19:07:11.678 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-mgr-dashboard-2:18.0.0-4940.g81c9387d.el8.noarch 2023-07-14T19:07:11.678 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-mgr-diskprediction-local-2:18.0.0-4940.g81c9387d.el8.noarch 2023-07-14T19:07:11.678 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-mgr-modules-core-2:18.0.0-4940.g81c9387d.el8.noarch 2023-07-14T19:07:11.678 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-mgr-rook-2:18.0.0-4940.g81c9387d.el8.noarch 2023-07-14T19:07:11.678 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-mon-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T19:07:11.678 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-osd-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T19:07:11.678 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-prometheus-alerts-2:18.0.0-4940.g81c9387d.el8.noarch 2023-07-14T19:07:11.679 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-radosgw-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T19:07:11.679 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-selinux-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T19:07:11.679 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-test-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T19:07:11.679 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-volume-2:18.0.0-4940.g81c9387d.el8.noarch 2023-07-14T19:07:11.679 INFO:teuthology.orchestra.run.smithi040.stdout: cephadm-2:18.0.0-4940.g81c9387d.el8.noarch 2023-07-14T19:07:11.679 INFO:teuthology.orchestra.run.smithi040.stdout: cephfs-mirror-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T19:07:11.679 INFO:teuthology.orchestra.run.smithi040.stdout: cephfs-top-2:18.0.0-4940.g81c9387d.el8.noarch 2023-07-14T19:07:11.679 INFO:teuthology.orchestra.run.smithi040.stdout: elfutils-libelf-devel-0.186-1.el8.x86_64 2023-07-14T19:07:11.680 INFO:teuthology.orchestra.run.smithi040.stdout: flex-2.6.1-9.el8.x86_64 2023-07-14T19:07:11.680 INFO:teuthology.orchestra.run.smithi040.stdout: fuse-2.9.7-15.el8.x86_64 2023-07-14T19:07:11.680 INFO:teuthology.orchestra.run.smithi040.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2023-07-14T19:07:11.680 INFO:teuthology.orchestra.run.smithi040.stdout: jq-1.6-3.el8.x86_64 2023-07-14T19:07:11.680 INFO:teuthology.orchestra.run.smithi040.stdout: keyutils-libs-devel-1.5.10-9.el8.x86_64 2023-07-14T19:07:11.680 INFO:teuthology.orchestra.run.smithi040.stdout: krb5-devel-1.18.2-14.el8.x86_64 2023-07-14T19:07:11.680 INFO:teuthology.orchestra.run.smithi040.stdout: libaio-devel-0.3.112-1.el8.x86_64 2023-07-14T19:07:11.681 INFO:teuthology.orchestra.run.smithi040.stdout: libcephfs-devel-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T19:07:11.681 INFO:teuthology.orchestra.run.smithi040.stdout: libcephfs2-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T19:07:11.681 INFO:teuthology.orchestra.run.smithi040.stdout: libcephsqlite-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T19:07:11.681 INFO:teuthology.orchestra.run.smithi040.stdout: libcom_err-devel-1.45.6-4.el8.x86_64 2023-07-14T19:07:11.681 INFO:teuthology.orchestra.run.smithi040.stdout: libgfortran-8.5.0-10.1.el8_6.x86_64 2023-07-14T19:07:11.681 INFO:teuthology.orchestra.run.smithi040.stdout: liboath-2.6.2-3.el8.x86_64 2023-07-14T19:07:11.681 INFO:teuthology.orchestra.run.smithi040.stdout: libpq-13.5-1.el8.x86_64 2023-07-14T19:07:11.681 INFO:teuthology.orchestra.run.smithi040.stdout: libquadmath-8.5.0-10.1.el8_6.x86_64 2023-07-14T19:07:11.682 INFO:teuthology.orchestra.run.smithi040.stdout: librabbitmq-0.9.0-3.el8.x86_64 2023-07-14T19:07:11.682 INFO:teuthology.orchestra.run.smithi040.stdout: librados-devel-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T19:07:11.682 INFO:teuthology.orchestra.run.smithi040.stdout: libradosstriper1-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T19:07:11.682 INFO:teuthology.orchestra.run.smithi040.stdout: librdkafka-0.11.4-3.el8.x86_64 2023-07-14T19:07:11.682 INFO:teuthology.orchestra.run.smithi040.stdout: librgw2-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T19:07:11.682 INFO:teuthology.orchestra.run.smithi040.stdout: libselinux-devel-2.9-5.el8.x86_64 2023-07-14T19:07:11.682 INFO:teuthology.orchestra.run.smithi040.stdout: libsepol-devel-2.9-3.el8.x86_64 2023-07-14T19:07:11.683 INFO:teuthology.orchestra.run.smithi040.stdout: libtool-2.4.6-25.el8.x86_64 2023-07-14T19:07:11.683 INFO:teuthology.orchestra.run.smithi040.stdout: libunwind-1.3.1-3.el8.x86_64 2023-07-14T19:07:11.683 INFO:teuthology.orchestra.run.smithi040.stdout: libuuid-devel-2.32.1-35.el8.x86_64 2023-07-14T19:07:11.683 INFO:teuthology.orchestra.run.smithi040.stdout: libverto-devel-0.3.0-5.el8.x86_64 2023-07-14T19:07:11.683 INFO:teuthology.orchestra.run.smithi040.stdout: lttng-ust-2.8.1-11.el8.x86_64 2023-07-14T19:07:11.683 INFO:teuthology.orchestra.run.smithi040.stdout: oniguruma-6.8.2-2.el8.x86_64 2023-07-14T19:07:11.683 INFO:teuthology.orchestra.run.smithi040.stdout: openblas-0.3.15-3.el8.x86_64 2023-07-14T19:07:11.684 INFO:teuthology.orchestra.run.smithi040.stdout: openblas-threads-0.3.15-3.el8.x86_64 2023-07-14T19:07:11.684 INFO:teuthology.orchestra.run.smithi040.stdout: openssl-devel-1:1.1.1k-7.el8_6.x86_64 2023-07-14T19:07:11.684 INFO:teuthology.orchestra.run.smithi040.stdout: pcre2-devel-10.32-2.el8.x86_64 2023-07-14T19:07:11.684 INFO:teuthology.orchestra.run.smithi040.stdout: pcre2-utf16-10.32-2.el8.x86_64 2023-07-14T19:07:11.684 INFO:teuthology.orchestra.run.smithi040.stdout: pcre2-utf32-10.32-2.el8.x86_64 2023-07-14T19:07:11.684 INFO:teuthology.orchestra.run.smithi040.stdout: postgresql-10.21-2.module+el8.6.0+15342+53518fac.x86_64 2023-07-14T19:07:11.684 INFO:teuthology.orchestra.run.smithi040.stdout: postgresql-contrib-10.21-2.module+el8.6.0+15342+53518fac.x86_64 2023-07-14T19:07:11.684 INFO:teuthology.orchestra.run.smithi040.stdout: postgresql-server-10.21-2.module+el8.6.0+15342+53518fac.x86_64 2023-07-14T19:07:11.685 INFO:teuthology.orchestra.run.smithi040.stdout: python3-asyncssh-2.7.0-2.el8.noarch 2023-07-14T19:07:11.685 INFO:teuthology.orchestra.run.smithi040.stdout: python3-babel-2.5.1-7.el8.noarch 2023-07-14T19:07:11.685 INFO:teuthology.orchestra.run.smithi040.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2023-07-14T19:07:11.685 INFO:teuthology.orchestra.run.smithi040.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2023-07-14T19:07:11.685 INFO:teuthology.orchestra.run.smithi040.stdout: python3-cachetools-3.1.1-4.el8.noarch 2023-07-14T19:07:11.685 INFO:teuthology.orchestra.run.smithi040.stdout: python3-ceph-argparse-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T19:07:11.685 INFO:teuthology.orchestra.run.smithi040.stdout: python3-ceph-common-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T19:07:11.685 INFO:teuthology.orchestra.run.smithi040.stdout: python3-cephfs-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T19:07:11.686 INFO:teuthology.orchestra.run.smithi040.stdout: python3-certifi-2018.10.15-7.el8.noarch 2023-07-14T19:07:11.686 INFO:teuthology.orchestra.run.smithi040.stdout: python3-cffi-1.11.5-5.el8.x86_64 2023-07-14T19:07:11.686 INFO:teuthology.orchestra.run.smithi040.stdout: python3-cheroot-8.5.2-1.el8.noarch 2023-07-14T19:07:11.686 INFO:teuthology.orchestra.run.smithi040.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2023-07-14T19:07:11.686 INFO:teuthology.orchestra.run.smithi040.stdout: python3-cryptography-3.2.1-5.el8.x86_64 2023-07-14T19:07:11.686 INFO:teuthology.orchestra.run.smithi040.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2023-07-14T19:07:11.686 INFO:teuthology.orchestra.run.smithi040.stdout: python3-jaraco-6.2-6.el8.noarch 2023-07-14T19:07:11.687 INFO:teuthology.orchestra.run.smithi040.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2023-07-14T19:07:11.687 INFO:teuthology.orchestra.run.smithi040.stdout: python3-jinja2-2.10.1-3.el8.noarch 2023-07-14T19:07:11.687 INFO:teuthology.orchestra.run.smithi040.stdout: python3-jsonpatch-1.21-2.el8.noarch 2023-07-14T19:07:11.687 INFO:teuthology.orchestra.run.smithi040.stdout: python3-jsonpointer-1.10-11.el8.noarch 2023-07-14T19:07:11.687 INFO:teuthology.orchestra.run.smithi040.stdout: python3-jwt-1.6.1-2.el8.noarch 2023-07-14T19:07:11.687 INFO:teuthology.orchestra.run.smithi040.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2023-07-14T19:07:11.687 INFO:teuthology.orchestra.run.smithi040.stdout: python3-logutils-0.3.5-11.el8.noarch 2023-07-14T19:07:11.687 INFO:teuthology.orchestra.run.smithi040.stdout: python3-mako-1.0.6-13.el8.noarch 2023-07-14T19:07:11.688 INFO:teuthology.orchestra.run.smithi040.stdout: python3-markupsafe-0.23-19.el8.x86_64 2023-07-14T19:07:11.688 INFO:teuthology.orchestra.run.smithi040.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2023-07-14T19:07:11.688 INFO:teuthology.orchestra.run.smithi040.stdout: python3-natsort-7.1.1-2.el8.noarch 2023-07-14T19:07:11.688 INFO:teuthology.orchestra.run.smithi040.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2023-07-14T19:07:11.688 INFO:teuthology.orchestra.run.smithi040.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2023-07-14T19:07:11.688 INFO:teuthology.orchestra.run.smithi040.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2023-07-14T19:07:11.688 INFO:teuthology.orchestra.run.smithi040.stdout: python3-pecan-1.3.2-9.el8.noarch 2023-07-14T19:07:11.688 INFO:teuthology.orchestra.run.smithi040.stdout: python3-portend-2.6-1.el8.noarch 2023-07-14T19:07:11.689 INFO:teuthology.orchestra.run.smithi040.stdout: python3-prettytable-0.7.2-14.el8.noarch 2023-07-14T19:07:11.689 INFO:teuthology.orchestra.run.smithi040.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2023-07-14T19:07:11.689 INFO:teuthology.orchestra.run.smithi040.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2023-07-14T19:07:11.689 INFO:teuthology.orchestra.run.smithi040.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2023-07-14T19:07:11.689 INFO:teuthology.orchestra.run.smithi040.stdout: python3-pycparser-2.14-14.el8.noarch 2023-07-14T19:07:11.689 INFO:teuthology.orchestra.run.smithi040.stdout: python3-pytz-2017.2-9.el8.noarch 2023-07-14T19:07:11.689 INFO:teuthology.orchestra.run.smithi040.stdout: python3-rados-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T19:07:11.690 INFO:teuthology.orchestra.run.smithi040.stdout: python3-rbd-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T19:07:11.690 INFO:teuthology.orchestra.run.smithi040.stdout: python3-repoze-lru-0.7-6.el8.noarch 2023-07-14T19:07:11.690 INFO:teuthology.orchestra.run.smithi040.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2023-07-14T19:07:11.690 INFO:teuthology.orchestra.run.smithi040.stdout: python3-rgw-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T19:07:11.690 INFO:teuthology.orchestra.run.smithi040.stdout: python3-routes-2.4.1-12.el8.noarch 2023-07-14T19:07:11.690 INFO:teuthology.orchestra.run.smithi040.stdout: python3-rsa-4.8-1.el8.noarch 2023-07-14T19:07:11.690 INFO:teuthology.orchestra.run.smithi040.stdout: python3-scipy-1.0.0-21.module+el8.5.0+10916+41bd434d.x86_64 2023-07-14T19:07:11.690 INFO:teuthology.orchestra.run.smithi040.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2023-07-14T19:07:11.691 INFO:teuthology.orchestra.run.smithi040.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2023-07-14T19:07:11.691 INFO:teuthology.orchestra.run.smithi040.stdout: python3-tempora-1.14.1-5.el8.noarch 2023-07-14T19:07:11.691 INFO:teuthology.orchestra.run.smithi040.stdout: python3-trustme-0.6.0-4.el8.noarch 2023-07-14T19:07:11.691 INFO:teuthology.orchestra.run.smithi040.stdout: python3-waitress-1.4.3-1.el8.noarch 2023-07-14T19:07:11.691 INFO:teuthology.orchestra.run.smithi040.stdout: python3-webob-1.8.5-1.el8.1.noarch 2023-07-14T19:07:11.691 INFO:teuthology.orchestra.run.smithi040.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2023-07-14T19:07:11.691 INFO:teuthology.orchestra.run.smithi040.stdout: python3-webtest-2.0.33-1.el8.noarch 2023-07-14T19:07:11.692 INFO:teuthology.orchestra.run.smithi040.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2023-07-14T19:07:11.692 INFO:teuthology.orchestra.run.smithi040.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2023-07-14T19:07:11.692 INFO:teuthology.orchestra.run.smithi040.stdout: rbd-fuse-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T19:07:11.692 INFO:teuthology.orchestra.run.smithi040.stdout: rbd-mirror-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T19:07:11.692 INFO:teuthology.orchestra.run.smithi040.stdout: rbd-nbd-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T19:07:11.692 INFO:teuthology.orchestra.run.smithi040.stdout: socat-1.7.4.1-1.el8.x86_64 2023-07-14T19:07:11.692 INFO:teuthology.orchestra.run.smithi040.stdout: thrift-0.13.0-2.el8.x86_64 2023-07-14T19:07:11.692 INFO:teuthology.orchestra.run.smithi040.stdout: uuid-1.6.2-43.el8.x86_64 2023-07-14T19:07:11.693 INFO:teuthology.orchestra.run.smithi040.stdout: xfsprogs-devel-5.0.0-10.el8.x86_64 2023-07-14T19:07:11.693 INFO:teuthology.orchestra.run.smithi040.stdout: xmlstarlet-1.6.1-20.el8.x86_64 2023-07-14T19:07:11.693 INFO:teuthology.orchestra.run.smithi040.stdout: zlib-devel-1.2.11-18.el8_5.x86_64 2023-07-14T19:07:11.693 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-07-14T19:07:11.694 INFO:teuthology.orchestra.run.smithi040.stdout:Complete! 2023-07-14T19:07:11.883 DEBUG:teuthology.parallel:result is None 2023-07-14T19:07:11.884 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=81c9387d2e7576da065747ec9c16d1c7d69016ec 2023-07-14T19:07:17.475 DEBUG:teuthology.orchestra.run.smithi040:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2023-07-14T19:07:17.505 INFO:teuthology.orchestra.run.smithi040.stdout:18.0.0-4940.g81c9387d.el8 2023-07-14T19:07:17.505 INFO:teuthology.packaging:The installed version of ceph is 18.0.0-4940.g81c9387d.el8 2023-07-14T19:07:17.506 INFO:teuthology.task.install:The correct ceph version 18.0.0-4940.g81c9387d is installed. 2023-07-14T19:07:17.507 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=81c9387d2e7576da065747ec9c16d1c7d69016ec 2023-07-14T19:07:23.036 DEBUG:teuthology.orchestra.run.smithi130:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2023-07-14T19:07:23.064 INFO:teuthology.orchestra.run.smithi130.stdout:18.0.0-4940.g81c9387d.el8 2023-07-14T19:07:23.065 INFO:teuthology.packaging:The installed version of ceph is 18.0.0-4940.g81c9387d.el8 2023-07-14T19:07:23.065 INFO:teuthology.task.install:The correct ceph version 18.0.0-4940.g81c9387d is installed. 2023-07-14T19:07:23.065 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=81c9387d2e7576da065747ec9c16d1c7d69016ec 2023-07-14T19:07:28.483 DEBUG:teuthology.orchestra.run.smithi191:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2023-07-14T19:07:28.511 INFO:teuthology.orchestra.run.smithi191.stdout:18.0.0-4940.g81c9387d.el8 2023-07-14T19:07:28.511 INFO:teuthology.packaging:The installed version of ceph is 18.0.0-4940.g81c9387d.el8 2023-07-14T19:07:28.512 INFO:teuthology.task.install:The correct ceph version 18.0.0-4940.g81c9387d is installed. 2023-07-14T19:07:28.512 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2023-07-14T19:07:28.513 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2023-07-14T19:07:28.513 DEBUG:teuthology.orchestra.run.smithi040:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-07-14T19:07:28.564 DEBUG:teuthology.orchestra.run.smithi130:> set -ex 2023-07-14T19:07:28.565 DEBUG:teuthology.orchestra.run.smithi130:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-07-14T19:07:28.613 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2023-07-14T19:07:28.613 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-07-14T19:07:28.660 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2023-07-14T19:07:28.661 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2023-07-14T19:07:28.662 DEBUG:teuthology.orchestra.run.smithi040:> sudo dd of=/usr/bin/daemon-helper 2023-07-14T19:07:28.694 DEBUG:teuthology.orchestra.run.smithi040:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-07-14T19:07:28.763 DEBUG:teuthology.orchestra.run.smithi130:> set -ex 2023-07-14T19:07:28.764 DEBUG:teuthology.orchestra.run.smithi130:> sudo dd of=/usr/bin/daemon-helper 2023-07-14T19:07:28.794 DEBUG:teuthology.orchestra.run.smithi130:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-07-14T19:07:28.864 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2023-07-14T19:07:28.864 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd of=/usr/bin/daemon-helper 2023-07-14T19:07:28.897 DEBUG:teuthology.orchestra.run.smithi191:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-07-14T19:07:28.968 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2023-07-14T19:07:28.969 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2023-07-14T19:07:28.969 DEBUG:teuthology.orchestra.run.smithi040:> sudo dd of=/usr/bin/adjust-ulimits 2023-07-14T19:07:29.000 DEBUG:teuthology.orchestra.run.smithi040:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-07-14T19:07:29.070 DEBUG:teuthology.orchestra.run.smithi130:> set -ex 2023-07-14T19:07:29.071 DEBUG:teuthology.orchestra.run.smithi130:> sudo dd of=/usr/bin/adjust-ulimits 2023-07-14T19:07:29.101 DEBUG:teuthology.orchestra.run.smithi130:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-07-14T19:07:29.172 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2023-07-14T19:07:29.172 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd of=/usr/bin/adjust-ulimits 2023-07-14T19:07:29.203 DEBUG:teuthology.orchestra.run.smithi191:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-07-14T19:07:29.274 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2023-07-14T19:07:29.274 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2023-07-14T19:07:29.274 DEBUG:teuthology.orchestra.run.smithi040:> sudo dd of=/usr/bin/stdin-killer 2023-07-14T19:07:29.307 DEBUG:teuthology.orchestra.run.smithi040:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-07-14T19:07:29.379 DEBUG:teuthology.orchestra.run.smithi130:> set -ex 2023-07-14T19:07:29.380 DEBUG:teuthology.orchestra.run.smithi130:> sudo dd of=/usr/bin/stdin-killer 2023-07-14T19:07:29.410 DEBUG:teuthology.orchestra.run.smithi130:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-07-14T19:07:29.478 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2023-07-14T19:07:29.479 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd of=/usr/bin/stdin-killer 2023-07-14T19:07:29.510 DEBUG:teuthology.orchestra.run.smithi191:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-07-14T19:07:29.580 INFO:teuthology.run_tasks:Running task cephadm... 2023-07-14T19:07:29.673 INFO:tasks.cephadm:Config: {'roleless': False, 'cephfs': {'max_mds': 5, 'session_timeout': 300, 'standby_replay': True}, 'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'ms die on skipped message': False}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds bal fragment size max': 10000, 'mds bal merge size': 5, 'mds bal split bits': 3, 'mds bal split size': 100, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'cephsqlite lock renewal timeout': 900000, 'debug cephsqlite': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180, 'osd shutdown pgref assert': True, 'osd_max_omap_entries_per_request': 10}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'overall HEALTH_', '\\(FS_DEGRADED\\)', '\\(MDS_FAILED\\)', '\\(MDS_DEGRADED\\)', '\\(FS_WITH_FAILED_MDS\\)', '\\(MDS_DAMAGE\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(FS_INLINE_DATA_DEPRECATED\\)', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'slow metadata IO', 'SLOW_OPS', 'slow request'], 'sha1': '81c9387d2e7576da065747ec9c16d1c7d69016ec', 'subvols': {'create': 2, 'subvol_options': '--namespace-isolated --size 25000000000'}} 2023-07-14T19:07:29.674 INFO:tasks.cephadm:Cluster image is quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec 2023-07-14T19:07:29.683 INFO:tasks.cephadm:Cluster fsid is adc52f04-2279-11ee-9b33-001a4aab830c 2023-07-14T19:07:29.684 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2023-07-14T19:07:29.684 INFO:tasks.cephadm:Monitor IPs: {'mon.a': '172.21.15.40', 'mon.b': '172.21.15.130', 'mon.c': '172.21.15.191'} 2023-07-14T19:07:29.685 INFO:tasks.cephadm:First mon is mon.a on smithi040 2023-07-14T19:07:29.685 INFO:tasks.cephadm:First mgr is x 2023-07-14T19:07:29.685 INFO:tasks.cephadm:Normalizing hostnames... 2023-07-14T19:07:29.685 DEBUG:teuthology.orchestra.run.smithi040:> sudo hostname $(hostname -s) 2023-07-14T19:07:29.720 DEBUG:teuthology.orchestra.run.smithi130:> sudo hostname $(hostname -s) 2023-07-14T19:07:29.752 DEBUG:teuthology.orchestra.run.smithi191:> sudo hostname $(hostname -s) 2023-07-14T19:07:29.785 INFO:tasks.cephadm:Downloading "compiled" cephadm from cachra 2023-07-14T19:07:29.787 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=81c9387d2e7576da065747ec9c16d1c7d69016ec 2023-07-14T19:07:35.295 INFO:tasks.cephadm:builder_project result: [{'status': 'ready', 'sha1': '81c9387d2e7576da065747ec9c16d1c7d69016ec', 'extra': {'build_url': 'https://jenkins.ceph.com/job/ceph-dev-new-build/ARCH=arm64,AVAILABLE_ARCH=arm64,AVAILABLE_DIST=centos8,DIST=centos8,MACHINE_SIZE=gigantic/71431/', 'root_build_cause': 'SCMTRIGGER', 'version': '18.0.0-4940-g81c9387d', 'node_name': '172.21.4.65+confusa03', 'job_name': 'ceph-dev-new-build/ARCH=arm64,AVAILABLE_ARCH=arm64,AVAILABLE_DIST=centos8,DIST=centos8,MACHINE_SIZE=gigantic', 'package_manager_version': '18.0.0-4940.g81c9387d'}, 'url': 'https://4.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20230714.134335/81c9387d2e7576da065747ec9c16d1c7d69016ec/centos/8/flavors/default/', 'distro_codename': None, 'modified': '2023-07-14 16:50:26.249583', 'distro_version': '8', 'project': 'ceph', 'flavor': 'default', 'ref': 'wip-pdonnell-testing-20230714.134335', 'chacra_url': 'https://4.chacra.ceph.com/repos/ceph/wip-pdonnell-testing-20230714.134335/81c9387d2e7576da065747ec9c16d1c7d69016ec/centos/8/flavors/default/', 'archs': ['source', 'x86_64', 'arm64'], 'distro': 'centos'}] 2023-07-14T19:07:40.814 INFO:tasks.util.chacra:got chacra host 4.chacra.ceph.com, ref wip-pdonnell-testing-20230714.134335, sha1 81c9387d2e7576da065747ec9c16d1c7d69016ec from https://shaman.ceph.com/api/search/?project=ceph&distros=centos%2F8%2Fx86_64&flavor=default&sha1=81c9387d2e7576da065747ec9c16d1c7d69016ec 2023-07-14T19:07:40.814 INFO:tasks.cephadm:Discovered cachra url: https://4.chacra.ceph.com/binaries/ceph/wip-pdonnell-testing-20230714.134335/81c9387d2e7576da065747ec9c16d1c7d69016ec/centos/8/x86_64/flavors/default/cephadm 2023-07-14T19:07:40.815 DEBUG:teuthology.orchestra.run.smithi040:> curl --silent -L https://4.chacra.ceph.com/binaries/ceph/wip-pdonnell-testing-20230714.134335/81c9387d2e7576da065747ec9c16d1c7d69016ec/centos/8/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2023-07-14T19:07:42.178 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-rw-r--. 1 ubuntu ubuntu 670007 Jul 14 19:07 /home/ubuntu/cephtest/cephadm 2023-07-14T19:07:42.178 DEBUG:teuthology.orchestra.run.smithi130:> curl --silent -L https://4.chacra.ceph.com/binaries/ceph/wip-pdonnell-testing-20230714.134335/81c9387d2e7576da065747ec9c16d1c7d69016ec/centos/8/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2023-07-14T19:07:43.367 INFO:teuthology.orchestra.run.smithi130.stdout:-rw-rw-r--. 1 ubuntu ubuntu 670007 Jul 14 19:07 /home/ubuntu/cephtest/cephadm 2023-07-14T19:07:43.368 DEBUG:teuthology.orchestra.run.smithi191:> curl --silent -L https://4.chacra.ceph.com/binaries/ceph/wip-pdonnell-testing-20230714.134335/81c9387d2e7576da065747ec9c16d1c7d69016ec/centos/8/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2023-07-14T19:07:44.369 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-rw-r--. 1 ubuntu ubuntu 670007 Jul 14 19:07 /home/ubuntu/cephtest/cephadm 2023-07-14T19:07:44.369 DEBUG:teuthology.orchestra.run.smithi040:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2023-07-14T19:07:44.391 DEBUG:teuthology.orchestra.run.smithi130:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2023-07-14T19:07:44.413 DEBUG:teuthology.orchestra.run.smithi191:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2023-07-14T19:07:44.448 INFO:tasks.cephadm:Pulling image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec on all hosts... 2023-07-14T19:07:44.449 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec pull 2023-07-14T19:07:44.451 DEBUG:teuthology.orchestra.run.smithi130:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec pull 2023-07-14T19:07:44.456 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec pull 2023-07-14T19:07:45.687 INFO:teuthology.orchestra.run.smithi130.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec... 2023-07-14T19:07:45.709 INFO:teuthology.orchestra.run.smithi191.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec... 2023-07-14T19:07:45.767 INFO:teuthology.orchestra.run.smithi040.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec... 2023-07-14T19:08:24.210 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2023-07-14T19:08:24.211 INFO:teuthology.orchestra.run.smithi040.stdout: "ceph_version": "ceph version 18.0.0-4940-g81c9387d (81c9387d2e7576da065747ec9c16d1c7d69016ec) reef (dev)", 2023-07-14T19:08:24.211 INFO:teuthology.orchestra.run.smithi040.stdout: "image_id": "219b180907a819a02c0d1790ec69e152120a3974adf061aaf947f90f6ccf9705", 2023-07-14T19:08:24.211 INFO:teuthology.orchestra.run.smithi040.stdout: "repo_digests": [ 2023-07-14T19:08:24.211 INFO:teuthology.orchestra.run.smithi040.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1302190083259ae87fcd27babf423c1dd04ae7417edb252031c1d861adeb3110" 2023-07-14T19:08:24.212 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2023-07-14T19:08:24.212 INFO:teuthology.orchestra.run.smithi040.stdout:} 2023-07-14T19:08:33.974 INFO:teuthology.orchestra.run.smithi130.stdout:{ 2023-07-14T19:08:33.974 INFO:teuthology.orchestra.run.smithi130.stdout: "ceph_version": "ceph version 18.0.0-4940-g81c9387d (81c9387d2e7576da065747ec9c16d1c7d69016ec) reef (dev)", 2023-07-14T19:08:33.975 INFO:teuthology.orchestra.run.smithi130.stdout: "image_id": "219b180907a819a02c0d1790ec69e152120a3974adf061aaf947f90f6ccf9705", 2023-07-14T19:08:33.975 INFO:teuthology.orchestra.run.smithi130.stdout: "repo_digests": [ 2023-07-14T19:08:33.975 INFO:teuthology.orchestra.run.smithi130.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1302190083259ae87fcd27babf423c1dd04ae7417edb252031c1d861adeb3110" 2023-07-14T19:08:33.975 INFO:teuthology.orchestra.run.smithi130.stdout: ] 2023-07-14T19:08:33.975 INFO:teuthology.orchestra.run.smithi130.stdout:} 2023-07-14T19:08:41.050 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2023-07-14T19:08:41.050 INFO:teuthology.orchestra.run.smithi191.stdout: "ceph_version": "ceph version 18.0.0-4940-g81c9387d (81c9387d2e7576da065747ec9c16d1c7d69016ec) reef (dev)", 2023-07-14T19:08:41.050 INFO:teuthology.orchestra.run.smithi191.stdout: "image_id": "219b180907a819a02c0d1790ec69e152120a3974adf061aaf947f90f6ccf9705", 2023-07-14T19:08:41.050 INFO:teuthology.orchestra.run.smithi191.stdout: "repo_digests": [ 2023-07-14T19:08:41.050 INFO:teuthology.orchestra.run.smithi191.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1302190083259ae87fcd27babf423c1dd04ae7417edb252031c1d861adeb3110" 2023-07-14T19:08:41.051 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2023-07-14T19:08:41.051 INFO:teuthology.orchestra.run.smithi191.stdout:} 2023-07-14T19:08:41.069 DEBUG:teuthology.orchestra.run.smithi040:> sudo mkdir -p /etc/ceph 2023-07-14T19:08:41.108 DEBUG:teuthology.orchestra.run.smithi130:> sudo mkdir -p /etc/ceph 2023-07-14T19:08:41.146 DEBUG:teuthology.orchestra.run.smithi191:> sudo mkdir -p /etc/ceph 2023-07-14T19:08:41.175 DEBUG:teuthology.orchestra.run.smithi040:> sudo chmod 777 /etc/ceph 2023-07-14T19:08:41.207 DEBUG:teuthology.orchestra.run.smithi130:> sudo chmod 777 /etc/ceph 2023-07-14T19:08:41.237 DEBUG:teuthology.orchestra.run.smithi191:> sudo chmod 777 /etc/ceph 2023-07-14T19:08:41.265 INFO:tasks.cephadm:Writing seed config... 2023-07-14T19:08:41.267 INFO:tasks.cephadm: override: [client] client mount timeout = 600 2023-07-14T19:08:41.268 INFO:tasks.cephadm: override: [client] debug client = 20 2023-07-14T19:08:41.268 INFO:tasks.cephadm: override: [client] debug ms = 1 2023-07-14T19:08:41.269 INFO:tasks.cephadm: override: [client] rados mon op timeout = 900 2023-07-14T19:08:41.269 INFO:tasks.cephadm: override: [client] rados osd op timeout = 900 2023-07-14T19:08:41.269 INFO:tasks.cephadm: override: [global] ms die on skipped message = False 2023-07-14T19:08:41.270 INFO:tasks.cephadm: override: [mds] debug mds = 20 2023-07-14T19:08:41.270 INFO:tasks.cephadm: override: [mds] debug mds balancer = 20 2023-07-14T19:08:41.271 INFO:tasks.cephadm: override: [mds] debug ms = 1 2023-07-14T19:08:41.271 INFO:tasks.cephadm: override: [mds] mds bal fragment size max = 10000 2023-07-14T19:08:41.271 INFO:tasks.cephadm: override: [mds] mds bal merge size = 5 2023-07-14T19:08:41.271 INFO:tasks.cephadm: override: [mds] mds bal split bits = 3 2023-07-14T19:08:41.272 INFO:tasks.cephadm: override: [mds] mds bal split size = 100 2023-07-14T19:08:41.272 INFO:tasks.cephadm: override: [mds] mds debug frag = True 2023-07-14T19:08:41.272 INFO:tasks.cephadm: override: [mds] mds debug scatterstat = True 2023-07-14T19:08:41.272 INFO:tasks.cephadm: override: [mds] mds op complaint time = 180 2023-07-14T19:08:41.272 INFO:tasks.cephadm: override: [mds] mds verify scatter = True 2023-07-14T19:08:41.273 INFO:tasks.cephadm: override: [mds] osd op complaint time = 180 2023-07-14T19:08:41.273 INFO:tasks.cephadm: override: [mds] rados mon op timeout = 900 2023-07-14T19:08:41.273 INFO:tasks.cephadm: override: [mds] rados osd op timeout = 900 2023-07-14T19:08:41.273 INFO:tasks.cephadm: override: [mgr] cephsqlite lock renewal timeout = 900000 2023-07-14T19:08:41.273 INFO:tasks.cephadm: override: [mgr] debug cephsqlite = 20 2023-07-14T19:08:41.274 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2023-07-14T19:08:41.274 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2023-07-14T19:08:41.274 INFO:tasks.cephadm: override: [mon] debug mon = 20 2023-07-14T19:08:41.274 INFO:tasks.cephadm: override: [mon] debug ms = 1 2023-07-14T19:08:41.275 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2023-07-14T19:08:41.275 INFO:tasks.cephadm: override: [mon] mon op complaint time = 120 2023-07-14T19:08:41.275 INFO:tasks.cephadm: override: [osd] bluestore block size = 96636764160 2023-07-14T19:08:41.275 INFO:tasks.cephadm: override: [osd] bluestore compression mode = aggressive 2023-07-14T19:08:41.275 INFO:tasks.cephadm: override: [osd] bluestore fsck on mount = True 2023-07-14T19:08:41.276 INFO:tasks.cephadm: override: [osd] debug bluefs = 20 2023-07-14T19:08:41.276 INFO:tasks.cephadm: override: [osd] debug bluestore = 20 2023-07-14T19:08:41.276 INFO:tasks.cephadm: override: [osd] debug ms = 1 2023-07-14T19:08:41.277 INFO:tasks.cephadm: override: [osd] debug osd = 20 2023-07-14T19:08:41.277 INFO:tasks.cephadm: override: [osd] debug rocksdb = 10 2023-07-14T19:08:41.278 INFO:tasks.cephadm: override: [osd] mon osd backfillfull_ratio = 0.85 2023-07-14T19:08:41.278 INFO:tasks.cephadm: override: [osd] mon osd full ratio = 0.9 2023-07-14T19:08:41.279 INFO:tasks.cephadm: override: [osd] mon osd nearfull ratio = 0.8 2023-07-14T19:08:41.279 INFO:tasks.cephadm: override: [osd] osd failsafe full ratio = 0.95 2023-07-14T19:08:41.280 INFO:tasks.cephadm: override: [osd] osd objectstore = bluestore 2023-07-14T19:08:41.280 INFO:tasks.cephadm: override: [osd] osd op complaint time = 180 2023-07-14T19:08:41.281 INFO:tasks.cephadm: override: [osd] osd shutdown pgref assert = True 2023-07-14T19:08:41.281 INFO:tasks.cephadm: override: [osd] osd_max_omap_entries_per_request = 10 2023-07-14T19:08:41.283 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2023-07-14T19:08:41.284 DEBUG:teuthology.orchestra.run.smithi040:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2023-07-14T19:08:41.303 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = adc52f04-2279-11ee-9b33-001a4aab830c ms die on skipped message = False [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = True bdev debug aio = true osd sloppy crc = true bluestore block size = 96636764160 bluestore compression mode = aggressive bluestore fsck on mount = True debug bluefs = 20 debug bluestore = 20 debug ms = 1 debug osd = 20 debug rocksdb = 10 mon osd backfillfull_ratio = 0.85 mon osd full ratio = 0.9 mon osd nearfull ratio = 0.8 osd failsafe full ratio = 0.95 osd objectstore = bluestore osd op complaint time = 180 osd_max_omap_entries_per_request = 10 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false cephsqlite lock renewal timeout = 900000 debug cephsqlite = 20 debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 mon op complaint time = 120 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true [client] client mount timeout = 600 debug client = 20 debug ms = 1 rados mon op timeout = 900 rados osd op timeout = 900 [mds] debug mds = 20 debug mds balancer = 20 debug ms = 1 mds bal fragment size max = 10000 mds bal merge size = 5 mds bal split bits = 3 mds bal split size = 100 mds debug frag = True mds debug scatterstat = True mds op complaint time = 180 mds verify scatter = True osd op complaint time = 180 rados mon op timeout = 900 rados osd op timeout = 900 2023-07-14T19:08:41.304 DEBUG:teuthology.orchestra.run.smithi040:mon.a> sudo journalctl -f -n 0 -u ceph-adc52f04-2279-11ee-9b33-001a4aab830c@mon.a.service 2023-07-14T19:08:41.345 DEBUG:teuthology.orchestra.run.smithi040:mgr.x> sudo journalctl -f -n 0 -u ceph-adc52f04-2279-11ee-9b33-001a4aab830c@mgr.x.service 2023-07-14T19:08:41.376 INFO:journalctl@ceph.mon.a.smithi040.stdout:-- Logs begin at Fri 2023-07-14 18:50:26 UTC. -- 2023-07-14T19:08:41.377 INFO:tasks.cephadm:Bootstrapping... 2023-07-14T19:08:41.377 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec -v bootstrap --fsid adc52f04-2279-11ee-9b33-001a4aab830c --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id x --orphan-initial-daemons --skip-monitoring-stack --mon-ip 172.21.15.40 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2023-07-14T19:08:41.406 INFO:journalctl@ceph.mgr.x.smithi040.stdout:-- Logs begin at Fri 2023-07-14 18:50:26 UTC. -- 2023-07-14T19:08:41.549 INFO:teuthology.orchestra.run.smithi040.stdout:-------------------------------------------------------------------------------- 2023-07-14T19:08:41.549 INFO:teuthology.orchestra.run.smithi040.stdout:cephadm ['--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec', '-v', 'bootstrap', '--fsid', 'adc52f04-2279-11ee-9b33-001a4aab830c', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-id', 'a', '--mgr-id', 'x', '--orphan-initial-daemons', '--skip-monitoring-stack', '--mon-ip', '172.21.15.40', '--skip-admin-label'] 2023-07-14T19:08:41.696 INFO:teuthology.orchestra.run.smithi040.stdout:/bin/podman: stdout 4.1.1 2023-07-14T19:08:41.696 INFO:teuthology.orchestra.run.smithi040.stderr:Specifying an fsid for your cluster offers no advantages and may increase the likelihood of fsid conflicts. 2023-07-14T19:08:41.697 INFO:teuthology.orchestra.run.smithi040.stdout:Verifying podman|docker is present... 2023-07-14T19:08:41.840 INFO:teuthology.orchestra.run.smithi040.stdout:/bin/podman: stdout 4.1.1 2023-07-14T19:08:41.840 INFO:teuthology.orchestra.run.smithi040.stdout:Verifying lvm2 is present... 2023-07-14T19:08:41.840 INFO:teuthology.orchestra.run.smithi040.stdout:Verifying time synchronization is in place... 2023-07-14T19:08:41.847 INFO:teuthology.orchestra.run.smithi040.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2023-07-14T19:08:41.847 INFO:teuthology.orchestra.run.smithi040.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2023-07-14T19:08:41.853 INFO:teuthology.orchestra.run.smithi040.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2023-07-14T19:08:41.853 INFO:teuthology.orchestra.run.smithi040.stdout:systemctl: stdout inactive 2023-07-14T19:08:41.859 INFO:teuthology.orchestra.run.smithi040.stdout:systemctl: stdout enabled 2023-07-14T19:08:41.865 INFO:teuthology.orchestra.run.smithi040.stdout:systemctl: stdout active 2023-07-14T19:08:41.865 INFO:teuthology.orchestra.run.smithi040.stdout:Unit chronyd.service is enabled and running 2023-07-14T19:08:41.866 INFO:teuthology.orchestra.run.smithi040.stdout:Repeating the final host check... 2023-07-14T19:08:42.012 INFO:teuthology.orchestra.run.smithi040.stdout:/bin/podman: stdout 4.1.1 2023-07-14T19:08:42.013 INFO:teuthology.orchestra.run.smithi040.stdout:podman (/bin/podman) version 4.1.1 is present 2023-07-14T19:08:42.013 INFO:teuthology.orchestra.run.smithi040.stdout:systemctl is present 2023-07-14T19:08:42.013 INFO:teuthology.orchestra.run.smithi040.stdout:lvcreate is present 2023-07-14T19:08:42.018 INFO:teuthology.orchestra.run.smithi040.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2023-07-14T19:08:42.018 INFO:teuthology.orchestra.run.smithi040.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2023-07-14T19:08:42.024 INFO:teuthology.orchestra.run.smithi040.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2023-07-14T19:08:42.024 INFO:teuthology.orchestra.run.smithi040.stdout:systemctl: stdout inactive 2023-07-14T19:08:42.030 INFO:teuthology.orchestra.run.smithi040.stdout:systemctl: stdout enabled 2023-07-14T19:08:42.036 INFO:teuthology.orchestra.run.smithi040.stdout:systemctl: stdout active 2023-07-14T19:08:42.036 INFO:teuthology.orchestra.run.smithi040.stdout:Unit chronyd.service is enabled and running 2023-07-14T19:08:42.036 INFO:teuthology.orchestra.run.smithi040.stdout:Host looks OK 2023-07-14T19:08:42.036 INFO:teuthology.orchestra.run.smithi040.stdout:Cluster fsid: adc52f04-2279-11ee-9b33-001a4aab830c 2023-07-14T19:08:42.037 INFO:teuthology.orchestra.run.smithi040.stdout:Acquiring lock 140157132151272 on /run/cephadm/adc52f04-2279-11ee-9b33-001a4aab830c.lock 2023-07-14T19:08:42.037 INFO:teuthology.orchestra.run.smithi040.stdout:Lock 140157132151272 acquired on /run/cephadm/adc52f04-2279-11ee-9b33-001a4aab830c.lock 2023-07-14T19:08:42.037 INFO:teuthology.orchestra.run.smithi040.stdout:Verifying IP 172.21.15.40 port 3300 ... 2023-07-14T19:08:42.037 INFO:teuthology.orchestra.run.smithi040.stdout:Verifying IP 172.21.15.40 port 6789 ... 2023-07-14T19:08:42.038 INFO:teuthology.orchestra.run.smithi040.stdout:Base mon IP(s) is [172.21.15.40:3300, 172.21.15.40:6789], mon addrv is [v2:172.21.15.40:3300,v1:172.21.15.40:6789] 2023-07-14T19:08:42.041 INFO:teuthology.orchestra.run.smithi040.stdout:/sbin/ip: stdout default via 172.21.15.254 dev ens1f0 proto dhcp metric 100 2023-07-14T19:08:42.041 INFO:teuthology.orchestra.run.smithi040.stdout:/sbin/ip: stdout 172.21.0.0/20 dev ens1f0 proto kernel scope link src 172.21.15.40 metric 100 2023-07-14T19:08:42.043 INFO:teuthology.orchestra.run.smithi040.stdout:/sbin/ip: stdout ::1 dev lo proto kernel metric 256 pref medium 2023-07-14T19:08:42.044 INFO:teuthology.orchestra.run.smithi040.stdout:/sbin/ip: stdout fe80::/64 dev ens1f0 proto kernel metric 256 pref medium 2023-07-14T19:08:42.044 INFO:teuthology.orchestra.run.smithi040.stdout:/sbin/ip: stdout default via fe80::327c:5e00:6487:71e0 dev ens1f0 proto ra metric 1024 expires 1780sec hoplimit 64 pref medium 2023-07-14T19:08:42.046 INFO:teuthology.orchestra.run.smithi040.stdout:/sbin/ip: stdout 1: lo: mtu 65536 state UNKNOWN qlen 1000 2023-07-14T19:08:42.046 INFO:teuthology.orchestra.run.smithi040.stdout:/sbin/ip: stdout inet6 ::1/128 scope host 2023-07-14T19:08:42.046 INFO:teuthology.orchestra.run.smithi040.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2023-07-14T19:08:42.047 INFO:teuthology.orchestra.run.smithi040.stdout:/sbin/ip: stdout 4: ens1f0: mtu 1500 state UP qlen 1000 2023-07-14T19:08:42.047 INFO:teuthology.orchestra.run.smithi040.stdout:/sbin/ip: stdout inet6 fe80::ec4:7aff:febd:12cc/64 scope link 2023-07-14T19:08:42.047 INFO:teuthology.orchestra.run.smithi040.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2023-07-14T19:08:42.047 INFO:teuthology.orchestra.run.smithi040.stdout:Mon IP `172.21.15.40` is in CIDR network `172.21.0.0/20` 2023-07-14T19:08:42.048 INFO:teuthology.orchestra.run.smithi040.stdout:Mon IP `172.21.15.40` is in CIDR network `172.21.0.0/20` 2023-07-14T19:08:42.048 INFO:teuthology.orchestra.run.smithi040.stdout:Inferred mon public CIDR from local network configuration ['172.21.0.0/20', '172.21.0.0/20'] 2023-07-14T19:08:42.048 INFO:teuthology.orchestra.run.smithi040.stdout:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2023-07-14T19:08:42.049 INFO:teuthology.orchestra.run.smithi040.stdout:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec... 2023-07-14T19:08:42.500 INFO:teuthology.orchestra.run.smithi040.stdout:/bin/podman: stdout 219b180907a819a02c0d1790ec69e152120a3974adf061aaf947f90f6ccf9705 2023-07-14T19:08:42.501 INFO:teuthology.orchestra.run.smithi040.stdout:/bin/podman: stderr Trying to pull quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec... 2023-07-14T19:08:42.501 INFO:teuthology.orchestra.run.smithi040.stdout:/bin/podman: stderr Getting image source signatures 2023-07-14T19:08:42.501 INFO:teuthology.orchestra.run.smithi040.stdout:/bin/podman: stderr Copying blob sha256:0e454b422ba90e2ad13e636c40610401086a3576fe973b0460943839fe5f2508 2023-07-14T19:08:42.501 INFO:teuthology.orchestra.run.smithi040.stdout:/bin/podman: stderr Copying blob sha256:6022c611d474f2b306ddd1edfc8360123ece5266119096f25fde843d7967e636 2023-07-14T19:08:42.501 INFO:teuthology.orchestra.run.smithi040.stdout:/bin/podman: stderr Copying config sha256:219b180907a819a02c0d1790ec69e152120a3974adf061aaf947f90f6ccf9705 2023-07-14T19:08:42.501 INFO:teuthology.orchestra.run.smithi040.stdout:/bin/podman: stderr Writing manifest to image destination 2023-07-14T19:08:42.501 INFO:teuthology.orchestra.run.smithi040.stdout:/bin/podman: stderr Storing signatures 2023-07-14T19:08:43.350 INFO:teuthology.orchestra.run.smithi040.stdout:ceph: stdout ceph version 18.0.0-4940-g81c9387d (81c9387d2e7576da065747ec9c16d1c7d69016ec) reef (dev) 2023-07-14T19:08:43.350 INFO:teuthology.orchestra.run.smithi040.stdout:Ceph version: ceph version 18.0.0-4940-g81c9387d (81c9387d2e7576da065747ec9c16d1c7d69016ec) reef (dev) 2023-07-14T19:08:43.350 INFO:teuthology.orchestra.run.smithi040.stdout:Extracting ceph user uid/gid from container image... 2023-07-14T19:08:44.098 INFO:teuthology.orchestra.run.smithi040.stdout:stat: stdout 167 167 2023-07-14T19:08:44.098 INFO:teuthology.orchestra.run.smithi040.stdout:Creating initial keys... 2023-07-14T19:08:44.850 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph-authtool: stdout AQA8nbFkkq6YGhAAS1S1yuNAJiSzWYC8C0JOAA== 2023-07-14T19:08:45.594 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph-authtool: stdout AQA9nbFkl62pDRAAFUKV0haJ9+R8v5aBZgxiZg== 2023-07-14T19:08:46.347 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph-authtool: stdout AQA9nbFkn7ZiOhAAWAIAYlBpCgzH99qVYwjDig== 2023-07-14T19:08:46.348 INFO:teuthology.orchestra.run.smithi040.stdout:Creating initial monmap... 2023-07-14T19:08:49.102 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: monmap file /tmp/monmap 2023-07-14T19:08:49.103 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/monmaptool: stdout setting min_mon_release = pacific 2023-07-14T19:08:49.103 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: set fsid to adc52f04-2279-11ee-9b33-001a4aab830c 2023-07-14T19:08:49.103 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2023-07-14T19:08:49.103 INFO:teuthology.orchestra.run.smithi040.stdout:monmaptool for a [v2:172.21.15.40:3300,v1:172.21.15.40:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2023-07-14T19:08:49.103 INFO:teuthology.orchestra.run.smithi040.stdout:setting min_mon_release = pacific 2023-07-14T19:08:49.104 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/monmaptool: set fsid to adc52f04-2279-11ee-9b33-001a4aab830c 2023-07-14T19:08:49.104 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2023-07-14T19:08:49.104 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-07-14T19:08:49.104 INFO:teuthology.orchestra.run.smithi040.stdout:Creating mon... 2023-07-14T19:08:53.272 INFO:teuthology.orchestra.run.smithi040.stdout:create mon.a on 2023-07-14T19:08:53.637 INFO:teuthology.orchestra.run.smithi040.stdout:systemctl: stderr Removed /etc/systemd/system/multi-user.target.wants/ceph.target. 2023-07-14T19:08:53.728 INFO:teuthology.orchestra.run.smithi040.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2023-07-14T19:08:54.078 INFO:teuthology.orchestra.run.smithi040.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph-adc52f04-2279-11ee-9b33-001a4aab830c.target → /etc/systemd/system/ceph-adc52f04-2279-11ee-9b33-001a4aab830c.target. 2023-07-14T19:08:54.078 INFO:teuthology.orchestra.run.smithi040.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph.target.wants/ceph-adc52f04-2279-11ee-9b33-001a4aab830c.target → /etc/systemd/system/ceph-adc52f04-2279-11ee-9b33-001a4aab830c.target. 2023-07-14T19:08:54.700 INFO:teuthology.orchestra.run.smithi040.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-adc52f04-2279-11ee-9b33-001a4aab830c@mon.a 2023-07-14T19:08:54.700 INFO:teuthology.orchestra.run.smithi040.stdout:systemctl: stderr Failed to reset failed state of unit ceph-adc52f04-2279-11ee-9b33-001a4aab830c@mon.a.service: Unit ceph-adc52f04-2279-11ee-9b33-001a4aab830c@mon.a.service not loaded. 2023-07-14T19:08:54.793 INFO:teuthology.orchestra.run.smithi040.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-adc52f04-2279-11ee-9b33-001a4aab830c.target.wants/ceph-adc52f04-2279-11ee-9b33-001a4aab830c@mon.a.service → /etc/systemd/system/ceph-adc52f04-2279-11ee-9b33-001a4aab830c@.service. 2023-07-14T19:08:54.972 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:08:54 smithi040 systemd[1]: Starting Ceph mon.a for adc52f04-2279-11ee-9b33-001a4aab830c... 2023-07-14T19:08:56.222 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:08:55 smithi040 podman[95978]: 2023-07-14T19:08:56.910 INFO:teuthology.orchestra.run.smithi040.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2023-07-14T19:08:56.911 INFO:teuthology.orchestra.run.smithi040.stdout:systemctl: stdout disabled 2023-07-14T19:08:56.918 INFO:teuthology.orchestra.run.smithi040.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2023-07-14T19:08:56.919 INFO:teuthology.orchestra.run.smithi040.stdout:systemctl: stdout inactive 2023-07-14T19:08:56.919 INFO:teuthology.orchestra.run.smithi040.stdout:firewalld.service is not enabled 2023-07-14T19:08:56.919 INFO:teuthology.orchestra.run.smithi040.stdout:Not possible to enable service . firewalld.service is not available 2023-07-14T19:08:56.920 INFO:teuthology.orchestra.run.smithi040.stdout:Waiting for mon to start... 2023-07-14T19:08:56.920 INFO:teuthology.orchestra.run.smithi040.stdout:Waiting for mon... 2023-07-14T19:08:57.147 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:08:56 smithi040 bash[95978]: 9b13594b5e2f4c4a90382fbdd7fd6f19a0f8ecc5aa9be30117ef0b833b88276a 2023-07-14T19:08:57.148 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:08:56 smithi040 systemd[1]: Started Ceph mon.a for adc52f04-2279-11ee-9b33-001a4aab830c. 2023-07-14T19:08:57.148 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:08:56 smithi040 ceph-mon[96019]: mkfs adc52f04-2279-11ee-9b33-001a4aab830c 2023-07-14T19:08:57.148 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:08:56 smithi040 ceph-mon[96019]: mon.a is new leader, mons a in quorum (ranks 0) 2023-07-14T19:08:57.974 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout cluster: 2023-07-14T19:08:57.975 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout id: adc52f04-2279-11ee-9b33-001a4aab830c 2023-07-14T19:08:57.975 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout health: HEALTH_OK 2023-07-14T19:08:57.975 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout 2023-07-14T19:08:57.975 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout services: 2023-07-14T19:08:57.976 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout mon: 1 daemons, quorum a (age 0.693819s) 2023-07-14T19:08:57.976 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout mgr: no daemons active 2023-07-14T19:08:57.976 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout osd: 0 osds: 0 up, 0 in 2023-07-14T19:08:57.976 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout 2023-07-14T19:08:57.976 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout data: 2023-07-14T19:08:57.976 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout pools: 0 pools, 0 pgs 2023-07-14T19:08:57.977 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout objects: 0 objects, 0 B 2023-07-14T19:08:57.977 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout usage: 0 B used, 0 B / 0 B avail 2023-07-14T19:08:57.977 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout pgs: 2023-07-14T19:08:57.977 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout 2023-07-14T19:08:57.977 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:08:57.447+0000 7f42da0cd700 1 Processor -- start 2023-07-14T19:08:57.977 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:08:57.447+0000 7f42da0cd700 1 -- start start 2023-07-14T19:08:57.977 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:08:57.447+0000 7f42da0cd700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f42d4105f40 0x7f42d4108320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:08:57.978 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:08:57.447+0000 7f42da0cd700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42d4079d20 con 0x7f42d4105f40 2023-07-14T19:08:57.978 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:08:57.448+0000 7f42d37fe700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f42d4105f40 0x7f42d4108320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:08:57.978 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:08:57.448+0000 7f42d37fe700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f42d4105f40 0x7f42d4108320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:59744/0 (socket says 172.21.15.40:59744) 2023-07-14T19:08:57.978 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:08:57.448+0000 7f42d37fe700 1 -- 172.21.15.40:0/2355314310 learned_addr learned my addr 172.21.15.40:0/2355314310 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-07-14T19:08:57.978 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:08:57.448+0000 7f42d37fe700 1 -- 172.21.15.40:0/2355314310 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42d4079e60 con 0x7f42d4105f40 2023-07-14T19:08:57.978 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:08:57.448+0000 7f42d37fe700 1 --2- 172.21.15.40:0/2355314310 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f42d4105f40 0x7f42d4108320 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f42c80098a0 tx=0x7f42c8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9e23bf98016297b1 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:08:57.978 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:08:57.448+0000 7f42d27fc700 1 -- 172.21.15.40:0/2355314310 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f42c8004160 con 0x7f42d4105f40 2023-07-14T19:08:57.979 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:08:57.449+0000 7f42d27fc700 1 -- 172.21.15.40:0/2355314310 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f42c80042c0 con 0x7f42d4105f40 2023-07-14T19:08:57.979 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:08:57.449+0000 7f42d27fc700 1 -- 172.21.15.40:0/2355314310 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f42c80045a0 con 0x7f42d4105f40 2023-07-14T19:08:57.979 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:08:57.449+0000 7f42da0cd700 1 -- 172.21.15.40:0/2355314310 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f42d4105f40 msgr2=0x7f42d4108320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:08:57.979 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:08:57.449+0000 7f42da0cd700 1 --2- 172.21.15.40:0/2355314310 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f42d4105f40 0x7f42d4108320 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f42c80098a0 tx=0x7f42c8009bb0 comp rx=0 tx=0).stop 2023-07-14T19:08:57.979 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:08:57.449+0000 7f42da0cd700 1 -- 172.21.15.40:0/2355314310 shutdown_connections 2023-07-14T19:08:57.979 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:08:57.449+0000 7f42da0cd700 1 --2- 172.21.15.40:0/2355314310 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f42d4105f40 0x7f42d4108320 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:08:57.979 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:08:57.449+0000 7f42da0cd700 1 -- 172.21.15.40:0/2355314310 >> 172.21.15.40:0/2355314310 conn(0x7f42d40ff9a0 msgr2=0x7f42d4101db0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:08:57.980 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:08:57.449+0000 7f42da0cd700 1 -- 172.21.15.40:0/2355314310 shutdown_connections 2023-07-14T19:08:57.980 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:08:57.449+0000 7f42da0cd700 1 -- 172.21.15.40:0/2355314310 wait complete. 2023-07-14T19:08:57.980 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:08:57.450+0000 7f42da0cd700 1 Processor -- start 2023-07-14T19:08:57.980 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:08:57.450+0000 7f42da0cd700 1 -- start start 2023-07-14T19:08:57.980 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:08:57.450+0000 7f42da0cd700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f42d4105f40 0x7f42d41058a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:08:57.980 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:08:57.450+0000 7f42da0cd700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42d4105de0 con 0x7f42d4105f40 2023-07-14T19:08:57.981 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:08:57.450+0000 7f42d37fe700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f42d4105f40 0x7f42d41058a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:08:57.981 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:08:57.450+0000 7f42d37fe700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f42d4105f40 0x7f42d41058a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:59746/0 (socket says 172.21.15.40:59746) 2023-07-14T19:08:57.981 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:08:57.450+0000 7f42d37fe700 1 -- 172.21.15.40:0/3841601539 learned_addr learned my addr 172.21.15.40:0/3841601539 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-07-14T19:08:57.981 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:08:57.451+0000 7f42d37fe700 1 -- 172.21.15.40:0/3841601539 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42c8009550 con 0x7f42d4105f40 2023-07-14T19:08:57.981 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:08:57.451+0000 7f42d37fe700 1 --2- 172.21.15.40:0/3841601539 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f42d4105f40 0x7f42d41058a0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f42c8004000 tx=0x7f42c8004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:08:57.982 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:08:57.451+0000 7f42d0ff9700 1 -- 172.21.15.40:0/3841601539 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f42c8011630 con 0x7f42d4105f40 2023-07-14T19:08:57.982 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:08:57.451+0000 7f42d0ff9700 1 -- 172.21.15.40:0/3841601539 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f42c8011790 con 0x7f42d4105f40 2023-07-14T19:08:57.982 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:08:57.451+0000 7f42da0cd700 1 -- 172.21.15.40:0/3841601539 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f42d4103f50 con 0x7f42d4105f40 2023-07-14T19:08:57.982 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:08:57.451+0000 7f42d0ff9700 1 -- 172.21.15.40:0/3841601539 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f42c8011a30 con 0x7f42d4105f40 2023-07-14T19:08:57.983 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:08:57.451+0000 7f42da0cd700 1 -- 172.21.15.40:0/3841601539 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f42d41043d0 con 0x7f42d4105f40 2023-07-14T19:08:57.983 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:08:57.451+0000 7f42d0ff9700 1 -- 172.21.15.40:0/3841601539 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f42c8022070 con 0x7f42d4105f40 2023-07-14T19:08:57.983 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:08:57.451+0000 7f42d0ff9700 1 -- 172.21.15.40:0/3841601539 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f42c801c450 con 0x7f42d4105f40 2023-07-14T19:08:57.983 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:08:57.452+0000 7f42da0cd700 1 -- 172.21.15.40:0/3841601539 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f42d4061c70 con 0x7f42d4105f40 2023-07-14T19:08:57.984 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:08:57.454+0000 7f42d0ff9700 1 -- 172.21.15.40:0/3841601539 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72446 (secure 0 0 0) 0x7f42c801b450 con 0x7f42d4105f40 2023-07-14T19:08:57.984 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:08:57.586+0000 7f42da0cd700 1 -- 172.21.15.40:0/3841601539 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "status"} v 0) v1 -- 0x7f42d41921e0 con 0x7f42d4105f40 2023-07-14T19:08:57.984 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:08:57.586+0000 7f42d0ff9700 1 -- 172.21.15.40:0/3841601539 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "status"}]=0 v0) v1 ==== 54+0+317 (secure 0 0 0) 0x7f42c801b5f0 con 0x7f42d4105f40 2023-07-14T19:08:57.984 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:08:57.587+0000 7f42da0cd700 1 -- 172.21.15.40:0/3841601539 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f42d4105f40 msgr2=0x7f42d41058a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:08:57.984 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:08:57.587+0000 7f42da0cd700 1 --2- 172.21.15.40:0/3841601539 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f42d4105f40 0x7f42d41058a0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f42c8004000 tx=0x7f42c8004750 comp rx=0 tx=0).stop 2023-07-14T19:08:57.985 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:08:57.587+0000 7f42da0cd700 1 -- 172.21.15.40:0/3841601539 shutdown_connections 2023-07-14T19:08:57.985 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:08:57.587+0000 7f42da0cd700 1 --2- 172.21.15.40:0/3841601539 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f42d4105f40 0x7f42d41058a0 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:08:57.985 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:08:57.587+0000 7f42da0cd700 1 -- 172.21.15.40:0/3841601539 >> 172.21.15.40:0/3841601539 conn(0x7f42d40ff9a0 msgr2=0x7f42d41012c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:08:57.985 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:08:57.587+0000 7f42da0cd700 1 -- 172.21.15.40:0/3841601539 shutdown_connections 2023-07-14T19:08:57.985 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:08:57.587+0000 7f42da0cd700 1 -- 172.21.15.40:0/3841601539 wait complete. 2023-07-14T19:08:57.985 INFO:teuthology.orchestra.run.smithi040.stdout:mon is available 2023-07-14T19:08:57.985 INFO:teuthology.orchestra.run.smithi040.stdout:Assimilating anything we can from ceph.conf... 2023-07-14T19:08:58.154 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:08:57 smithi040 ceph-mon[96019]: mon.a is new leader, mons a in quorum (ranks 0) 2023-07-14T19:08:58.154 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:08:57 smithi040 ceph-mon[96019]: monmap e1: 1 mons at {a=[v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0]} removed_ranks: {} 2023-07-14T19:08:58.154 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:08:57 smithi040 ceph-mon[96019]: fsmap 2023-07-14T19:08:58.154 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:08:57 smithi040 ceph-mon[96019]: osdmap e1: 0 total, 0 up, 0 in 2023-07-14T19:08:58.155 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:08:57 smithi040 ceph-mon[96019]: mgrmap e1: no daemons active 2023-07-14T19:08:58.155 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:08:57 smithi040 ceph-mon[96019]: from='client.? 172.21.15.40:0/3841601539' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2023-07-14T19:08:58.918 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout 2023-07-14T19:08:58.923 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout [global] 2023-07-14T19:08:58.923 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout fsid = adc52f04-2279-11ee-9b33-001a4aab830c 2023-07-14T19:08:58.923 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout mon_host = [v2:172.21.15.40:3300,v1:172.21.15.40:6789] 2023-07-14T19:08:58.924 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2023-07-14T19:08:58.924 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2023-07-14T19:08:58.924 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2023-07-14T19:08:58.924 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2023-07-14T19:08:58.924 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout 2023-07-14T19:08:58.924 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout [mgr] 2023-07-14T19:08:58.925 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2023-07-14T19:08:58.925 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout 2023-07-14T19:08:58.925 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout [osd] 2023-07-14T19:08:58.925 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2023-07-14T19:08:58.925 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2023-07-14T19:08:58.925 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2023-07-14T19:08:58.925 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2023-07-14T19:08:58.926 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2023-07-14T19:08:58.926 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:08:58.433+0000 7f50a5d8c700 1 Processor -- start 2023-07-14T19:08:58.926 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:08:58.433+0000 7f50a5d8c700 1 -- start start 2023-07-14T19:08:58.926 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:08:58.433+0000 7f50a5d8c700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f50a0105430 0x7f50a0105840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:08:58.926 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:08:58.433+0000 7f50a5d8c700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50a0105d80 con 0x7f50a0105430 2023-07-14T19:08:58.926 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:08:58.433+0000 7f509f7fe700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f50a0105430 0x7f50a0105840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:08:58.926 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:08:58.434+0000 7f509f7fe700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f50a0105430 0x7f50a0105840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:59748/0 (socket says 172.21.15.40:59748) 2023-07-14T19:08:58.926 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:08:58.434+0000 7f509f7fe700 1 -- 172.21.15.40:0/1405856268 learned_addr learned my addr 172.21.15.40:0/1405856268 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-07-14T19:08:58.927 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:08:58.434+0000 7f509f7fe700 1 -- 172.21.15.40:0/1405856268 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50a0105ec0 con 0x7f50a0105430 2023-07-14T19:08:58.927 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:08:58.434+0000 7f509f7fe700 1 --2- 172.21.15.40:0/1405856268 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f50a0105430 0x7f50a0105840 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f50900098a0 tx=0x7f5090009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3294a6cc30535226 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:08:58.927 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:08:58.434+0000 7f509e7fc700 1 -- 172.21.15.40:0/1405856268 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5090004160 con 0x7f50a0105430 2023-07-14T19:08:58.927 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:08:58.434+0000 7f509e7fc700 1 -- 172.21.15.40:0/1405856268 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f50900042c0 con 0x7f50a0105430 2023-07-14T19:08:58.927 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:08:58.434+0000 7f509e7fc700 1 -- 172.21.15.40:0/1405856268 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f50900045a0 con 0x7f50a0105430 2023-07-14T19:08:58.927 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:08:58.435+0000 7f50a5d8c700 1 -- 172.21.15.40:0/1405856268 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f50a0105430 msgr2=0x7f50a0105840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:08:58.927 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:08:58.435+0000 7f50a5d8c700 1 --2- 172.21.15.40:0/1405856268 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f50a0105430 0x7f50a0105840 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f50900098a0 tx=0x7f5090009bb0 comp rx=0 tx=0).stop 2023-07-14T19:08:58.928 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:08:58.435+0000 7f50a5d8c700 1 -- 172.21.15.40:0/1405856268 shutdown_connections 2023-07-14T19:08:58.928 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:08:58.435+0000 7f50a5d8c700 1 --2- 172.21.15.40:0/1405856268 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f50a0105430 0x7f50a0105840 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:08:58.928 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:08:58.435+0000 7f50a5d8c700 1 -- 172.21.15.40:0/1405856268 >> 172.21.15.40:0/1405856268 conn(0x7f50a0100a80 msgr2=0x7f50a0102eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:08:58.928 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:08:58.435+0000 7f50a5d8c700 1 -- 172.21.15.40:0/1405856268 shutdown_connections 2023-07-14T19:08:58.928 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:08:58.435+0000 7f50a5d8c700 1 -- 172.21.15.40:0/1405856268 wait complete. 2023-07-14T19:08:58.928 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:08:58.436+0000 7f50a5d8c700 1 Processor -- start 2023-07-14T19:08:58.928 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:08:58.436+0000 7f50a5d8c700 1 -- start start 2023-07-14T19:08:58.929 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:08:58.436+0000 7f50a5d8c700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f50a0105430 0x7f50a01988e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:08:58.929 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:08:58.436+0000 7f50a5d8c700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50a0198e20 con 0x7f50a0105430 2023-07-14T19:08:58.929 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:08:58.436+0000 7f509f7fe700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f50a0105430 0x7f50a01988e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:08:58.929 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:08:58.436+0000 7f509f7fe700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f50a0105430 0x7f50a01988e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:59750/0 (socket says 172.21.15.40:59750) 2023-07-14T19:08:58.929 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:08:58.436+0000 7f509f7fe700 1 -- 172.21.15.40:0/3358708176 learned_addr learned my addr 172.21.15.40:0/3358708176 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-07-14T19:08:58.929 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:08:58.436+0000 7f509f7fe700 1 -- 172.21.15.40:0/3358708176 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5090009550 con 0x7f50a0105430 2023-07-14T19:08:58.929 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:08:58.437+0000 7f509f7fe700 1 --2- 172.21.15.40:0/3358708176 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f50a0105430 0x7f50a01988e0 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f50a0106160 tx=0x7f5090004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:08:58.930 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:08:58.437+0000 7f509cff9700 1 -- 172.21.15.40:0/3358708176 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5090011630 con 0x7f50a0105430 2023-07-14T19:08:58.930 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:08:58.437+0000 7f509cff9700 1 -- 172.21.15.40:0/3358708176 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5090011790 con 0x7f50a0105430 2023-07-14T19:08:58.930 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:08:58.437+0000 7f50a5d8c700 1 -- 172.21.15.40:0/3358708176 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f50a0199020 con 0x7f50a0105430 2023-07-14T19:08:58.930 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:08:58.437+0000 7f509cff9700 1 -- 172.21.15.40:0/3358708176 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5090011a30 con 0x7f50a0105430 2023-07-14T19:08:58.930 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:08:58.437+0000 7f50a5d8c700 1 -- 172.21.15.40:0/3358708176 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f50a0199440 con 0x7f50a0105430 2023-07-14T19:08:58.930 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:08:58.437+0000 7f509cff9700 1 -- 172.21.15.40:0/3358708176 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f5090022070 con 0x7f50a0105430 2023-07-14T19:08:58.930 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:08:58.437+0000 7f509cff9700 1 -- 172.21.15.40:0/3358708176 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f509001c450 con 0x7f50a0105430 2023-07-14T19:08:58.931 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:08:58.438+0000 7f50a5d8c700 1 -- 172.21.15.40:0/3358708176 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f50a0192120 con 0x7f50a0105430 2023-07-14T19:08:58.931 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:08:58.440+0000 7f509cff9700 1 -- 172.21.15.40:0/3358708176 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72446 (secure 0 0 0) 0x7f509001b450 con 0x7f50a0105430 2023-07-14T19:08:58.931 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:08:58.566+0000 7f50a5d8c700 1 -- 172.21.15.40:0/3358708176 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) v1 -- 0x7f50a002cb10 con 0x7f50a0105430 2023-07-14T19:08:58.931 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:08:58.569+0000 7f509cff9700 1 -- 172.21.15.40:0/3358708176 <== mon.0 v2:172.21.15.40:3300/0 7 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f509001b5f0 con 0x7f50a0105430 2023-07-14T19:08:58.931 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:08:58.569+0000 7f509cff9700 1 -- 172.21.15.40:0/3358708176 <== mon.0 v2:172.21.15.40:3300/0 8 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v2) v1 ==== 70+0+429 (secure 0 0 0) 0x7f50900375f0 con 0x7f50a0105430 2023-07-14T19:08:58.931 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:08:58.570+0000 7f50a5d8c700 1 -- 172.21.15.40:0/3358708176 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f50a0105430 msgr2=0x7f50a01988e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:08:58.931 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:08:58.570+0000 7f50a5d8c700 1 --2- 172.21.15.40:0/3358708176 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f50a0105430 0x7f50a01988e0 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f50a0106160 tx=0x7f5090004750 comp rx=0 tx=0).stop 2023-07-14T19:08:58.932 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:08:58.570+0000 7f50a5d8c700 1 -- 172.21.15.40:0/3358708176 shutdown_connections 2023-07-14T19:08:58.932 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:08:58.570+0000 7f50a5d8c700 1 --2- 172.21.15.40:0/3358708176 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f50a0105430 0x7f50a01988e0 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:08:58.932 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:08:58.570+0000 7f50a5d8c700 1 -- 172.21.15.40:0/3358708176 >> 172.21.15.40:0/3358708176 conn(0x7f50a0100a80 msgr2=0x7f50a018ce90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:08:58.932 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:08:58.570+0000 7f50a5d8c700 1 -- 172.21.15.40:0/3358708176 shutdown_connections 2023-07-14T19:08:58.932 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:08:58.570+0000 7f50a5d8c700 1 -- 172.21.15.40:0/3358708176 wait complete. 2023-07-14T19:08:58.932 INFO:teuthology.orchestra.run.smithi040.stdout:Generating new minimal ceph.conf... 2023-07-14T19:08:59.153 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:08:58 smithi040 ceph-mon[96019]: from='client.? 172.21.15.40:0/3358708176' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2023-07-14T19:08:59.154 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:08:58 smithi040 ceph-mon[96019]: from='client.? 172.21.15.40:0/3358708176' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2023-07-14T19:08:59.867 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:08:59.367+0000 7f2ebdb8b700 1 Processor -- start 2023-07-14T19:08:59.867 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:08:59.367+0000 7f2ebdb8b700 1 -- start start 2023-07-14T19:08:59.867 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:08:59.367+0000 7f2ebdb8b700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2eb8105f00 0x7f2eb81082e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:08:59.867 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:08:59.367+0000 7f2ebdb8b700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2eb8079d60 con 0x7f2eb8105f00 2023-07-14T19:08:59.867 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:08:59.367+0000 7f2eb77fe700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2eb8105f00 0x7f2eb81082e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:08:59.868 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:08:59.367+0000 7f2eb77fe700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2eb8105f00 0x7f2eb81082e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:59752/0 (socket says 172.21.15.40:59752) 2023-07-14T19:08:59.868 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:08:59.367+0000 7f2eb77fe700 1 -- 172.21.15.40:0/2206844009 learned_addr learned my addr 172.21.15.40:0/2206844009 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-07-14T19:08:59.868 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:08:59.367+0000 7f2eb77fe700 1 -- 172.21.15.40:0/2206844009 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2eb8079ea0 con 0x7f2eb8105f00 2023-07-14T19:08:59.868 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:08:59.368+0000 7f2eb77fe700 1 --2- 172.21.15.40:0/2206844009 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2eb8105f00 0x7f2eb81082e0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f2ea00098a0 tx=0x7f2ea0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=22acef7004842d3e server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:08:59.868 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:08:59.368+0000 7f2eb67fc700 1 -- 172.21.15.40:0/2206844009 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f2ea0004160 con 0x7f2eb8105f00 2023-07-14T19:08:59.868 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:08:59.368+0000 7f2eb67fc700 1 -- 172.21.15.40:0/2206844009 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2ea00042c0 con 0x7f2eb8105f00 2023-07-14T19:08:59.868 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:08:59.368+0000 7f2eb67fc700 1 -- 172.21.15.40:0/2206844009 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f2ea0011a00 con 0x7f2eb8105f00 2023-07-14T19:08:59.869 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:08:59.368+0000 7f2ebdb8b700 1 -- 172.21.15.40:0/2206844009 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2eb8105f00 msgr2=0x7f2eb81082e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:08:59.869 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:08:59.368+0000 7f2ebdb8b700 1 --2- 172.21.15.40:0/2206844009 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2eb8105f00 0x7f2eb81082e0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f2ea00098a0 tx=0x7f2ea0009bb0 comp rx=0 tx=0).stop 2023-07-14T19:08:59.869 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:08:59.369+0000 7f2ebdb8b700 1 -- 172.21.15.40:0/2206844009 shutdown_connections 2023-07-14T19:08:59.869 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:08:59.369+0000 7f2ebdb8b700 1 --2- 172.21.15.40:0/2206844009 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2eb8105f00 0x7f2eb81082e0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:08:59.869 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:08:59.369+0000 7f2ebdb8b700 1 -- 172.21.15.40:0/2206844009 >> 172.21.15.40:0/2206844009 conn(0x7f2eb80ff9a0 msgr2=0x7f2eb8101db0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:08:59.869 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:08:59.369+0000 7f2ebdb8b700 1 -- 172.21.15.40:0/2206844009 shutdown_connections 2023-07-14T19:08:59.869 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:08:59.369+0000 7f2ebdb8b700 1 -- 172.21.15.40:0/2206844009 wait complete. 2023-07-14T19:08:59.870 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:08:59.369+0000 7f2ebdb8b700 1 Processor -- start 2023-07-14T19:08:59.870 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:08:59.369+0000 7f2ebdb8b700 1 -- start start 2023-07-14T19:08:59.870 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:08:59.370+0000 7f2ebdb8b700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2eb8105f00 0x7f2eb8199100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:08:59.870 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:08:59.370+0000 7f2ebdb8b700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2eb8199640 con 0x7f2eb8105f00 2023-07-14T19:08:59.870 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:08:59.370+0000 7f2eb77fe700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2eb8105f00 0x7f2eb8199100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:08:59.870 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:08:59.370+0000 7f2eb77fe700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2eb8105f00 0x7f2eb8199100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:59754/0 (socket says 172.21.15.40:59754) 2023-07-14T19:08:59.870 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:08:59.370+0000 7f2eb77fe700 1 -- 172.21.15.40:0/307015566 learned_addr learned my addr 172.21.15.40:0/307015566 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-07-14T19:08:59.870 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:08:59.370+0000 7f2eb77fe700 1 -- 172.21.15.40:0/307015566 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ea0009550 con 0x7f2eb8105f00 2023-07-14T19:08:59.871 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:08:59.370+0000 7f2eb77fe700 1 --2- 172.21.15.40:0/307015566 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2eb8105f00 0x7f2eb8199100 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f2ea0009fb0 tx=0x7f2ea0011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:08:59.871 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:08:59.370+0000 7f2eb4ff9700 1 -- 172.21.15.40:0/307015566 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f2ea0003930 con 0x7f2eb8105f00 2023-07-14T19:08:59.871 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:08:59.370+0000 7f2eb4ff9700 1 -- 172.21.15.40:0/307015566 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2ea0003a90 con 0x7f2eb8105f00 2023-07-14T19:08:59.871 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:08:59.370+0000 7f2eb4ff9700 1 -- 172.21.15.40:0/307015566 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f2ea00195e0 con 0x7f2eb8105f00 2023-07-14T19:08:59.871 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:08:59.370+0000 7f2ebdb8b700 1 -- 172.21.15.40:0/307015566 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2eb8199840 con 0x7f2eb8105f00 2023-07-14T19:08:59.871 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:08:59.370+0000 7f2ebdb8b700 1 -- 172.21.15.40:0/307015566 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2eb8199c60 con 0x7f2eb8105f00 2023-07-14T19:08:59.871 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:08:59.371+0000 7f2eb4ff9700 1 -- 172.21.15.40:0/307015566 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f2ea0019b50 con 0x7f2eb8105f00 2023-07-14T19:08:59.872 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:08:59.371+0000 7f2eb4ff9700 1 -- 172.21.15.40:0/307015566 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f2ea00243f0 con 0x7f2eb8105f00 2023-07-14T19:08:59.872 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:08:59.372+0000 7f2ebdb8b700 1 -- 172.21.15.40:0/307015566 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2eb8061c70 con 0x7f2eb8105f00 2023-07-14T19:08:59.872 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:08:59.374+0000 7f2eb4ff9700 1 -- 172.21.15.40:0/307015566 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72446 (secure 0 0 0) 0x7f2ea002de30 con 0x7f2eb8105f00 2023-07-14T19:08:59.872 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:08:59.501+0000 7f2ebdb8b700 1 -- 172.21.15.40:0/307015566 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) v1 -- 0x7f2eb819c790 con 0x7f2eb8105f00 2023-07-14T19:08:59.873 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:08:59.501+0000 7f2eb4ff9700 1 -- 172.21.15.40:0/307015566 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v2) v1 ==== 76+0+175 (secure 0 0 0) 0x7f2ea0003c00 con 0x7f2eb8105f00 2023-07-14T19:08:59.873 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:08:59.502+0000 7f2ebdb8b700 1 -- 172.21.15.40:0/307015566 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2eb8105f00 msgr2=0x7f2eb8199100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:08:59.873 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:08:59.502+0000 7f2ebdb8b700 1 --2- 172.21.15.40:0/307015566 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2eb8105f00 0x7f2eb8199100 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f2ea0009fb0 tx=0x7f2ea0011ad0 comp rx=0 tx=0).stop 2023-07-14T19:08:59.873 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:08:59.502+0000 7f2ebdb8b700 1 -- 172.21.15.40:0/307015566 shutdown_connections 2023-07-14T19:08:59.874 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:08:59.502+0000 7f2ebdb8b700 1 --2- 172.21.15.40:0/307015566 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2eb8105f00 0x7f2eb8199100 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:08:59.874 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:08:59.502+0000 7f2ebdb8b700 1 -- 172.21.15.40:0/307015566 >> 172.21.15.40:0/307015566 conn(0x7f2eb80ff9a0 msgr2=0x7f2eb8100780 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:08:59.874 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:08:59.502+0000 7f2ebdb8b700 1 -- 172.21.15.40:0/307015566 shutdown_connections 2023-07-14T19:08:59.874 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:08:59.502+0000 7f2ebdb8b700 1 -- 172.21.15.40:0/307015566 wait complete. 2023-07-14T19:08:59.874 INFO:teuthology.orchestra.run.smithi040.stdout:Restarting the monitor... 2023-07-14T19:09:00.215 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:08:59 smithi040 systemd[1]: Stopping Ceph mon.a for adc52f04-2279-11ee-9b33-001a4aab830c... 2023-07-14T19:09:00.215 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:08:59 smithi040 ceph-mon[96019]: from='client.? 172.21.15.40:0/307015566' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:09:00.472 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:09:00 smithi040 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-mon-a[95994]: 2023-07-14T19:09:00.212+0000 7f1ec2280700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2023-07-14T19:09:00.472 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:09:00 smithi040 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-mon-a[95994]: 2023-07-14T19:09:00.212+0000 7f1ec2280700 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2023-07-14T19:09:00.756 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:09:00 smithi040 bash[96357]: ceph-adc52f04-2279-11ee-9b33-001a4aab830c-mon-a 2023-07-14T19:09:01.222 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:09:00 smithi040 systemd[1]: ceph-adc52f04-2279-11ee-9b33-001a4aab830c@mon.a.service: Succeeded. 2023-07-14T19:09:01.222 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:09:00 smithi040 systemd[1]: Stopped Ceph mon.a for adc52f04-2279-11ee-9b33-001a4aab830c. 2023-07-14T19:09:01.223 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:09:00 smithi040 systemd[1]: Starting Ceph mon.a for adc52f04-2279-11ee-9b33-001a4aab830c... 2023-07-14T19:09:01.534 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:09:01 smithi040 podman[96477]: 2023-07-14T19:09:01.535 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:09:01 smithi040 ceph-mon[96518]: set uid:gid to 167:167 (ceph:ceph) 2023-07-14T19:09:01.535 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:09:01 smithi040 ceph-mon[96518]: ceph version 18.0.0-4940-g81c9387d (81c9387d2e7576da065747ec9c16d1c7d69016ec) reef (dev), process ceph-mon, pid 7 2023-07-14T19:09:01.535 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:09:01 smithi040 ceph-mon[96518]: pidfile_write: ignore empty --pid-file 2023-07-14T19:09:01.535 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:09:01 smithi040 ceph-mon[96518]: load: jerasure load: lrc 2023-07-14T19:09:01.535 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:09:01 smithi040 ceph-mon[96518]: rocksdb: RocksDB version: 7.9.2 2023-07-14T19:09:01.536 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:09:01 smithi040 ceph-mon[96518]: rocksdb: Git sha 0 2023-07-14T19:09:01.536 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:09:01 smithi040 ceph-mon[96518]: rocksdb: Compile date 2023-07-14 14:45:42 2023-07-14T19:09:01.536 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:09:01 smithi040 ceph-mon[96518]: rocksdb: DB SUMMARY 2023-07-14T19:09:01.536 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:09:01 smithi040 ceph-mon[96518]: rocksdb: DB Session ID: 12YC7MVKHO7GI56LLYLY 2023-07-14T19:09:01.536 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:09:01 smithi040 ceph-mon[96518]: rocksdb: CURRENT file: CURRENT 2023-07-14T19:09:01.536 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:09:01 smithi040 ceph-mon[96518]: rocksdb: IDENTITY file: IDENTITY 2023-07-14T19:09:01.536 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:09:01 smithi040 ceph-mon[96518]: rocksdb: MANIFEST file: MANIFEST-000010 size: 179 Bytes 2023-07-14T19:09:01.537 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:09:01 smithi040 ceph-mon[96518]: rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000008.sst 2023-07-14T19:09:01.537 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:09:01 smithi040 ceph-mon[96518]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000009.log size: 98689 ; 2023-07-14T19:09:01.537 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:09:01 smithi040 ceph-mon[96518]: rocksdb: Options.error_if_exists: 0 2023-07-14T19:09:01.537 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:09:01 smithi040 ceph-mon[96518]: rocksdb: Options.create_if_missing: 0 2023-07-14T19:09:01.537 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:09:01 smithi040 ceph-mon[96518]: rocksdb: Options.paranoid_checks: 1 2023-07-14T19:09:01.537 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:09:01 smithi040 ceph-mon[96518]: rocksdb: Options.flush_verify_memtable_count: 1 2023-07-14T19:09:01.537 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:09:01 smithi040 ceph-mon[96518]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2023-07-14T19:09:01.538 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:09:01 smithi040 ceph-mon[96518]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2023-07-14T19:09:01.538 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:09:01 smithi040 ceph-mon[96518]: rocksdb: Options.env: 0x55f535761c60 2023-07-14T19:09:01.538 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:09:01 smithi040 ceph-mon[96518]: rocksdb: Options.fs: PosixFileSystem 2023-07-14T19:09:01.538 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:09:01 smithi040 ceph-mon[96518]: rocksdb: Options.info_log: 0x55f536d012a0 2023-07-14T19:09:01.538 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:09:01 smithi040 ceph-mon[96518]: rocksdb: Options.max_file_opening_threads: 16 2023-07-14T19:09:01.538 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:09:01 smithi040 ceph-mon[96518]: rocksdb: Options.statistics: (nil) 2023-07-14T19:09:01.539 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:09:01 smithi040 ceph-mon[96518]: rocksdb: Options.use_fsync: 0 2023-07-14T19:09:01.539 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:09:01 smithi040 ceph-mon[96518]: rocksdb: Options.max_log_file_size: 0 2023-07-14T19:09:01.539 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:09:01 smithi040 ceph-mon[96518]: rocksdb: Options.max_manifest_file_size: 1073741824 2023-07-14T19:09:01.539 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:09:01 smithi040 ceph-mon[96518]: rocksdb: Options.log_file_time_to_roll: 0 2023-07-14T19:09:01.539 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:09:01 smithi040 ceph-mon[96518]: rocksdb: Options.keep_log_file_num: 1000 2023-07-14T19:09:01.539 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:09:01 smithi040 ceph-mon[96518]: rocksdb: Options.recycle_log_file_num: 0 2023-07-14T19:09:01.539 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:09:01 smithi040 ceph-mon[96518]: rocksdb: Options.allow_fallocate: 1 2023-07-14T19:09:01.540 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:09:01 smithi040 ceph-mon[96518]: rocksdb: Options.allow_mmap_reads: 0 2023-07-14T19:09:01.540 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:09:01 smithi040 ceph-mon[96518]: rocksdb: Options.allow_mmap_writes: 0 2023-07-14T19:09:01.540 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:09:01 smithi040 ceph-mon[96518]: rocksdb: Options.use_direct_reads: 0 2023-07-14T19:09:01.540 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:09:01 smithi040 ceph-mon[96518]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2023-07-14T19:09:01.540 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:09:01 smithi040 ceph-mon[96518]: rocksdb: Options.create_missing_column_families: 0 2023-07-14T19:09:01.540 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:09:01 smithi040 ceph-mon[96518]: rocksdb: Options.db_log_dir: 2023-07-14T19:09:01.540 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:09:01 smithi040 ceph-mon[96518]: rocksdb: Options.wal_dir: 2023-07-14T19:09:01.542 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:09:01 smithi040 ceph-mon[96518]: rocksdb: Options.table_cache_numshardbits: 6 2023-07-14T19:09:01.542 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:09:01 smithi040 ceph-mon[96518]: rocksdb: Options.WAL_ttl_seconds: 0 2023-07-14T19:09:01.542 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:09:01 smithi040 ceph-mon[96518]: rocksdb: Options.WAL_size_limit_MB: 0 2023-07-14T19:09:01.542 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:09:01 smithi040 ceph-mon[96518]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2023-07-14T19:09:01.542 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:09:01 smithi040 ceph-mon[96518]: rocksdb: Options.manifest_preallocation_size: 4194304 2023-07-14T19:09:01.542 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:09:01 smithi040 ceph-mon[96518]: rocksdb: Options.is_fd_close_on_exec: 1 2023-07-14T19:09:01.543 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:09:01 smithi040 ceph-mon[96518]: rocksdb: Options.advise_random_on_open: 1 2023-07-14T19:09:01.543 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:09:01 smithi040 ceph-mon[96518]: rocksdb: Options.db_write_buffer_size: 0 2023-07-14T19:09:01.543 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:09:01 smithi040 ceph-mon[96518]: rocksdb: Options.write_buffer_manager: 0x55f536c223c0 2023-07-14T19:09:01.543 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:09:01 smithi040 ceph-mon[96518]: rocksdb: Options.access_hint_on_compaction_start: 1 2023-07-14T19:09:01.543 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:09:01 smithi040 ceph-mon[96518]: rocksdb: Options.random_access_max_buffer_size: 1048576 2023-07-14T19:09:01.543 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:09:01 smithi040 ceph-mon[96518]: rocksdb: Options.use_adaptive_mutex: 0 2023-07-14T19:09:01.543 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:09:01 smithi040 ceph-mon[96518]: rocksdb: Options.rate_limiter: (nil) 2023-07-14T19:09:01.543 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:09:01 smithi040 ceph-mon[96518]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2023-07-14T19:09:01.544 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:09:01 smithi040 ceph-mon[96518]: rocksdb: Options.wal_recovery_mode: 2 2023-07-14T19:09:01.544 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:09:01 smithi040 ceph-mon[96518]: rocksdb: Options.enable_thread_tracking: 0 2023-07-14T19:09:01.544 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:09:01 smithi040 ceph-mon[96518]: rocksdb: Options.enable_pipelined_write: 0 2023-07-14T19:09:01.544 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:09:01 smithi040 ceph-mon[96518]: rocksdb: Options.unordered_write: 0 2023-07-14T19:09:01.544 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:09:01 smithi040 ceph-mon[96518]: rocksdb: Options.allow_concurrent_memtable_write: 1 2023-07-14T19:09:01.544 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:09:01 smithi040 ceph-mon[96518]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2023-07-14T19:09:01.544 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:09:01 smithi040 ceph-mon[96518]: rocksdb: Options.write_thread_max_yield_usec: 100 2023-07-14T19:09:01.545 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:09:01 smithi040 ceph-mon[96518]: rocksdb: Options.write_thread_slow_yield_usec: 3 2023-07-14T19:09:01.545 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:09:01 smithi040 ceph-mon[96518]: rocksdb: Options.row_cache: None 2023-07-14T19:09:01.545 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:09:01 smithi040 ceph-mon[96518]: rocksdb: Options.wal_filter: None 2023-07-14T19:09:01.545 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:09:01 smithi040 ceph-mon[96518]: rocksdb: Options.avoid_flush_during_recovery: 0 2023-07-14T19:09:01.545 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:09:01 smithi040 ceph-mon[96518]: rocksdb: Options.allow_ingest_behind: 0 2023-07-14T19:09:01.545 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:09:01 smithi040 ceph-mon[96518]: rocksdb: Options.two_write_queues: 0 2023-07-14T19:09:01.545 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:09:01 smithi040 ceph-mon[96518]: rocksdb: Options.manual_wal_flush: 0 2023-07-14T19:09:01.545 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:09:01 smithi040 ceph-mon[96518]: rocksdb: Options.wal_compression: 0 2023-07-14T19:09:01.546 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:09:01 smithi040 ceph-mon[96518]: rocksdb: Options.atomic_flush: 0 2023-07-14T19:09:01.546 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:09:01 smithi040 ceph-mon[96518]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2023-07-14T19:09:01.546 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:09:01 smithi040 ceph-mon[96518]: rocksdb: Options.persist_stats_to_disk: 0 2023-07-14T19:09:01.546 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:09:01 smithi040 ceph-mon[96518]: rocksdb: Options.write_dbid_to_manifest: 0 2023-07-14T19:09:01.546 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:09:01 smithi040 ceph-mon[96518]: rocksdb: Options.log_readahead_size: 0 2023-07-14T19:09:01.546 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:09:01 smithi040 ceph-mon[96518]: rocksdb: Options.file_checksum_gen_factory: Unknown 2023-07-14T19:09:01.546 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:09:01 smithi040 ceph-mon[96518]: rocksdb: Options.best_efforts_recovery: 0 2023-07-14T19:09:01.546 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:09:01 smithi040 ceph-mon[96518]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2023-07-14T19:09:01.547 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:09:01 smithi040 ceph-mon[96518]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2023-07-14T19:09:01.547 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:09:01 smithi040 ceph-mon[96518]: rocksdb: Options.allow_data_in_errors: 0 2023-07-14T19:09:01.547 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:09:01 smithi040 ceph-mon[96518]: rocksdb: Options.db_host_id: __hostname__ 2023-07-14T19:09:01.547 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:09:01 smithi040 ceph-mon[96518]: rocksdb: Options.enforce_single_del_contracts: true 2023-07-14T19:09:01.547 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:09:01 smithi040 ceph-mon[96518]: rocksdb: Options.max_background_jobs: 2 2023-07-14T19:09:01.547 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:09:01 smithi040 ceph-mon[96518]: rocksdb: Options.max_background_compactions: -1 2023-07-14T19:09:01.547 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:09:01 smithi040 ceph-mon[96518]: rocksdb: Options.max_subcompactions: 1 2023-07-14T19:09:01.548 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:09:01 smithi040 ceph-mon[96518]: rocksdb: Options.avoid_flush_during_shutdown: 0 2023-07-14T19:09:01.548 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:09:01 smithi040 ceph-mon[96518]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2023-07-14T19:09:01.548 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:09:01 smithi040 ceph-mon[96518]: rocksdb: Options.delayed_write_rate : 16777216 2023-07-14T19:09:01.548 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:09:01 smithi040 ceph-mon[96518]: rocksdb: Options.max_total_wal_size: 0 2023-07-14T19:09:01.548 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:09:01 smithi040 ceph-mon[96518]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2023-07-14T19:09:01.548 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:09:01 smithi040 ceph-mon[96518]: rocksdb: Options.stats_dump_period_sec: 600 2023-07-14T19:09:01.548 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:09:01 smithi040 ceph-mon[96518]: rocksdb: Options.stats_persist_period_sec: 600 2023-07-14T19:09:01.549 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:09:01 smithi040 ceph-mon[96518]: rocksdb: Options.stats_history_buffer_size: 1048576 2023-07-14T19:09:01.549 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:09:01 smithi040 ceph-mon[96518]: rocksdb: Options.max_open_files: -1 2023-07-14T19:09:01.549 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:09:01 smithi040 ceph-mon[96518]: rocksdb: Options.bytes_per_sync: 0 2023-07-14T19:09:01.549 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:09:01 smithi040 ceph-mon[96518]: rocksdb: Options.wal_bytes_per_sync: 0 2023-07-14T19:09:01.549 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:09:01 smithi040 ceph-mon[96518]: rocksdb: Options.strict_bytes_per_sync: 0 2023-07-14T19:09:01.549 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:09:01 smithi040 ceph-mon[96518]: rocksdb: Options.compaction_readahead_size: 0 2023-07-14T19:09:01.549 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:09:01 smithi040 ceph-mon[96518]: rocksdb: Options.max_background_flushes: -1 2023-07-14T19:09:01.549 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:09:01 smithi040 ceph-mon[96518]: rocksdb: Compression algorithms supported: 2023-07-14T19:09:01.550 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:09:01 smithi040 ceph-mon[96518]: rocksdb: kZSTDNotFinalCompression supported: 0 2023-07-14T19:09:01.550 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:09:01 smithi040 bash[96477]: 84c8a307e84fb6dfba8d7d5ecfed7a72e997fa768d5f6bf213d7788b79529642 2023-07-14T19:09:01.550 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:09:01 smithi040 ceph-mon[96518]: rocksdb: kZSTD supported: 0 2023-07-14T19:09:01.550 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:09:01 smithi040 ceph-mon[96518]: rocksdb: kXpressCompression supported: 0 2023-07-14T19:09:01.550 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:09:01 smithi040 ceph-mon[96518]: rocksdb: kLZ4HCCompression supported: 1 2023-07-14T19:09:01.550 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:09:01 smithi040 ceph-mon[96518]: rocksdb: kZlibCompression supported: 1 2023-07-14T19:09:01.550 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:09:01 smithi040 ceph-mon[96518]: rocksdb: kSnappyCompression supported: 1 2023-07-14T19:09:01.550 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:09:01 smithi040 ceph-mon[96518]: rocksdb: kLZ4Compression supported: 1 2023-07-14T19:09:01.551 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:09:01 smithi040 ceph-mon[96518]: rocksdb: kBZip2Compression supported: 0 2023-07-14T19:09:01.551 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:09:01 smithi040 ceph-mon[96518]: rocksdb: Fast CRC32 supported: Supported on x86 2023-07-14T19:09:01.551 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:09:01 smithi040 ceph-mon[96518]: rocksdb: DMutex implementation: pthread_mutex_t 2023-07-14T19:09:01.551 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:09:01 smithi040 ceph-mon[96518]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 2023-07-14T19:09:01.551 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:09:01 smithi040 ceph-mon[96518]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2023-07-14T19:09:01.551 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:09:01 smithi040 ceph-mon[96518]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2023-07-14T19:09:01.551 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:09:01 smithi040 ceph-mon[96518]: rocksdb: Options.merge_operator: 2023-07-14T19:09:01.552 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:09:01 smithi040 ceph-mon[96518]: rocksdb: Options.compaction_filter: None 2023-07-14T19:09:01.552 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:09:01 smithi040 ceph-mon[96518]: rocksdb: Options.compaction_filter_factory: None 2023-07-14T19:09:01.552 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:09:01 smithi040 ceph-mon[96518]: rocksdb: Options.sst_partitioner_factory: None 2023-07-14T19:09:01.552 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:09:01 smithi040 ceph-mon[96518]: rocksdb: Options.memtable_factory: SkipListFactory 2023-07-14T19:09:01.552 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:09:01 smithi040 ceph-mon[96518]: rocksdb: Options.table_factory: BlockBasedTable 2023-07-14T19:09:01.552 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:09:01 smithi040 ceph-mon[96518]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55f536d013a0) 2023-07-14T19:09:01.552 INFO:journalctl@ceph.mon.a.smithi040.stdout: cache_index_and_filter_blocks: 1 2023-07-14T19:09:01.553 INFO:journalctl@ceph.mon.a.smithi040.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2023-07-14T19:09:01.553 INFO:journalctl@ceph.mon.a.smithi040.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2023-07-14T19:09:01.553 INFO:journalctl@ceph.mon.a.smithi040.stdout: pin_top_level_index_and_filter: 1 2023-07-14T19:09:01.553 INFO:journalctl@ceph.mon.a.smithi040.stdout: index_type: 0 2023-07-14T19:09:01.553 INFO:journalctl@ceph.mon.a.smithi040.stdout: data_block_index_type: 0 2023-07-14T19:09:01.553 INFO:journalctl@ceph.mon.a.smithi040.stdout: index_shortening: 1 2023-07-14T19:09:01.553 INFO:journalctl@ceph.mon.a.smithi040.stdout: data_block_hash_table_util_ratio: 0.750000 2023-07-14T19:09:01.553 INFO:journalctl@ceph.mon.a.smithi040.stdout: checksum: 4 2023-07-14T19:09:01.554 INFO:journalctl@ceph.mon.a.smithi040.stdout: no_block_cache: 0 2023-07-14T19:09:01.554 INFO:journalctl@ceph.mon.a.smithi040.stdout: block_cache: 0x55f536cc11f0 2023-07-14T19:09:01.554 INFO:journalctl@ceph.mon.a.smithi040.stdout: block_cache_name: BinnedLRUCache 2023-07-14T19:09:01.554 INFO:journalctl@ceph.mon.a.smithi040.stdout: block_cache_options: 2023-07-14T19:09:01.554 INFO:journalctl@ceph.mon.a.smithi040.stdout: capacity : 536870912 2023-07-14T19:09:01.554 INFO:journalctl@ceph.mon.a.smithi040.stdout: num_shard_bits : 4 2023-07-14T19:09:01.554 INFO:journalctl@ceph.mon.a.smithi040.stdout: strict_capacity_limit : 0 2023-07-14T19:09:01.554 INFO:journalctl@ceph.mon.a.smithi040.stdout: high_pri_pool_ratio: 0.000 2023-07-14T19:09:01.555 INFO:journalctl@ceph.mon.a.smithi040.stdout: block_cache_compressed: (nil) 2023-07-14T19:09:01.555 INFO:journalctl@ceph.mon.a.smithi040.stdout: persistent_cache: (nil) 2023-07-14T19:09:01.555 INFO:journalctl@ceph.mon.a.smithi040.stdout: block_size: 4096 2023-07-14T19:09:01.555 INFO:journalctl@ceph.mon.a.smithi040.stdout: block_size_deviation: 10 2023-07-14T19:09:01.555 INFO:journalctl@ceph.mon.a.smithi040.stdout: block_restart_interval: 16 2023-07-14T19:09:01.555 INFO:journalctl@ceph.mon.a.smithi040.stdout: index_block_restart_interval: 1 2023-07-14T19:09:01.555 INFO:journalctl@ceph.mon.a.smithi040.stdout: metadata_block_size: 4096 2023-07-14T19:09:01.555 INFO:journalctl@ceph.mon.a.smithi040.stdout: partition_filters: 0 2023-07-14T19:09:01.556 INFO:journalctl@ceph.mon.a.smithi040.stdout: use_delta_encoding: 1 2023-07-14T19:09:01.556 INFO:journalctl@ceph.mon.a.smithi040.stdout: filter_policy: bloomfilter 2023-07-14T19:09:01.556 INFO:journalctl@ceph.mon.a.smithi040.stdout: whole_key_filtering: 1 2023-07-14T19:09:01.556 INFO:journalctl@ceph.mon.a.smithi040.stdout: verify_compression: 0 2023-07-14T19:09:01.556 INFO:journalctl@ceph.mon.a.smithi040.stdout: read_amp_bytes_per_bit: 0 2023-07-14T19:09:01.556 INFO:journalctl@ceph.mon.a.smithi040.stdout: format_version: 5 2023-07-14T19:09:01.556 INFO:journalctl@ceph.mon.a.smithi040.stdout: enable_index_compression: 1 2023-07-14T19:09:01.557 INFO:journalctl@ceph.mon.a.smithi040.stdout: block_align: 0 2023-07-14T19:09:01.557 INFO:journalctl@ceph.mon.a.smithi040.stdout: max_auto_readahead_size: 262144 2023-07-14T19:09:01.557 INFO:journalctl@ceph.mon.a.smithi040.stdout: prepopulate_block_cache: 0 2023-07-14T19:09:01.557 INFO:journalctl@ceph.mon.a.smithi040.stdout: initial_auto_readahead_size: 8192 2023-07-14T19:09:01.557 INFO:journalctl@ceph.mon.a.smithi040.stdout: num_file_reads_for_auto_readahead: 2 2023-07-14T19:09:01.557 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:09:01 smithi040 ceph-mon[96518]: rocksdb: Options.write_buffer_size: 33554432 2023-07-14T19:09:01.557 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:09:01 smithi040 ceph-mon[96518]: rocksdb: Options.max_write_buffer_number: 2 2023-07-14T19:09:01.557 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:09:01 smithi040 ceph-mon[96518]: rocksdb: Options.compression: NoCompression 2023-07-14T19:09:01.558 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:09:01 smithi040 ceph-mon[96518]: rocksdb: Options.bottommost_compression: Disabled 2023-07-14T19:09:01.558 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:09:01 smithi040 ceph-mon[96518]: rocksdb: Options.prefix_extractor: nullptr 2023-07-14T19:09:01.558 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:09:01 smithi040 ceph-mon[96518]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2023-07-14T19:09:01.558 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:09:01 smithi040 ceph-mon[96518]: rocksdb: Options.num_levels: 7 2023-07-14T19:09:01.558 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:09:01 smithi040 ceph-mon[96518]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2023-07-14T19:09:01.558 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:09:01 smithi040 ceph-mon[96518]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2023-07-14T19:09:01.558 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:09:01 smithi040 ceph-mon[96518]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2023-07-14T19:09:01.559 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:09:01 smithi040 ceph-mon[96518]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2023-07-14T19:09:01.559 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:09:01 smithi040 ceph-mon[96518]: rocksdb: Options.bottommost_compression_opts.level: 32767 2023-07-14T19:09:01.559 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:09:01 smithi040 ceph-mon[96518]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2023-07-14T19:09:01.559 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:09:01 smithi040 ceph-mon[96518]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2023-07-14T19:09:01.559 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:09:01 smithi040 ceph-mon[96518]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2023-07-14T19:09:01.559 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:09:01 smithi040 ceph-mon[96518]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2023-07-14T19:09:01.559 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:09:01 smithi040 ceph-mon[96518]: rocksdb: Options.bottommost_compression_opts.enabled: false 2023-07-14T19:09:01.559 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:09:01 smithi040 ceph-mon[96518]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2023-07-14T19:09:01.560 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:09:01 smithi040 ceph-mon[96518]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2023-07-14T19:09:01.560 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:09:01 smithi040 ceph-mon[96518]: rocksdb: Options.compression_opts.window_bits: -14 2023-07-14T19:09:01.560 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:09:01 smithi040 ceph-mon[96518]: rocksdb: Options.compression_opts.level: 32767 2023-07-14T19:09:01.560 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:09:01 smithi040 ceph-mon[96518]: rocksdb: Options.compression_opts.strategy: 0 2023-07-14T19:09:01.560 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:09:01 smithi040 ceph-mon[96518]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2023-07-14T19:09:01.560 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:09:01 smithi040 ceph-mon[96518]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2023-07-14T19:09:01.560 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:09:01 smithi040 ceph-mon[96518]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2023-07-14T19:09:01.560 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:09:01 smithi040 ceph-mon[96518]: rocksdb: Options.compression_opts.parallel_threads: 1 2023-07-14T19:09:01.561 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:09:01 smithi040 ceph-mon[96518]: rocksdb: Options.compression_opts.enabled: false 2023-07-14T19:09:01.561 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:09:01 smithi040 ceph-mon[96518]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2023-07-14T19:09:01.561 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:09:01 smithi040 ceph-mon[96518]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2023-07-14T19:09:01.561 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:09:01 smithi040 ceph-mon[96518]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2023-07-14T19:09:01.561 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:09:01 smithi040 ceph-mon[96518]: rocksdb: Options.level0_stop_writes_trigger: 36 2023-07-14T19:09:01.561 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:09:01 smithi040 ceph-mon[96518]: rocksdb: Options.target_file_size_base: 67108864 2023-07-14T19:09:01.561 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:09:01 smithi040 ceph-mon[96518]: rocksdb: Options.target_file_size_multiplier: 1 2023-07-14T19:09:01.562 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:09:01 smithi040 ceph-mon[96518]: rocksdb: Options.max_bytes_for_level_base: 268435456 2023-07-14T19:09:01.562 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:09:01 smithi040 ceph-mon[96518]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2023-07-14T19:09:01.562 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:09:01 smithi040 ceph-mon[96518]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2023-07-14T19:09:01.562 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:09:01 smithi040 ceph-mon[96518]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2023-07-14T19:09:01.562 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:09:01 smithi040 ceph-mon[96518]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2023-07-14T19:09:01.562 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:09:01 smithi040 ceph-mon[96518]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2023-07-14T19:09:01.562 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:09:01 smithi040 ceph-mon[96518]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2023-07-14T19:09:01.562 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:09:01 smithi040 ceph-mon[96518]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2023-07-14T19:09:01.563 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:09:01 smithi040 ceph-mon[96518]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2023-07-14T19:09:01.563 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:09:01 smithi040 ceph-mon[96518]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2023-07-14T19:09:01.563 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:09:01 smithi040 ceph-mon[96518]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2023-07-14T19:09:01.563 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:09:01 smithi040 ceph-mon[96518]: rocksdb: Options.max_compaction_bytes: 1677721600 2023-07-14T19:09:01.563 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:09:01 smithi040 ceph-mon[96518]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2023-07-14T19:09:01.563 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:09:01 smithi040 ceph-mon[96518]: rocksdb: Options.arena_block_size: 1048576 2023-07-14T19:09:01.563 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:09:01 smithi040 ceph-mon[96518]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2023-07-14T19:09:01.563 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:09:01 smithi040 ceph-mon[96518]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2023-07-14T19:09:01.564 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:09:01 smithi040 ceph-mon[96518]: rocksdb: Options.disable_auto_compactions: 0 2023-07-14T19:09:01.564 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:09:01 smithi040 ceph-mon[96518]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2023-07-14T19:09:01.564 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:09:01 smithi040 ceph-mon[96518]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2023-07-14T19:09:01.564 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:09:01 smithi040 ceph-mon[96518]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2023-07-14T19:09:01.564 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:09:01 smithi040 ceph-mon[96518]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2023-07-14T19:09:01.564 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:09:01 smithi040 ceph-mon[96518]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2023-07-14T19:09:01.564 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:09:01 smithi040 ceph-mon[96518]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2023-07-14T19:09:01.565 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:09:01 smithi040 ceph-mon[96518]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2023-07-14T19:09:01.565 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:09:01 smithi040 ceph-mon[96518]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2023-07-14T19:09:01.565 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:09:01 smithi040 ceph-mon[96518]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2023-07-14T19:09:01.565 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:09:01 smithi040 ceph-mon[96518]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2023-07-14T19:09:01.565 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:09:01 smithi040 ceph-mon[96518]: rocksdb: Options.table_properties_collectors: 2023-07-14T19:09:01.565 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:09:01 smithi040 ceph-mon[96518]: rocksdb: Options.inplace_update_support: 0 2023-07-14T19:09:01.565 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:09:01 smithi040 ceph-mon[96518]: rocksdb: Options.inplace_update_num_locks: 10000 2023-07-14T19:09:01.565 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:09:01 smithi040 ceph-mon[96518]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2023-07-14T19:09:01.566 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:09:01 smithi040 ceph-mon[96518]: rocksdb: Options.memtable_whole_key_filtering: 0 2023-07-14T19:09:01.566 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:09:01 smithi040 ceph-mon[96518]: rocksdb: Options.memtable_huge_page_size: 0 2023-07-14T19:09:01.566 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:09:01 smithi040 ceph-mon[96518]: rocksdb: Options.bloom_locality: 0 2023-07-14T19:09:01.566 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:09:01 smithi040 ceph-mon[96518]: rocksdb: Options.max_successive_merges: 0 2023-07-14T19:09:01.566 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:09:01 smithi040 ceph-mon[96518]: rocksdb: Options.optimize_filters_for_hits: 0 2023-07-14T19:09:01.566 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:09:01 smithi040 ceph-mon[96518]: rocksdb: Options.paranoid_file_checks: 0 2023-07-14T19:09:01.566 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:09:01 smithi040 ceph-mon[96518]: rocksdb: Options.force_consistency_checks: 1 2023-07-14T19:09:01.566 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:09:01 smithi040 ceph-mon[96518]: rocksdb: Options.report_bg_io_stats: 0 2023-07-14T19:09:01.567 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:09:01 smithi040 ceph-mon[96518]: rocksdb: Options.ttl: 2592000 2023-07-14T19:09:01.567 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:09:01 smithi040 ceph-mon[96518]: rocksdb: Options.periodic_compaction_seconds: 0 2023-07-14T19:09:01.567 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:09:01 smithi040 ceph-mon[96518]: rocksdb: Options.preclude_last_level_data_seconds: 0 2023-07-14T19:09:01.567 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:09:01 smithi040 ceph-mon[96518]: rocksdb: Options.preserve_internal_time_seconds: 0 2023-07-14T19:09:01.567 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:09:01 smithi040 ceph-mon[96518]: rocksdb: Options.enable_blob_files: false 2023-07-14T19:09:01.567 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:09:01 smithi040 ceph-mon[96518]: rocksdb: Options.min_blob_size: 0 2023-07-14T19:09:01.567 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:09:01 smithi040 ceph-mon[96518]: rocksdb: Options.blob_file_size: 268435456 2023-07-14T19:09:01.567 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:09:01 smithi040 ceph-mon[96518]: rocksdb: Options.blob_compression_type: NoCompression 2023-07-14T19:09:01.568 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:09:01 smithi040 ceph-mon[96518]: rocksdb: Options.enable_blob_garbage_collection: false 2023-07-14T19:09:01.568 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:09:01 smithi040 ceph-mon[96518]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2023-07-14T19:09:01.568 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:09:01 smithi040 ceph-mon[96518]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2023-07-14T19:09:01.568 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:09:01 smithi040 ceph-mon[96518]: rocksdb: Options.blob_compaction_readahead_size: 0 2023-07-14T19:09:01.568 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:09:01 smithi040 ceph-mon[96518]: rocksdb: Options.blob_file_starting_level: 0 2023-07-14T19:09:01.568 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:09:01 smithi040 ceph-mon[96518]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2023-07-14T19:09:01.568 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:09:01 smithi040 ceph-mon[96518]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 succeeded,manifest_file_number is 10, next_file_number is 12, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 5 2023-07-14T19:09:01.569 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:09:01 smithi040 ceph-mon[96518]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 5 2023-07-14T19:09:01.569 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:09:01 smithi040 ceph-mon[96518]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 82dc0a6b-7679-4a55-9fda-68131e90eec2 2023-07-14T19:09:01.569 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:09:01 smithi040 ceph-mon[96518]: rocksdb: EVENT_LOG_v1 {"time_micros": 1689361741458692, "job": 1, "event": "recovery_started", "wal_files": [9]} 2023-07-14T19:09:01.569 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:09:01 smithi040 ceph-mon[96518]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #9 mode 2 2023-07-14T19:09:01.569 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:09:01 smithi040 ceph-mon[96518]: rocksdb: EVENT_LOG_v1 {"time_micros": 1689361741459754, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 13, "file_size": 93930, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 8, "largest_seqno": 314, "table_properties": {"data_size": 91935, "index_size": 277, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 709, "raw_key_size": 13640, "raw_average_key_size": 49, "raw_value_size": 84500, "raw_average_value_size": 309, "num_data_blocks": 11, "num_entries": 273, "num_filter_entries": 273, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1689361741, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "82dc0a6b-7679-4a55-9fda-68131e90eec2", "db_session_id": "12YC7MVKHO7GI56LLYLY", "orig_file_number": 13, "seqno_to_time_mapping": "N/A"}} 2023-07-14T19:09:01.569 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:09:01 smithi040 ceph-mon[96518]: rocksdb: EVENT_LOG_v1 {"time_micros": 1689361741459827, "job": 1, "event": "recovery_finished"} 2023-07-14T19:09:01.569 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:09:01 smithi040 ceph-mon[96518]: rocksdb: [db/version_set.cc:5047] Creating manifest 15 2023-07-14T19:09:01.569 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:09:01 smithi040 ceph-mon[96518]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000009.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2023-07-14T19:09:01.570 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:09:01 smithi040 ceph-mon[96518]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55f536d5a000 2023-07-14T19:09:01.570 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:09:01 smithi040 ceph-mon[96518]: rocksdb: DB pointer 0x55f536d46000 2023-07-14T19:09:01.570 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:09:01 smithi040 ceph-mon[96518]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2023-07-14T19:09:01.570 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:09:01 smithi040 ceph-mon[96518]: rocksdb: [db/db_impl/db_impl.cc:1111] 2023-07-14T19:09:01.570 INFO:journalctl@ceph.mon.a.smithi040.stdout: ** DB Stats ** 2023-07-14T19:09:01.570 INFO:journalctl@ceph.mon.a.smithi040.stdout: Uptime(secs): 0.0 total, 0.0 interval 2023-07-14T19:09:01.570 INFO:journalctl@ceph.mon.a.smithi040.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2023-07-14T19:09:01.571 INFO:journalctl@ceph.mon.a.smithi040.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2023-07-14T19:09:01.571 INFO:journalctl@ceph.mon.a.smithi040.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2023-07-14T19:09:01.571 INFO:journalctl@ceph.mon.a.smithi040.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2023-07-14T19:09:01.571 INFO:journalctl@ceph.mon.a.smithi040.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2023-07-14T19:09:01.571 INFO:journalctl@ceph.mon.a.smithi040.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2023-07-14T19:09:01.571 INFO:journalctl@ceph.mon.a.smithi040.stdout: 2023-07-14T19:09:01.571 INFO:journalctl@ceph.mon.a.smithi040.stdout: ** Compaction Stats [default] ** 2023-07-14T19:09:01.571 INFO:journalctl@ceph.mon.a.smithi040.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2023-07-14T19:09:01.572 INFO:journalctl@ceph.mon.a.smithi040.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2023-07-14T19:09:01.572 INFO:journalctl@ceph.mon.a.smithi040.stdout: L0 2/0 93.56 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 133.1 0.00 0.00 1 0.001 0 0 0.0 0.0 2023-07-14T19:09:01.572 INFO:journalctl@ceph.mon.a.smithi040.stdout: Sum 2/0 93.56 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 133.1 0.00 0.00 1 0.001 0 0 0.0 0.0 2023-07-14T19:09:01.572 INFO:journalctl@ceph.mon.a.smithi040.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 133.1 0.00 0.00 1 0.001 0 0 0.0 0.0 2023-07-14T19:09:01.572 INFO:journalctl@ceph.mon.a.smithi040.stdout: 2023-07-14T19:09:01.572 INFO:journalctl@ceph.mon.a.smithi040.stdout: ** Compaction Stats [default] ** 2023-07-14T19:09:01.572 INFO:journalctl@ceph.mon.a.smithi040.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2023-07-14T19:09:01.573 INFO:journalctl@ceph.mon.a.smithi040.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2023-07-14T19:09:01.573 INFO:journalctl@ceph.mon.a.smithi040.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 133.1 0.00 0.00 1 0.001 0 0 0.0 0.0 2023-07-14T19:09:01.573 INFO:journalctl@ceph.mon.a.smithi040.stdout: 2023-07-14T19:09:01.573 INFO:journalctl@ceph.mon.a.smithi040.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2023-07-14T19:09:01.573 INFO:journalctl@ceph.mon.a.smithi040.stdout: 2023-07-14T19:09:01.573 INFO:journalctl@ceph.mon.a.smithi040.stdout: Uptime(secs): 0.0 total, 0.0 interval 2023-07-14T19:09:01.573 INFO:journalctl@ceph.mon.a.smithi040.stdout: Flush(GB): cumulative 0.000, interval 0.000 2023-07-14T19:09:01.574 INFO:journalctl@ceph.mon.a.smithi040.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2023-07-14T19:09:01.574 INFO:journalctl@ceph.mon.a.smithi040.stdout: AddFile(Total Files): cumulative 0, interval 0 2023-07-14T19:09:01.574 INFO:journalctl@ceph.mon.a.smithi040.stdout: AddFile(L0 Files): cumulative 0, interval 0 2023-07-14T19:09:01.574 INFO:journalctl@ceph.mon.a.smithi040.stdout: AddFile(Keys): cumulative 0, interval 0 2023-07-14T19:09:01.574 INFO:journalctl@ceph.mon.a.smithi040.stdout: Cumulative compaction: 0.00 GB write, 25.65 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-07-14T19:09:01.574 INFO:journalctl@ceph.mon.a.smithi040.stdout: Interval compaction: 0.00 GB write, 25.65 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-07-14T19:09:01.574 INFO:journalctl@ceph.mon.a.smithi040.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2023-07-14T19:09:01.574 INFO:journalctl@ceph.mon.a.smithi040.stdout: Block cache BinnedLRUCache@0x55f536cc11f0#7 capacity: 512.00 MB usage: 1.36 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 8e-06 secs_since: 0 2023-07-14T19:09:01.575 INFO:journalctl@ceph.mon.a.smithi040.stdout: Block cache entry stats(count,size,portion): FilterBlock(2,0.89 KB,0.000169873%) IndexBlock(2,0.47 KB,8.9407e-05%) Misc(1,0.00 KB,0%) 2023-07-14T19:09:01.575 INFO:journalctl@ceph.mon.a.smithi040.stdout: 2023-07-14T19:09:01.575 INFO:journalctl@ceph.mon.a.smithi040.stdout: ** File Read Latency Histogram By Level [default] ** 2023-07-14T19:09:01.575 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:09:01 smithi040 ceph-mon[96518]: starting mon.a rank 0 at public addrs [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] at bind addrs [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid adc52f04-2279-11ee-9b33-001a4aab830c 2023-07-14T19:09:01.575 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:09:01 smithi040 ceph-mon[96518]: mon.a@-1(???) e1 preinit fsid adc52f04-2279-11ee-9b33-001a4aab830c 2023-07-14T19:09:01.575 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:09:01 smithi040 ceph-mon[96518]: mon.a@-1(???).mds e1 new map 2023-07-14T19:09:01.575 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:09:01 smithi040 ceph-mon[96518]: mon.a@-1(???).mds e1 print_map 2023-07-14T19:09:01.575 INFO:journalctl@ceph.mon.a.smithi040.stdout: e1 2023-07-14T19:09:01.576 INFO:journalctl@ceph.mon.a.smithi040.stdout: enable_multiple, ever_enabled_multiple: 1,1 2023-07-14T19:09:01.576 INFO:journalctl@ceph.mon.a.smithi040.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2023-07-14T19:09:01.576 INFO:journalctl@ceph.mon.a.smithi040.stdout: legacy client fscid: -1 2023-07-14T19:09:01.576 INFO:journalctl@ceph.mon.a.smithi040.stdout: 2023-07-14T19:09:01.576 INFO:journalctl@ceph.mon.a.smithi040.stdout: No filesystems configured 2023-07-14T19:09:01.576 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:09:01 smithi040 ceph-mon[96518]: mon.a@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2023-07-14T19:09:01.576 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:09:01 smithi040 ceph-mon[96518]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2023-07-14T19:09:01.577 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:09:01 smithi040 ceph-mon[96518]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2023-07-14T19:09:01.577 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:09:01 smithi040 ceph-mon[96518]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2023-07-14T19:09:01.577 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:09:01 smithi040 ceph-mon[96518]: mon.a@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2023-07-14T19:09:01.577 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:09:01 smithi040 ceph-mon[96518]: expand_channel_meta expand map: {default=false} 2023-07-14T19:09:01.577 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:09:01 smithi040 ceph-mon[96518]: expand_channel_meta from 'false' to 'false' 2023-07-14T19:09:01.577 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:09:01 smithi040 ceph-mon[96518]: expand_channel_meta expanded map: {default=false} 2023-07-14T19:09:01.577 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:09:01 smithi040 ceph-mon[96518]: expand_channel_meta expand map: {default=info} 2023-07-14T19:09:01.578 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:09:01 smithi040 ceph-mon[96518]: expand_channel_meta from 'info' to 'info' 2023-07-14T19:09:01.578 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:09:01 smithi040 ceph-mon[96518]: expand_channel_meta expanded map: {default=info} 2023-07-14T19:09:01.578 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:09:01 smithi040 ceph-mon[96518]: expand_channel_meta expand map: {default=daemon} 2023-07-14T19:09:01.578 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:09:01 smithi040 ceph-mon[96518]: expand_channel_meta from 'daemon' to 'daemon' 2023-07-14T19:09:01.578 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:09:01 smithi040 ceph-mon[96518]: expand_channel_meta expanded map: {default=daemon} 2023-07-14T19:09:01.578 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:09:01 smithi040 ceph-mon[96518]: expand_channel_meta expand map: {default=debug} 2023-07-14T19:09:01.578 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:09:01 smithi040 ceph-mon[96518]: expand_channel_meta from 'debug' to 'debug' 2023-07-14T19:09:01.579 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:09:01 smithi040 ceph-mon[96518]: expand_channel_meta expanded map: {default=debug} 2023-07-14T19:09:01.579 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:09:01 smithi040 ceph-mon[96518]: mon.a is new leader, mons a in quorum (ranks 0) 2023-07-14T19:09:01.579 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:09:01 smithi040 ceph-mon[96518]: monmap e1: 1 mons at {a=[v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0]} removed_ranks: {} 2023-07-14T19:09:01.579 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:09:01 smithi040 ceph-mon[96518]: fsmap 2023-07-14T19:09:01.579 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:09:01 smithi040 ceph-mon[96518]: osdmap e1: 0 total, 0 up, 0 in 2023-07-14T19:09:01.579 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:09:01 smithi040 ceph-mon[96518]: mgrmap e1: no daemons active 2023-07-14T19:09:01.580 INFO:teuthology.orchestra.run.smithi040.stdout:Setting public_network to 172.21.0.0/20 in mon config section 2023-07-14T19:09:01.789 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:09:01 smithi040 systemd[1]: Started Ceph mon.a for adc52f04-2279-11ee-9b33-001a4aab830c. 2023-07-14T19:09:02.597 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:02.026+0000 7fa695722700 1 Processor -- start 2023-07-14T19:09:02.598 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:02.026+0000 7fa695722700 1 -- start start 2023-07-14T19:09:02.598 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:02.027+0000 7fa695722700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa690105f40 0x7fa690108320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:09:02.598 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:02.027+0000 7fa695722700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa690079d20 con 0x7fa690105f40 2023-07-14T19:09:02.598 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:02.027+0000 7fa68effd700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa690105f40 0x7fa690108320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:09:02.598 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:02.027+0000 7fa68effd700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa690105f40 0x7fa690108320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:59756/0 (socket says 172.21.15.40:59756) 2023-07-14T19:09:02.598 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:02.027+0000 7fa68effd700 1 -- 172.21.15.40:0/325327672 learned_addr learned my addr 172.21.15.40:0/325327672 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-07-14T19:09:02.599 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:02.027+0000 7fa68effd700 1 -- 172.21.15.40:0/325327672 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa690079e60 con 0x7fa690105f40 2023-07-14T19:09:02.599 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:02.028+0000 7fa68effd700 1 --2- 172.21.15.40:0/325327672 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa690105f40 0x7fa690108320 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fa6840098a0 tx=0x7fa684009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b488897c0dd4f837 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:09:02.599 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:02.028+0000 7fa68dffb700 1 -- 172.21.15.40:0/325327672 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa684004160 con 0x7fa690105f40 2023-07-14T19:09:02.599 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:02.028+0000 7fa68dffb700 1 -- 172.21.15.40:0/325327672 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa6840042c0 con 0x7fa690105f40 2023-07-14T19:09:02.599 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:02.028+0000 7fa68dffb700 1 -- 172.21.15.40:0/325327672 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa684011a00 con 0x7fa690105f40 2023-07-14T19:09:02.599 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:02.028+0000 7fa695722700 1 -- 172.21.15.40:0/325327672 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa690105f40 msgr2=0x7fa690108320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:09:02.599 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:02.028+0000 7fa695722700 1 --2- 172.21.15.40:0/325327672 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa690105f40 0x7fa690108320 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fa6840098a0 tx=0x7fa684009bb0 comp rx=0 tx=0).stop 2023-07-14T19:09:02.600 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:02.028+0000 7fa695722700 1 -- 172.21.15.40:0/325327672 shutdown_connections 2023-07-14T19:09:02.600 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:02.028+0000 7fa695722700 1 --2- 172.21.15.40:0/325327672 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa690105f40 0x7fa690108320 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:09:02.600 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:02.028+0000 7fa695722700 1 -- 172.21.15.40:0/325327672 >> 172.21.15.40:0/325327672 conn(0x7fa6900ffa00 msgr2=0x7fa690101e10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:09:02.600 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:02.028+0000 7fa695722700 1 -- 172.21.15.40:0/325327672 shutdown_connections 2023-07-14T19:09:02.600 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:02.029+0000 7fa695722700 1 -- 172.21.15.40:0/325327672 wait complete. 2023-07-14T19:09:02.600 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:02.029+0000 7fa695722700 1 Processor -- start 2023-07-14T19:09:02.600 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:02.030+0000 7fa695722700 1 -- start start 2023-07-14T19:09:02.601 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:02.030+0000 7fa695722700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa690105f40 0x7fa6901990b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:09:02.601 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:02.030+0000 7fa695722700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6901995f0 con 0x7fa690105f40 2023-07-14T19:09:02.601 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:02.030+0000 7fa68effd700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa690105f40 0x7fa6901990b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:09:02.602 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:02.030+0000 7fa68effd700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa690105f40 0x7fa6901990b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:59758/0 (socket says 172.21.15.40:59758) 2023-07-14T19:09:02.602 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:02.030+0000 7fa68effd700 1 -- 172.21.15.40:0/4052061466 learned_addr learned my addr 172.21.15.40:0/4052061466 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-07-14T19:09:02.602 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:02.030+0000 7fa68effd700 1 -- 172.21.15.40:0/4052061466 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa684009550 con 0x7fa690105f40 2023-07-14T19:09:02.602 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:02.030+0000 7fa68effd700 1 --2- 172.21.15.40:0/4052061466 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa690105f40 0x7fa6901990b0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fa684006af0 tx=0x7fa684011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:09:02.602 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:02.030+0000 7fa677fff700 1 -- 172.21.15.40:0/4052061466 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa684003a40 con 0x7fa690105f40 2023-07-14T19:09:02.602 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:02.030+0000 7fa677fff700 1 -- 172.21.15.40:0/4052061466 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa684003ba0 con 0x7fa690105f40 2023-07-14T19:09:02.603 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:02.031+0000 7fa677fff700 1 -- 172.21.15.40:0/4052061466 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa68401a700 con 0x7fa690105f40 2023-07-14T19:09:02.603 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:02.031+0000 7fa695722700 1 -- 172.21.15.40:0/4052061466 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa690199850 con 0x7fa690105f40 2023-07-14T19:09:02.603 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:02.031+0000 7fa695722700 1 -- 172.21.15.40:0/4052061466 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa690199cd0 con 0x7fa690105f40 2023-07-14T19:09:02.603 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:02.031+0000 7fa677fff700 1 -- 172.21.15.40:0/4052061466 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7fa68401ac30 con 0x7fa690105f40 2023-07-14T19:09:02.603 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:02.031+0000 7fa677fff700 1 -- 172.21.15.40:0/4052061466 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7fa684019700 con 0x7fa690105f40 2023-07-14T19:09:02.603 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:02.032+0000 7fa695722700 1 -- 172.21.15.40:0/4052061466 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa690109460 con 0x7fa690105f40 2023-07-14T19:09:02.603 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:02.034+0000 7fa677fff700 1 -- 172.21.15.40:0/4052061466 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72446 (secure 0 0 0) 0x7fa68402de30 con 0x7fa690105f40 2023-07-14T19:09:02.604 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:02.159+0000 7fa695722700 1 -- 172.21.15.40:0/4052061466 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command([{prefix=config set, name=public_network}] v 0) v1 -- 0x7fa69019ca70 con 0x7fa690105f40 2023-07-14T19:09:02.604 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:02.161+0000 7fa677fff700 1 -- 172.21.15.40:0/4052061466 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{prefix=config set, name=public_network}]=0 v3)=0 v3) v1 ==== 124+0+0 (secure 0 0 0) 0x7fa684024ad0 con 0x7fa690105f40 2023-07-14T19:09:02.604 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:02.162+0000 7fa695722700 1 -- 172.21.15.40:0/4052061466 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa690105f40 msgr2=0x7fa6901990b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:09:02.604 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:02.162+0000 7fa695722700 1 --2- 172.21.15.40:0/4052061466 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa690105f40 0x7fa6901990b0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fa684006af0 tx=0x7fa684011ad0 comp rx=0 tx=0).stop 2023-07-14T19:09:02.604 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:02.162+0000 7fa695722700 1 -- 172.21.15.40:0/4052061466 shutdown_connections 2023-07-14T19:09:02.605 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:02.162+0000 7fa695722700 1 --2- 172.21.15.40:0/4052061466 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa690105f40 0x7fa6901990b0 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:09:02.605 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:02.162+0000 7fa695722700 1 -- 172.21.15.40:0/4052061466 >> 172.21.15.40:0/4052061466 conn(0x7fa6900ffa00 msgr2=0x7fa690101080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:09:02.605 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:02.162+0000 7fa695722700 1 -- 172.21.15.40:0/4052061466 shutdown_connections 2023-07-14T19:09:02.605 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:02.162+0000 7fa695722700 1 -- 172.21.15.40:0/4052061466 wait complete. 2023-07-14T19:09:02.605 INFO:teuthology.orchestra.run.smithi040.stdout:Wrote config to /etc/ceph/ceph.conf 2023-07-14T19:09:02.624 INFO:teuthology.orchestra.run.smithi040.stdout:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2023-07-14T19:09:02.625 INFO:teuthology.orchestra.run.smithi040.stdout:Creating mgr... 2023-07-14T19:09:02.626 INFO:teuthology.orchestra.run.smithi040.stdout:Verifying port 9283 ... 2023-07-14T19:09:02.626 INFO:teuthology.orchestra.run.smithi040.stdout:Verifying port 8765 ... 2023-07-14T19:09:02.833 INFO:teuthology.orchestra.run.smithi040.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-adc52f04-2279-11ee-9b33-001a4aab830c@mgr.x 2023-07-14T19:09:02.833 INFO:teuthology.orchestra.run.smithi040.stdout:systemctl: stderr Failed to reset failed state of unit ceph-adc52f04-2279-11ee-9b33-001a4aab830c@mgr.x.service: Unit ceph-adc52f04-2279-11ee-9b33-001a4aab830c@mgr.x.service not loaded. 2023-07-14T19:09:02.927 INFO:teuthology.orchestra.run.smithi040.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-adc52f04-2279-11ee-9b33-001a4aab830c.target.wants/ceph-adc52f04-2279-11ee-9b33-001a4aab830c@mgr.x.service → /etc/systemd/system/ceph-adc52f04-2279-11ee-9b33-001a4aab830c@.service. 2023-07-14T19:09:03.163 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Jul 14 19:09:02 smithi040 systemd[1]: Starting Ceph mgr.x for adc52f04-2279-11ee-9b33-001a4aab830c... 2023-07-14T19:09:03.463 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:09:03 smithi040 ceph-mon[96518]: from='client.? 172.21.15.40:0/4052061466' entity='client.admin' 2023-07-14T19:09:03.463 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Jul 14 19:09:03 smithi040 podman[96746]: 2023-07-14T19:09:03.538 INFO:teuthology.orchestra.run.smithi040.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2023-07-14T19:09:03.538 INFO:teuthology.orchestra.run.smithi040.stdout:systemctl: stdout disabled 2023-07-14T19:09:03.547 INFO:teuthology.orchestra.run.smithi040.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2023-07-14T19:09:03.547 INFO:teuthology.orchestra.run.smithi040.stdout:systemctl: stdout inactive 2023-07-14T19:09:03.547 INFO:teuthology.orchestra.run.smithi040.stdout:firewalld.service is not enabled 2023-07-14T19:09:03.547 INFO:teuthology.orchestra.run.smithi040.stdout:Not possible to enable service . firewalld.service is not available 2023-07-14T19:09:03.557 INFO:teuthology.orchestra.run.smithi040.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2023-07-14T19:09:03.557 INFO:teuthology.orchestra.run.smithi040.stdout:systemctl: stdout disabled 2023-07-14T19:09:03.567 INFO:teuthology.orchestra.run.smithi040.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2023-07-14T19:09:03.567 INFO:teuthology.orchestra.run.smithi040.stdout:systemctl: stdout inactive 2023-07-14T19:09:03.567 INFO:teuthology.orchestra.run.smithi040.stdout:firewalld.service is not enabled 2023-07-14T19:09:03.568 INFO:teuthology.orchestra.run.smithi040.stdout:Not possible to open ports <[9283, 8765]>. firewalld.service is not available 2023-07-14T19:09:03.568 INFO:teuthology.orchestra.run.smithi040.stdout:Waiting for mgr to start... 2023-07-14T19:09:03.568 INFO:teuthology.orchestra.run.smithi040.stdout:Waiting for mgr... 2023-07-14T19:09:03.717 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Jul 14 19:09:03 smithi040 bash[96746]: 51022a857c7c5fccb9f9e799b486f6e6342b2cddf836f36bbf6b26af24bf4410 2023-07-14T19:09:03.717 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Jul 14 19:09:03 smithi040 systemd[1]: Started Ceph mgr.x for adc52f04-2279-11ee-9b33-001a4aab830c. 2023-07-14T19:09:03.718 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Jul 14 19:09:03 smithi040 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-mgr-x[96762]: 2023-07-14T19:09:03.617+0000 7f5cf9a5d200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-07-14T19:09:03.718 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Jul 14 19:09:03 smithi040 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-mgr-x[96762]: 2023-07-14T19:09:03.714+0000 7f5cf9a5d200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-07-14T19:09:03.971 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Jul 14 19:09:03 smithi040 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-mgr-x[96762]: 2023-07-14T19:09:03.825+0000 7f5cf9a5d200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-07-14T19:09:03.972 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Jul 14 19:09:03 smithi040 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-mgr-x[96762]: 2023-07-14T19:09:03.939+0000 7f5cf9a5d200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-07-14T19:09:04.430 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:09:04 smithi040 ceph-mon[96518]: from='client.? 172.21.15.40:0/410482089' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2023-07-14T19:09:04.431 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Jul 14 19:09:04 smithi040 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-mgr-x[96762]: 2023-07-14T19:09:04.152+0000 7f5cf9a5d200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-07-14T19:09:04.615 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout 2023-07-14T19:09:04.617 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout { 2023-07-14T19:09:04.617 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "fsid": "adc52f04-2279-11ee-9b33-001a4aab830c", 2023-07-14T19:09:04.618 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "health": { 2023-07-14T19:09:04.618 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2023-07-14T19:09:04.618 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "checks": {}, 2023-07-14T19:09:04.618 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "mutes": [] 2023-07-14T19:09:04.618 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout }, 2023-07-14T19:09:04.618 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2023-07-14T19:09:04.618 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "quorum": [ 2023-07-14T19:09:04.619 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout 0 2023-07-14T19:09:04.619 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout ], 2023-07-14T19:09:04.619 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2023-07-14T19:09:04.619 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "a" 2023-07-14T19:09:04.619 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout ], 2023-07-14T19:09:04.619 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "quorum_age": 2, 2023-07-14T19:09:04.619 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "monmap": { 2023-07-14T19:09:04.619 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-07-14T19:09:04.620 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "reef", 2023-07-14T19:09:04.620 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "num_mons": 1 2023-07-14T19:09:04.620 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout }, 2023-07-14T19:09:04.620 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "osdmap": { 2023-07-14T19:09:04.620 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-07-14T19:09:04.620 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2023-07-14T19:09:04.621 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2023-07-14T19:09:04.621 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2023-07-14T19:09:04.627 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2023-07-14T19:09:04.628 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2023-07-14T19:09:04.629 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2023-07-14T19:09:04.629 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout }, 2023-07-14T19:09:04.629 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "pgmap": { 2023-07-14T19:09:04.630 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2023-07-14T19:09:04.630 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2023-07-14T19:09:04.630 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2023-07-14T19:09:04.630 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2023-07-14T19:09:04.631 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2023-07-14T19:09:04.631 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2023-07-14T19:09:04.631 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2023-07-14T19:09:04.631 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2023-07-14T19:09:04.631 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout }, 2023-07-14T19:09:04.632 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "fsmap": { 2023-07-14T19:09:04.632 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-07-14T19:09:04.632 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "by_rank": [], 2023-07-14T19:09:04.632 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "up:standby": 0 2023-07-14T19:09:04.633 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout }, 2023-07-14T19:09:04.633 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "mgrmap": { 2023-07-14T19:09:04.633 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "available": false, 2023-07-14T19:09:04.634 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2023-07-14T19:09:04.634 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "modules": [ 2023-07-14T19:09:04.634 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "iostat", 2023-07-14T19:09:04.634 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "nfs", 2023-07-14T19:09:04.634 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "restful" 2023-07-14T19:09:04.635 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout ], 2023-07-14T19:09:04.635 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "services": {} 2023-07-14T19:09:04.635 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout }, 2023-07-14T19:09:04.635 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "servicemap": { 2023-07-14T19:09:04.635 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-07-14T19:09:04.635 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "modified": "2023-07-14T19:08:56.894377+0000", 2023-07-14T19:09:04.635 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "services": {} 2023-07-14T19:09:04.635 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout }, 2023-07-14T19:09:04.636 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "progress_events": {} 2023-07-14T19:09:04.636 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout } 2023-07-14T19:09:04.636 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:04.144+0000 7f45046c8700 1 Processor -- start 2023-07-14T19:09:04.636 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:04.144+0000 7f45046c8700 1 -- start start 2023-07-14T19:09:04.636 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:04.145+0000 7f45046c8700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f44fc105f40 0x7f44fc108320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:09:04.636 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:04.145+0000 7f45046c8700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44fc079d20 con 0x7f44fc105f40 2023-07-14T19:09:04.637 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:04.145+0000 7f4502464700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f44fc105f40 0x7f44fc108320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:09:04.637 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:04.145+0000 7f4502464700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f44fc105f40 0x7f44fc108320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:59764/0 (socket says 172.21.15.40:59764) 2023-07-14T19:09:04.637 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:04.145+0000 7f4502464700 1 -- 172.21.15.40:0/2916798032 learned_addr learned my addr 172.21.15.40:0/2916798032 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-07-14T19:09:04.637 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:04.145+0000 7f4502464700 1 -- 172.21.15.40:0/2916798032 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44fc079e60 con 0x7f44fc105f40 2023-07-14T19:09:04.638 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:04.145+0000 7f4502464700 1 --2- 172.21.15.40:0/2916798032 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f44fc105f40 0x7f44fc108320 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f44f80098a0 tx=0x7f44f8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a32e86167a12fa94 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:09:04.638 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:04.146+0000 7f4501462700 1 -- 172.21.15.40:0/2916798032 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f44f8004160 con 0x7f44fc105f40 2023-07-14T19:09:04.638 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:04.146+0000 7f4501462700 1 -- 172.21.15.40:0/2916798032 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f44f80042c0 con 0x7f44fc105f40 2023-07-14T19:09:04.638 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:04.146+0000 7f4501462700 1 -- 172.21.15.40:0/2916798032 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f44f8011a00 con 0x7f44fc105f40 2023-07-14T19:09:04.638 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:04.146+0000 7f45046c8700 1 -- 172.21.15.40:0/2916798032 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f44fc105f40 msgr2=0x7f44fc108320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:09:04.638 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:04.146+0000 7f45046c8700 1 --2- 172.21.15.40:0/2916798032 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f44fc105f40 0x7f44fc108320 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f44f80098a0 tx=0x7f44f8009bb0 comp rx=0 tx=0).stop 2023-07-14T19:09:04.639 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:04.146+0000 7f45046c8700 1 -- 172.21.15.40:0/2916798032 shutdown_connections 2023-07-14T19:09:04.639 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:04.146+0000 7f45046c8700 1 --2- 172.21.15.40:0/2916798032 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f44fc105f40 0x7f44fc108320 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:09:04.639 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:04.146+0000 7f45046c8700 1 -- 172.21.15.40:0/2916798032 >> 172.21.15.40:0/2916798032 conn(0x7f44fc0ff9a0 msgr2=0x7f44fc101db0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:09:04.639 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:04.147+0000 7f45046c8700 1 -- 172.21.15.40:0/2916798032 shutdown_connections 2023-07-14T19:09:04.639 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:04.147+0000 7f45046c8700 1 -- 172.21.15.40:0/2916798032 wait complete. 2023-07-14T19:09:04.639 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:04.147+0000 7f45046c8700 1 Processor -- start 2023-07-14T19:09:04.639 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:04.147+0000 7f45046c8700 1 -- start start 2023-07-14T19:09:04.639 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:04.148+0000 7f45046c8700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f44fc105f40 0x7f44fc1058a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:09:04.640 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:04.148+0000 7f45046c8700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44fc105de0 con 0x7f44fc105f40 2023-07-14T19:09:04.640 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:04.148+0000 7f4502464700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f44fc105f40 0x7f44fc1058a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:09:04.640 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:04.148+0000 7f4502464700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f44fc105f40 0x7f44fc1058a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:59766/0 (socket says 172.21.15.40:59766) 2023-07-14T19:09:04.640 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:04.148+0000 7f4502464700 1 -- 172.21.15.40:0/410482089 learned_addr learned my addr 172.21.15.40:0/410482089 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-07-14T19:09:04.640 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:04.148+0000 7f4502464700 1 -- 172.21.15.40:0/410482089 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44f8009550 con 0x7f44fc105f40 2023-07-14T19:09:04.640 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:04.148+0000 7f4502464700 1 --2- 172.21.15.40:0/410482089 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f44fc105f40 0x7f44fc1058a0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f44f8009fb0 tx=0x7f44f8011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:09:04.640 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:04.148+0000 7f44f37fe700 1 -- 172.21.15.40:0/410482089 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f44f8003930 con 0x7f44fc105f40 2023-07-14T19:09:04.641 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:04.148+0000 7f44f37fe700 1 -- 172.21.15.40:0/410482089 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f44f8003a90 con 0x7f44fc105f40 2023-07-14T19:09:04.641 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:04.148+0000 7f44f37fe700 1 -- 172.21.15.40:0/410482089 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f44f80195e0 con 0x7f44fc105f40 2023-07-14T19:09:04.641 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:04.148+0000 7f45046c8700 1 -- 172.21.15.40:0/410482089 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f44fc103f50 con 0x7f44fc105f40 2023-07-14T19:09:04.641 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:04.148+0000 7f45046c8700 1 -- 172.21.15.40:0/410482089 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f44fc1043d0 con 0x7f44fc105f40 2023-07-14T19:09:04.641 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:04.149+0000 7f44f37fe700 1 -- 172.21.15.40:0/410482089 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f44f8019850 con 0x7f44fc105f40 2023-07-14T19:09:04.641 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:04.150+0000 7f44f37fe700 1 -- 172.21.15.40:0/410482089 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f44f802d970 con 0x7f44fc105f40 2023-07-14T19:09:04.641 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:04.149+0000 7f45046c8700 1 -- 172.21.15.40:0/410482089 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f44fc109460 con 0x7f44fc105f40 2023-07-14T19:09:04.642 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:04.152+0000 7f44f37fe700 1 -- 172.21.15.40:0/410482089 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72446 (secure 0 0 0) 0x7f44f802d440 con 0x7f44fc105f40 2023-07-14T19:09:04.642 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:04.284+0000 7f45046c8700 1 -- 172.21.15.40:0/410482089 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f44fc109680 con 0x7f44fc105f40 2023-07-14T19:09:04.642 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:04.284+0000 7f44f37fe700 1 -- 172.21.15.40:0/410482089 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1238 (secure 0 0 0) 0x7f44f8003c00 con 0x7f44fc105f40 2023-07-14T19:09:04.642 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:04.285+0000 7f45046c8700 1 -- 172.21.15.40:0/410482089 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f44fc105f40 msgr2=0x7f44fc1058a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:09:04.642 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:04.285+0000 7f45046c8700 1 --2- 172.21.15.40:0/410482089 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f44fc105f40 0x7f44fc1058a0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f44f8009fb0 tx=0x7f44f8011ad0 comp rx=0 tx=0).stop 2023-07-14T19:09:04.642 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:04.285+0000 7f45046c8700 1 -- 172.21.15.40:0/410482089 shutdown_connections 2023-07-14T19:09:04.642 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:04.285+0000 7f45046c8700 1 --2- 172.21.15.40:0/410482089 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f44fc105f40 0x7f44fc1058a0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:09:04.643 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:04.285+0000 7f45046c8700 1 -- 172.21.15.40:0/410482089 >> 172.21.15.40:0/410482089 conn(0x7f44fc0ff9a0 msgr2=0x7f44fc101070 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:09:04.643 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:04.286+0000 7f45046c8700 1 -- 172.21.15.40:0/410482089 shutdown_connections 2023-07-14T19:09:04.643 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:04.286+0000 7f45046c8700 1 -- 172.21.15.40:0/410482089 wait complete. 2023-07-14T19:09:04.643 INFO:teuthology.orchestra.run.smithi040.stdout:mgr not available, waiting (1/15)... 2023-07-14T19:09:04.722 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Jul 14 19:09:04 smithi040 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-mgr-x[96762]: 2023-07-14T19:09:04.558+0000 7f5cf9a5d200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-07-14T19:09:04.722 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Jul 14 19:09:04 smithi040 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-mgr-x[96762]: 2023-07-14T19:09:04.669+0000 7f5cf9a5d200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-07-14T19:09:05.221 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Jul 14 19:09:04 smithi040 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-mgr-x[96762]: 2023-07-14T19:09:04.820+0000 7f5cf9a5d200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-07-14T19:09:05.222 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Jul 14 19:09:04 smithi040 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-mgr-x[96762]: 2023-07-14T19:09:04.935+0000 7f5cf9a5d200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-07-14T19:09:05.722 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Jul 14 19:09:05 smithi040 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-mgr-x[96762]: 2023-07-14T19:09:05.451+0000 7f5cf9a5d200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-07-14T19:09:06.472 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Jul 14 19:09:06 smithi040 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-mgr-x[96762]: 2023-07-14T19:09:06.146+0000 7f5cf9a5d200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-07-14T19:09:06.472 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Jul 14 19:09:06 smithi040 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-mgr-x[96762]: 2023-07-14T19:09:06.337+0000 7f5cf9a5d200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-07-14T19:09:07.651 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:09:07 smithi040 ceph-mon[96518]: from='client.? 172.21.15.40:0/1441173458' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2023-07-14T19:09:07.651 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Jul 14 19:09:07 smithi040 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-mgr-x[96762]: 2023-07-14T19:09:07.319+0000 7f5cf9a5d200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-07-14T19:09:07.651 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Jul 14 19:09:07 smithi040 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-mgr-x[96762]: 2023-07-14T19:09:07.436+0000 7f5cf9a5d200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-07-14T19:09:07.672 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout 2023-07-14T19:09:07.672 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout { 2023-07-14T19:09:07.672 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "fsid": "adc52f04-2279-11ee-9b33-001a4aab830c", 2023-07-14T19:09:07.673 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "health": { 2023-07-14T19:09:07.673 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2023-07-14T19:09:07.673 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "checks": {}, 2023-07-14T19:09:07.673 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "mutes": [] 2023-07-14T19:09:07.673 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout }, 2023-07-14T19:09:07.674 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2023-07-14T19:09:07.674 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "quorum": [ 2023-07-14T19:09:07.674 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout 0 2023-07-14T19:09:07.674 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout ], 2023-07-14T19:09:07.674 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2023-07-14T19:09:07.674 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "a" 2023-07-14T19:09:07.674 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout ], 2023-07-14T19:09:07.675 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "quorum_age": 5, 2023-07-14T19:09:07.675 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "monmap": { 2023-07-14T19:09:07.675 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-07-14T19:09:07.675 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "reef", 2023-07-14T19:09:07.675 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "num_mons": 1 2023-07-14T19:09:07.675 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout }, 2023-07-14T19:09:07.675 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "osdmap": { 2023-07-14T19:09:07.675 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-07-14T19:09:07.676 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2023-07-14T19:09:07.676 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2023-07-14T19:09:07.676 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2023-07-14T19:09:07.676 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2023-07-14T19:09:07.676 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2023-07-14T19:09:07.676 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2023-07-14T19:09:07.676 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout }, 2023-07-14T19:09:07.677 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "pgmap": { 2023-07-14T19:09:07.677 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2023-07-14T19:09:07.677 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2023-07-14T19:09:07.677 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2023-07-14T19:09:07.677 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2023-07-14T19:09:07.677 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2023-07-14T19:09:07.677 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2023-07-14T19:09:07.678 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2023-07-14T19:09:07.678 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2023-07-14T19:09:07.678 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout }, 2023-07-14T19:09:07.678 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "fsmap": { 2023-07-14T19:09:07.678 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-07-14T19:09:07.678 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "by_rank": [], 2023-07-14T19:09:07.678 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "up:standby": 0 2023-07-14T19:09:07.679 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout }, 2023-07-14T19:09:07.679 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "mgrmap": { 2023-07-14T19:09:07.679 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "available": false, 2023-07-14T19:09:07.679 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2023-07-14T19:09:07.679 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "modules": [ 2023-07-14T19:09:07.679 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "iostat", 2023-07-14T19:09:07.679 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "nfs", 2023-07-14T19:09:07.680 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "restful" 2023-07-14T19:09:07.680 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout ], 2023-07-14T19:09:07.680 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "services": {} 2023-07-14T19:09:07.680 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout }, 2023-07-14T19:09:07.680 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "servicemap": { 2023-07-14T19:09:07.680 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-07-14T19:09:07.680 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "modified": "2023-07-14T19:08:56.894377+0000", 2023-07-14T19:09:07.681 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "services": {} 2023-07-14T19:09:07.681 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout }, 2023-07-14T19:09:07.681 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "progress_events": {} 2023-07-14T19:09:07.681 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout } 2023-07-14T19:09:07.681 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:07.126+0000 7f049c34b700 1 Processor -- start 2023-07-14T19:09:07.681 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:07.126+0000 7f049c34b700 1 -- start start 2023-07-14T19:09:07.681 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:07.126+0000 7f049c34b700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f04941042f0 0x7f0494104700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:09:07.681 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:07.126+0000 7f049c34b700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0494104c40 con 0x7f04941042f0 2023-07-14T19:09:07.682 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:07.127+0000 7f049a0e7700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f04941042f0 0x7f0494104700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:09:07.682 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:07.127+0000 7f049a0e7700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f04941042f0 0x7f0494104700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:59768/0 (socket says 172.21.15.40:59768) 2023-07-14T19:09:07.682 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:07.127+0000 7f049a0e7700 1 -- 172.21.15.40:0/2524111320 learned_addr learned my addr 172.21.15.40:0/2524111320 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-07-14T19:09:07.682 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:07.127+0000 7f049a0e7700 1 -- 172.21.15.40:0/2524111320 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0494104d80 con 0x7f04941042f0 2023-07-14T19:09:07.682 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:07.127+0000 7f049a0e7700 1 --2- 172.21.15.40:0/2524111320 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f04941042f0 0x7f0494104700 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f04900098a0 tx=0x7f0490009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=54176a2cca1b14dd server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:09:07.682 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:07.127+0000 7f04990e5700 1 -- 172.21.15.40:0/2524111320 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0490004160 con 0x7f04941042f0 2023-07-14T19:09:07.682 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:07.127+0000 7f04990e5700 1 -- 172.21.15.40:0/2524111320 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f04900042c0 con 0x7f04941042f0 2023-07-14T19:09:07.683 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:07.127+0000 7f04990e5700 1 -- 172.21.15.40:0/2524111320 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0490011a00 con 0x7f04941042f0 2023-07-14T19:09:07.683 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:07.128+0000 7f049c34b700 1 -- 172.21.15.40:0/2524111320 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f04941042f0 msgr2=0x7f0494104700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:09:07.683 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:07.128+0000 7f049c34b700 1 --2- 172.21.15.40:0/2524111320 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f04941042f0 0x7f0494104700 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f04900098a0 tx=0x7f0490009bb0 comp rx=0 tx=0).stop 2023-07-14T19:09:07.683 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:07.128+0000 7f049c34b700 1 -- 172.21.15.40:0/2524111320 shutdown_connections 2023-07-14T19:09:07.683 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:07.128+0000 7f049c34b700 1 --2- 172.21.15.40:0/2524111320 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f04941042f0 0x7f0494104700 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:09:07.683 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:07.128+0000 7f049c34b700 1 -- 172.21.15.40:0/2524111320 >> 172.21.15.40:0/2524111320 conn(0x7f04940ff9a0 msgr2=0x7f0494101db0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:09:07.683 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:07.129+0000 7f049c34b700 1 -- 172.21.15.40:0/2524111320 shutdown_connections 2023-07-14T19:09:07.683 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:07.129+0000 7f049c34b700 1 -- 172.21.15.40:0/2524111320 wait complete. 2023-07-14T19:09:07.684 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:07.129+0000 7f049c34b700 1 Processor -- start 2023-07-14T19:09:07.684 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:07.129+0000 7f049c34b700 1 -- start start 2023-07-14T19:09:07.684 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:07.130+0000 7f049c34b700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f04941042f0 0x7f0494197fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:09:07.684 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:07.130+0000 7f049c34b700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0494198500 con 0x7f04941042f0 2023-07-14T19:09:07.684 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:07.130+0000 7f049a0e7700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f04941042f0 0x7f0494197fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:09:07.685 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:07.130+0000 7f049a0e7700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f04941042f0 0x7f0494197fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:59770/0 (socket says 172.21.15.40:59770) 2023-07-14T19:09:07.685 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:07.130+0000 7f049a0e7700 1 -- 172.21.15.40:0/1441173458 learned_addr learned my addr 172.21.15.40:0/1441173458 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-07-14T19:09:07.685 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:07.130+0000 7f049a0e7700 1 -- 172.21.15.40:0/1441173458 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0490009550 con 0x7f04941042f0 2023-07-14T19:09:07.685 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:07.130+0000 7f049a0e7700 1 --2- 172.21.15.40:0/1441173458 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f04941042f0 0x7f0494197fc0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f0490006af0 tx=0x7f0490011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:09:07.685 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:07.130+0000 7f048b7fe700 1 -- 172.21.15.40:0/1441173458 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0490003a40 con 0x7f04941042f0 2023-07-14T19:09:07.685 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:07.130+0000 7f048b7fe700 1 -- 172.21.15.40:0/1441173458 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0490003ba0 con 0x7f04941042f0 2023-07-14T19:09:07.685 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:07.130+0000 7f049c34b700 1 -- 172.21.15.40:0/1441173458 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0494198700 con 0x7f04941042f0 2023-07-14T19:09:07.685 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:07.130+0000 7f048b7fe700 1 -- 172.21.15.40:0/1441173458 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f049001a6e0 con 0x7f04941042f0 2023-07-14T19:09:07.686 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:07.130+0000 7f049c34b700 1 -- 172.21.15.40:0/1441173458 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0494198b20 con 0x7f04941042f0 2023-07-14T19:09:07.686 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:07.131+0000 7f048b7fe700 1 -- 172.21.15.40:0/1441173458 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f049001ab60 con 0x7f04941042f0 2023-07-14T19:09:07.686 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:07.131+0000 7f048b7fe700 1 -- 172.21.15.40:0/1441173458 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f0490019700 con 0x7f04941042f0 2023-07-14T19:09:07.686 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:07.132+0000 7f049c34b700 1 -- 172.21.15.40:0/1441173458 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0494061c70 con 0x7f04941042f0 2023-07-14T19:09:07.686 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:07.134+0000 7f048b7fe700 1 -- 172.21.15.40:0/1441173458 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72446 (secure 0 0 0) 0x7f049002dde0 con 0x7f04941042f0 2023-07-14T19:09:07.686 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:07.266+0000 7f049c34b700 1 -- 172.21.15.40:0/1441173458 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f04941061b0 con 0x7f04941042f0 2023-07-14T19:09:07.686 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:07.267+0000 7f048b7fe700 1 -- 172.21.15.40:0/1441173458 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1238 (secure 0 0 0) 0x7f0490024a60 con 0x7f04941042f0 2023-07-14T19:09:07.687 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:07.268+0000 7f049c34b700 1 -- 172.21.15.40:0/1441173458 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f04941042f0 msgr2=0x7f0494197fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:09:07.687 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:07.268+0000 7f049c34b700 1 --2- 172.21.15.40:0/1441173458 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f04941042f0 0x7f0494197fc0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f0490006af0 tx=0x7f0490011ad0 comp rx=0 tx=0).stop 2023-07-14T19:09:07.687 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:07.268+0000 7f049c34b700 1 -- 172.21.15.40:0/1441173458 shutdown_connections 2023-07-14T19:09:07.687 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:07.268+0000 7f049c34b700 1 --2- 172.21.15.40:0/1441173458 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f04941042f0 0x7f0494197fc0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:09:07.687 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:07.268+0000 7f049c34b700 1 -- 172.21.15.40:0/1441173458 >> 172.21.15.40:0/1441173458 conn(0x7f04940ff9a0 msgr2=0x7f0494100780 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:09:07.687 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:07.268+0000 7f049c34b700 1 -- 172.21.15.40:0/1441173458 shutdown_connections 2023-07-14T19:09:07.687 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:07.268+0000 7f049c34b700 1 -- 172.21.15.40:0/1441173458 wait complete. 2023-07-14T19:09:07.687 INFO:teuthology.orchestra.run.smithi040.stdout:mgr not available, waiting (2/15)... 2023-07-14T19:09:07.971 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Jul 14 19:09:07 smithi040 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-mgr-x[96762]: 2023-07-14T19:09:07.655+0000 7f5cf9a5d200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-07-14T19:09:08.262 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Jul 14 19:09:08 smithi040 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-mgr-x[96762]: 2023-07-14T19:09:08.004+0000 7f5cf9a5d200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-07-14T19:09:08.721 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Jul 14 19:09:08 smithi040 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-mgr-x[96762]: 2023-07-14T19:09:08.259+0000 7f5cf9a5d200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-07-14T19:09:08.722 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Jul 14 19:09:08 smithi040 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-mgr-x[96762]: 2023-07-14T19:09:08.440+0000 7f5cf9a5d200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-07-14T19:09:09.222 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Jul 14 19:09:08 smithi040 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-mgr-x[96762]: 2023-07-14T19:09:08.766+0000 7f5cf9a5d200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-07-14T19:09:09.903 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Jul 14 19:09:09 smithi040 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-mgr-x[96762]: 2023-07-14T19:09:09.584+0000 7f5cf9a5d200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-07-14T19:09:09.903 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Jul 14 19:09:09 smithi040 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-mgr-x[96762]: 2023-07-14T19:09:09.748+0000 7f5cf9a5d200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-07-14T19:09:10.222 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Jul 14 19:09:09 smithi040 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-mgr-x[96762]: 2023-07-14T19:09:09.997+0000 7f5cf9a5d200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-07-14T19:09:10.223 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Jul 14 19:09:10 smithi040 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-mgr-x[96762]: 2023-07-14T19:09:10.136+0000 7f5cf9a5d200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-07-14T19:09:10.687 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:09:10 smithi040 ceph-mon[96518]: from='client.? 172.21.15.40:0/2196241610' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2023-07-14T19:09:10.688 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Jul 14 19:09:10 smithi040 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-mgr-x[96762]: 2023-07-14T19:09:10.414+0000 7f5cf9a5d200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-07-14T19:09:10.722 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout 2023-07-14T19:09:10.722 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout { 2023-07-14T19:09:10.722 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "fsid": "adc52f04-2279-11ee-9b33-001a4aab830c", 2023-07-14T19:09:10.722 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "health": { 2023-07-14T19:09:10.722 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2023-07-14T19:09:10.723 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "checks": {}, 2023-07-14T19:09:10.723 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "mutes": [] 2023-07-14T19:09:10.723 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout }, 2023-07-14T19:09:10.723 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2023-07-14T19:09:10.723 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "quorum": [ 2023-07-14T19:09:10.723 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout 0 2023-07-14T19:09:10.723 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout ], 2023-07-14T19:09:10.723 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2023-07-14T19:09:10.724 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "a" 2023-07-14T19:09:10.724 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout ], 2023-07-14T19:09:10.724 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "quorum_age": 8, 2023-07-14T19:09:10.724 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "monmap": { 2023-07-14T19:09:10.724 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-07-14T19:09:10.725 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "reef", 2023-07-14T19:09:10.725 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "num_mons": 1 2023-07-14T19:09:10.725 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout }, 2023-07-14T19:09:10.725 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "osdmap": { 2023-07-14T19:09:10.725 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-07-14T19:09:10.725 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2023-07-14T19:09:10.725 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2023-07-14T19:09:10.726 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2023-07-14T19:09:10.726 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2023-07-14T19:09:10.726 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2023-07-14T19:09:10.726 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2023-07-14T19:09:10.726 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout }, 2023-07-14T19:09:10.726 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "pgmap": { 2023-07-14T19:09:10.726 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2023-07-14T19:09:10.726 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2023-07-14T19:09:10.727 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2023-07-14T19:09:10.727 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2023-07-14T19:09:10.727 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2023-07-14T19:09:10.728 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2023-07-14T19:09:10.728 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2023-07-14T19:09:10.728 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2023-07-14T19:09:10.728 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout }, 2023-07-14T19:09:10.728 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "fsmap": { 2023-07-14T19:09:10.728 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-07-14T19:09:10.729 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "by_rank": [], 2023-07-14T19:09:10.729 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "up:standby": 0 2023-07-14T19:09:10.729 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout }, 2023-07-14T19:09:10.729 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "mgrmap": { 2023-07-14T19:09:10.729 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "available": false, 2023-07-14T19:09:10.729 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2023-07-14T19:09:10.729 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "modules": [ 2023-07-14T19:09:10.729 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "iostat", 2023-07-14T19:09:10.730 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "nfs", 2023-07-14T19:09:10.730 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "restful" 2023-07-14T19:09:10.730 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout ], 2023-07-14T19:09:10.730 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "services": {} 2023-07-14T19:09:10.730 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout }, 2023-07-14T19:09:10.730 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "servicemap": { 2023-07-14T19:09:10.730 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-07-14T19:09:10.731 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "modified": "2023-07-14T19:08:56.894377+0000", 2023-07-14T19:09:10.731 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "services": {} 2023-07-14T19:09:10.731 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout }, 2023-07-14T19:09:10.731 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "progress_events": {} 2023-07-14T19:09:10.731 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout } 2023-07-14T19:09:10.731 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:10.180+0000 7f0180f19700 1 Processor -- start 2023-07-14T19:09:10.731 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:10.180+0000 7f0180f19700 1 -- start start 2023-07-14T19:09:10.731 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:10.180+0000 7f0180f19700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f017c103d30 0x7f017c104140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:09:10.732 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:10.180+0000 7f0180f19700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f017c079d20 con 0x7f017c103d30 2023-07-14T19:09:10.732 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:10.180+0000 7f017a59c700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f017c103d30 0x7f017c104140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:09:10.732 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:10.180+0000 7f017a59c700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f017c103d30 0x7f017c104140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:59772/0 (socket says 172.21.15.40:59772) 2023-07-14T19:09:10.732 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:10.180+0000 7f017a59c700 1 -- 172.21.15.40:0/1232913398 learned_addr learned my addr 172.21.15.40:0/1232913398 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-07-14T19:09:10.732 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:10.180+0000 7f017a59c700 1 -- 172.21.15.40:0/1232913398 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f017c079e60 con 0x7f017c103d30 2023-07-14T19:09:10.732 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:10.181+0000 7f017a59c700 1 --2- 172.21.15.40:0/1232913398 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f017c103d30 0x7f017c104140 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f016c0098a0 tx=0x7f016c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=97827ec012974a74 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:09:10.732 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:10.181+0000 7f017959a700 1 -- 172.21.15.40:0/1232913398 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f016c004160 con 0x7f017c103d30 2023-07-14T19:09:10.733 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:10.181+0000 7f017959a700 1 -- 172.21.15.40:0/1232913398 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f016c0042c0 con 0x7f017c103d30 2023-07-14T19:09:10.733 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:10.181+0000 7f017959a700 1 -- 172.21.15.40:0/1232913398 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f016c011a00 con 0x7f017c103d30 2023-07-14T19:09:10.733 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:10.181+0000 7f0180f19700 1 -- 172.21.15.40:0/1232913398 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f017c103d30 msgr2=0x7f017c104140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:09:10.733 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:10.181+0000 7f0180f19700 1 --2- 172.21.15.40:0/1232913398 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f017c103d30 0x7f017c104140 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f016c0098a0 tx=0x7f016c009bb0 comp rx=0 tx=0).stop 2023-07-14T19:09:10.733 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:10.182+0000 7f0180f19700 1 -- 172.21.15.40:0/1232913398 shutdown_connections 2023-07-14T19:09:10.733 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:10.182+0000 7f0180f19700 1 --2- 172.21.15.40:0/1232913398 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f017c103d30 0x7f017c104140 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:09:10.733 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:10.182+0000 7f0180f19700 1 -- 172.21.15.40:0/1232913398 >> 172.21.15.40:0/1232913398 conn(0x7f017c0ff9a0 msgr2=0x7f017c101db0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:09:10.733 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:10.182+0000 7f0180f19700 1 -- 172.21.15.40:0/1232913398 shutdown_connections 2023-07-14T19:09:10.734 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:10.182+0000 7f0180f19700 1 -- 172.21.15.40:0/1232913398 wait complete. 2023-07-14T19:09:10.734 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:10.182+0000 7f0180f19700 1 Processor -- start 2023-07-14T19:09:10.734 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:10.183+0000 7f0180f19700 1 -- start start 2023-07-14T19:09:10.734 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:10.183+0000 7f0180f19700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f017c103d30 0x7f017c19b2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:09:10.734 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:10.183+0000 7f0180f19700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f017c19b7e0 con 0x7f017c103d30 2023-07-14T19:09:10.734 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:10.183+0000 7f017a59c700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f017c103d30 0x7f017c19b2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:09:10.734 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:10.183+0000 7f017a59c700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f017c103d30 0x7f017c19b2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:59774/0 (socket says 172.21.15.40:59774) 2023-07-14T19:09:10.735 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:10.183+0000 7f017a59c700 1 -- 172.21.15.40:0/2196241610 learned_addr learned my addr 172.21.15.40:0/2196241610 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-07-14T19:09:10.735 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:10.183+0000 7f017a59c700 1 -- 172.21.15.40:0/2196241610 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f016c009550 con 0x7f017c103d30 2023-07-14T19:09:10.735 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:10.183+0000 7f017a59c700 1 --2- 172.21.15.40:0/2196241610 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f017c103d30 0x7f017c19b2a0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f016c006af0 tx=0x7f016c011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:09:10.735 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:10.184+0000 7f016b7fe700 1 -- 172.21.15.40:0/2196241610 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f016c003a40 con 0x7f017c103d30 2023-07-14T19:09:10.735 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:10.184+0000 7f016b7fe700 1 -- 172.21.15.40:0/2196241610 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f016c003ba0 con 0x7f017c103d30 2023-07-14T19:09:10.736 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:10.184+0000 7f016b7fe700 1 -- 172.21.15.40:0/2196241610 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f016c01a700 con 0x7f017c103d30 2023-07-14T19:09:10.736 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:10.184+0000 7f0180f19700 1 -- 172.21.15.40:0/2196241610 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f017c19ba40 con 0x7f017c103d30 2023-07-14T19:09:10.736 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:10.184+0000 7f0180f19700 1 -- 172.21.15.40:0/2196241610 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f017c19bec0 con 0x7f017c103d30 2023-07-14T19:09:10.736 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:10.184+0000 7f016b7fe700 1 -- 172.21.15.40:0/2196241610 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f016c01ac30 con 0x7f017c103d30 2023-07-14T19:09:10.736 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:10.184+0000 7f016b7fe700 1 -- 172.21.15.40:0/2196241610 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f016c019700 con 0x7f017c103d30 2023-07-14T19:09:10.737 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:10.185+0000 7f0180f19700 1 -- 172.21.15.40:0/2196241610 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f017c194ae0 con 0x7f017c103d30 2023-07-14T19:09:10.737 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:10.187+0000 7f016b7fe700 1 -- 172.21.15.40:0/2196241610 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72446 (secure 0 0 0) 0x7f016c02ddf0 con 0x7f017c103d30 2023-07-14T19:09:10.737 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:10.320+0000 7f0180f19700 1 -- 172.21.15.40:0/2196241610 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f017c02cb30 con 0x7f017c103d30 2023-07-14T19:09:10.737 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:10.320+0000 7f016b7fe700 1 -- 172.21.15.40:0/2196241610 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1238 (secure 0 0 0) 0x7f016c024ad0 con 0x7f017c103d30 2023-07-14T19:09:10.737 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:10.321+0000 7f0180f19700 1 -- 172.21.15.40:0/2196241610 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f017c103d30 msgr2=0x7f017c19b2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:09:10.738 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:10.321+0000 7f0180f19700 1 --2- 172.21.15.40:0/2196241610 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f017c103d30 0x7f017c19b2a0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f016c006af0 tx=0x7f016c011ad0 comp rx=0 tx=0).stop 2023-07-14T19:09:10.738 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:10.321+0000 7f0180f19700 1 -- 172.21.15.40:0/2196241610 shutdown_connections 2023-07-14T19:09:10.738 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:10.321+0000 7f0180f19700 1 --2- 172.21.15.40:0/2196241610 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f017c103d30 0x7f017c19b2a0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:09:10.738 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:10.321+0000 7f0180f19700 1 -- 172.21.15.40:0/2196241610 >> 172.21.15.40:0/2196241610 conn(0x7f017c0ff9a0 msgr2=0x7f017c18f8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:09:10.739 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:10.322+0000 7f0180f19700 1 -- 172.21.15.40:0/2196241610 shutdown_connections 2023-07-14T19:09:10.739 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:10.322+0000 7f0180f19700 1 -- 172.21.15.40:0/2196241610 wait complete. 2023-07-14T19:09:10.739 INFO:teuthology.orchestra.run.smithi040.stdout:mgr not available, waiting (3/15)... 2023-07-14T19:09:11.722 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Jul 14 19:09:11 smithi040 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-mgr-x[96762]: 2023-07-14T19:09:11.455+0000 7f5cf9a5d200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-07-14T19:09:12.722 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:09:12 smithi040 ceph-mon[96518]: Activating manager daemon x 2023-07-14T19:09:12.722 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:09:12 smithi040 ceph-mon[96518]: mgrmap e2: x(active, starting, since 0.00264823s) 2023-07-14T19:09:12.722 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:09:12 smithi040 ceph-mon[96518]: from='mgr.14100 172.21.15.40:0/2833086995' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2023-07-14T19:09:12.722 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:09:12 smithi040 ceph-mon[96518]: from='mgr.14100 172.21.15.40:0/2833086995' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2023-07-14T19:09:12.722 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:09:12 smithi040 ceph-mon[96518]: from='mgr.14100 172.21.15.40:0/2833086995' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2023-07-14T19:09:12.722 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:09:12 smithi040 ceph-mon[96518]: from='mgr.14100 172.21.15.40:0/2833086995' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-07-14T19:09:12.723 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:09:12 smithi040 ceph-mon[96518]: from='mgr.14100 172.21.15.40:0/2833086995' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2023-07-14T19:09:12.723 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:09:12 smithi040 ceph-mon[96518]: Manager daemon x is now available 2023-07-14T19:09:12.723 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:09:12 smithi040 ceph-mon[96518]: from='mgr.14100 172.21.15.40:0/2833086995' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2023-07-14T19:09:12.723 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:09:12 smithi040 ceph-mon[96518]: from='mgr.14100 172.21.15.40:0/2833086995' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2023-07-14T19:09:12.723 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:09:12 smithi040 ceph-mon[96518]: from='mgr.14100 172.21.15.40:0/2833086995' entity='mgr.x' 2023-07-14T19:09:12.723 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:09:12 smithi040 ceph-mon[96518]: from='mgr.14100 172.21.15.40:0/2833086995' entity='mgr.x' 2023-07-14T19:09:12.723 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:09:12 smithi040 ceph-mon[96518]: from='mgr.14100 172.21.15.40:0/2833086995' entity='mgr.x' 2023-07-14T19:09:13.796 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:09:13 smithi040 ceph-mon[96518]: mgrmap e3: x(active, since 1.00644s) 2023-07-14T19:09:14.187 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout 2023-07-14T19:09:14.188 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout { 2023-07-14T19:09:14.188 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "fsid": "adc52f04-2279-11ee-9b33-001a4aab830c", 2023-07-14T19:09:14.189 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "health": { 2023-07-14T19:09:14.189 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2023-07-14T19:09:14.190 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "checks": {}, 2023-07-14T19:09:14.190 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "mutes": [] 2023-07-14T19:09:14.191 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout }, 2023-07-14T19:09:14.191 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2023-07-14T19:09:14.192 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "quorum": [ 2023-07-14T19:09:14.192 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout 0 2023-07-14T19:09:14.192 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout ], 2023-07-14T19:09:14.193 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2023-07-14T19:09:14.193 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "a" 2023-07-14T19:09:14.194 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout ], 2023-07-14T19:09:14.194 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "quorum_age": 12, 2023-07-14T19:09:14.194 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "monmap": { 2023-07-14T19:09:14.195 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-07-14T19:09:14.195 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "reef", 2023-07-14T19:09:14.195 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "num_mons": 1 2023-07-14T19:09:14.195 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout }, 2023-07-14T19:09:14.195 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "osdmap": { 2023-07-14T19:09:14.195 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-07-14T19:09:14.195 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2023-07-14T19:09:14.196 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2023-07-14T19:09:14.196 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2023-07-14T19:09:14.196 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2023-07-14T19:09:14.196 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2023-07-14T19:09:14.196 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2023-07-14T19:09:14.196 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout }, 2023-07-14T19:09:14.196 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "pgmap": { 2023-07-14T19:09:14.197 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2023-07-14T19:09:14.197 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2023-07-14T19:09:14.197 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2023-07-14T19:09:14.197 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2023-07-14T19:09:14.197 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2023-07-14T19:09:14.197 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2023-07-14T19:09:14.197 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2023-07-14T19:09:14.197 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2023-07-14T19:09:14.198 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout }, 2023-07-14T19:09:14.198 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "fsmap": { 2023-07-14T19:09:14.198 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-07-14T19:09:14.198 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "by_rank": [], 2023-07-14T19:09:14.198 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "up:standby": 0 2023-07-14T19:09:14.198 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout }, 2023-07-14T19:09:14.198 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "mgrmap": { 2023-07-14T19:09:14.198 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "available": true, 2023-07-14T19:09:14.199 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2023-07-14T19:09:14.199 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "modules": [ 2023-07-14T19:09:14.199 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "iostat", 2023-07-14T19:09:14.199 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "nfs", 2023-07-14T19:09:14.199 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "restful" 2023-07-14T19:09:14.199 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout ], 2023-07-14T19:09:14.199 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "services": {} 2023-07-14T19:09:14.200 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout }, 2023-07-14T19:09:14.200 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "servicemap": { 2023-07-14T19:09:14.200 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-07-14T19:09:14.200 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "modified": "2023-07-14T19:08:56.894377+0000", 2023-07-14T19:09:14.200 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "services": {} 2023-07-14T19:09:14.200 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout }, 2023-07-14T19:09:14.200 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "progress_events": {} 2023-07-14T19:09:14.200 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout } 2023-07-14T19:09:14.201 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:13.243+0000 7fb2e07c1700 1 Processor -- start 2023-07-14T19:09:14.201 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:13.244+0000 7fb2e07c1700 1 -- start start 2023-07-14T19:09:14.201 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:13.244+0000 7fb2e07c1700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb2d807be00 0x7fb2d807a300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:09:14.201 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:13.244+0000 7fb2e07c1700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2d807a840 con 0x7fb2d807be00 2023-07-14T19:09:14.201 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:13.244+0000 7fb2de55d700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb2d807be00 0x7fb2d807a300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:09:14.201 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:13.244+0000 7fb2de55d700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb2d807be00 0x7fb2d807a300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:59800/0 (socket says 172.21.15.40:59800) 2023-07-14T19:09:14.201 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:13.244+0000 7fb2de55d700 1 -- 172.21.15.40:0/2476192908 learned_addr learned my addr 172.21.15.40:0/2476192908 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-07-14T19:09:14.202 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:13.244+0000 7fb2de55d700 1 -- 172.21.15.40:0/2476192908 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2d807a980 con 0x7fb2d807be00 2023-07-14T19:09:14.202 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:13.245+0000 7fb2de55d700 1 --2- 172.21.15.40:0/2476192908 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb2d807be00 0x7fb2d807a300 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fb2d40098a0 tx=0x7fb2d4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=660d1f5d36d958ce server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:09:14.202 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:13.245+0000 7fb2dd55b700 1 -- 172.21.15.40:0/2476192908 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb2d4004160 con 0x7fb2d807be00 2023-07-14T19:09:14.202 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:13.245+0000 7fb2dd55b700 1 -- 172.21.15.40:0/2476192908 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb2d40042c0 con 0x7fb2d807be00 2023-07-14T19:09:14.202 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:13.245+0000 7fb2dd55b700 1 -- 172.21.15.40:0/2476192908 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb2d4011a00 con 0x7fb2d807be00 2023-07-14T19:09:14.202 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:13.245+0000 7fb2e07c1700 1 -- 172.21.15.40:0/2476192908 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb2d807be00 msgr2=0x7fb2d807a300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:09:14.202 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:13.245+0000 7fb2e07c1700 1 --2- 172.21.15.40:0/2476192908 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb2d807be00 0x7fb2d807a300 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fb2d40098a0 tx=0x7fb2d4009bb0 comp rx=0 tx=0).stop 2023-07-14T19:09:14.203 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:13.245+0000 7fb2e07c1700 1 -- 172.21.15.40:0/2476192908 shutdown_connections 2023-07-14T19:09:14.203 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:13.245+0000 7fb2e07c1700 1 --2- 172.21.15.40:0/2476192908 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb2d807be00 0x7fb2d807a300 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:09:14.203 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:13.245+0000 7fb2e07c1700 1 -- 172.21.15.40:0/2476192908 >> 172.21.15.40:0/2476192908 conn(0x7fb2d8100a80 msgr2=0x7fb2d8102e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:09:14.203 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:13.245+0000 7fb2e07c1700 1 -- 172.21.15.40:0/2476192908 shutdown_connections 2023-07-14T19:09:14.203 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:13.246+0000 7fb2e07c1700 1 -- 172.21.15.40:0/2476192908 wait complete. 2023-07-14T19:09:14.203 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:13.246+0000 7fb2e07c1700 1 Processor -- start 2023-07-14T19:09:14.204 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:13.246+0000 7fb2e07c1700 1 -- start start 2023-07-14T19:09:14.204 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:13.246+0000 7fb2e07c1700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb2d807be00 0x7fb2d819b3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:09:14.204 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:13.246+0000 7fb2e07c1700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2d819b8f0 con 0x7fb2d807be00 2023-07-14T19:09:14.205 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:13.246+0000 7fb2de55d700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb2d807be00 0x7fb2d819b3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:09:14.205 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:13.247+0000 7fb2de55d700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb2d807be00 0x7fb2d819b3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:59802/0 (socket says 172.21.15.40:59802) 2023-07-14T19:09:14.205 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:13.247+0000 7fb2de55d700 1 -- 172.21.15.40:0/1717085627 learned_addr learned my addr 172.21.15.40:0/1717085627 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-07-14T19:09:14.205 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:13.247+0000 7fb2de55d700 1 -- 172.21.15.40:0/1717085627 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2d4009550 con 0x7fb2d807be00 2023-07-14T19:09:14.205 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:13.247+0000 7fb2de55d700 1 --2- 172.21.15.40:0/1717085627 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb2d807be00 0x7fb2d819b3b0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fb2d4009130 tx=0x7fb2d4011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:09:14.205 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:13.247+0000 7fb2cb7fe700 1 -- 172.21.15.40:0/1717085627 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb2d4003ab0 con 0x7fb2d807be00 2023-07-14T19:09:14.205 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:13.247+0000 7fb2cb7fe700 1 -- 172.21.15.40:0/1717085627 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb2d4003c10 con 0x7fb2d807be00 2023-07-14T19:09:14.206 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:13.247+0000 7fb2cb7fe700 1 -- 172.21.15.40:0/1717085627 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb2d40247d0 con 0x7fb2d807be00 2023-07-14T19:09:14.206 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:13.247+0000 7fb2e07c1700 1 -- 172.21.15.40:0/1717085627 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb2d819baf0 con 0x7fb2d807be00 2023-07-14T19:09:14.206 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:13.247+0000 7fb2e07c1700 1 -- 172.21.15.40:0/1717085627 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb2d819bf10 con 0x7fb2d807be00 2023-07-14T19:09:14.206 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:13.248+0000 7fb2cb7fe700 1 -- 172.21.15.40:0/1717085627 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 3) v1 ==== 45366+0+0 (secure 0 0 0) 0x7fb2d401abb0 con 0x7fb2d807be00 2023-07-14T19:09:14.206 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:13.248+0000 7fb2cb7fe700 1 --2- 172.21.15.40:0/1717085627 >> [v2:172.21.15.40:6800/2569015667,v1:172.21.15.40:6801/2569015667] conn(0x7fb2c40385e0 0x7fb2c403aa90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:09:14.206 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:13.248+0000 7fb2e07c1700 1 -- 172.21.15.40:0/1717085627 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb2d8061c70 con 0x7fb2d807be00 2023-07-14T19:09:14.206 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:13.248+0000 7fb2cb7fe700 1 -- 172.21.15.40:0/1717085627 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7fb2d404c200 con 0x7fb2d807be00 2023-07-14T19:09:14.207 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:13.248+0000 7fb2ddd5c700 1 --2- 172.21.15.40:0/1717085627 >> [v2:172.21.15.40:6800/2569015667,v1:172.21.15.40:6801/2569015667] conn(0x7fb2c40385e0 0x7fb2c403aa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:09:14.207 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:13.249+0000 7fb2ddd5c700 1 --2- 172.21.15.40:0/1717085627 >> [v2:172.21.15.40:6800/2569015667,v1:172.21.15.40:6801/2569015667] conn(0x7fb2c40385e0 0x7fb2c403aa90 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fb2cc006fd0 tx=0x7fb2cc006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:09:14.207 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:13.254+0000 7fb2cb7fe700 1 -- 172.21.15.40:0/1717085627 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fb2d402d3f0 con 0x7fb2d807be00 2023-07-14T19:09:14.207 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:13.768+0000 7fb2e07c1700 1 -- 172.21.15.40:0/1717085627 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7fb2d819e950 con 0x7fb2d807be00 2023-07-14T19:09:14.207 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:13.768+0000 7fb2cb7fe700 1 -- 172.21.15.40:0/1717085627 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1238 (secure 0 0 0) 0x7fb2d4018350 con 0x7fb2d807be00 2023-07-14T19:09:14.207 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:13.770+0000 7fb2e07c1700 1 -- 172.21.15.40:0/1717085627 >> [v2:172.21.15.40:6800/2569015667,v1:172.21.15.40:6801/2569015667] conn(0x7fb2c40385e0 msgr2=0x7fb2c403aa90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:09:14.207 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:13.770+0000 7fb2e07c1700 1 --2- 172.21.15.40:0/1717085627 >> [v2:172.21.15.40:6800/2569015667,v1:172.21.15.40:6801/2569015667] conn(0x7fb2c40385e0 0x7fb2c403aa90 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fb2cc006fd0 tx=0x7fb2cc006e40 comp rx=0 tx=0).stop 2023-07-14T19:09:14.208 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:13.770+0000 7fb2e07c1700 1 -- 172.21.15.40:0/1717085627 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb2d807be00 msgr2=0x7fb2d819b3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:09:14.208 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:13.770+0000 7fb2e07c1700 1 --2- 172.21.15.40:0/1717085627 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb2d807be00 0x7fb2d819b3b0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fb2d4009130 tx=0x7fb2d4011ad0 comp rx=0 tx=0).stop 2023-07-14T19:09:14.208 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:13.770+0000 7fb2e07c1700 1 -- 172.21.15.40:0/1717085627 shutdown_connections 2023-07-14T19:09:14.208 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:13.770+0000 7fb2e07c1700 1 --2- 172.21.15.40:0/1717085627 >> [v2:172.21.15.40:6800/2569015667,v1:172.21.15.40:6801/2569015667] conn(0x7fb2c40385e0 0x7fb2c403aa90 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:09:14.208 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:13.770+0000 7fb2e07c1700 1 --2- 172.21.15.40:0/1717085627 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb2d807be00 0x7fb2d819b3b0 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:09:14.208 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:13.770+0000 7fb2e07c1700 1 -- 172.21.15.40:0/1717085627 >> 172.21.15.40:0/1717085627 conn(0x7fb2d8100a80 msgr2=0x7fb2d8102190 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:09:14.208 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:13.770+0000 7fb2e07c1700 1 -- 172.21.15.40:0/1717085627 shutdown_connections 2023-07-14T19:09:14.208 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:13.770+0000 7fb2e07c1700 1 -- 172.21.15.40:0/1717085627 wait complete. 2023-07-14T19:09:14.209 INFO:teuthology.orchestra.run.smithi040.stdout:mgr is available 2023-07-14T19:09:14.721 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:09:14 smithi040 ceph-mon[96518]: from='client.? 172.21.15.40:0/1717085627' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2023-07-14T19:09:15.421 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout 2023-07-14T19:09:15.421 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout [global] 2023-07-14T19:09:15.421 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout fsid = adc52f04-2279-11ee-9b33-001a4aab830c 2023-07-14T19:09:15.421 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2023-07-14T19:09:15.421 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2023-07-14T19:09:15.422 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2023-07-14T19:09:15.422 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2023-07-14T19:09:15.422 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout 2023-07-14T19:09:15.422 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout [mgr] 2023-07-14T19:09:15.422 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2023-07-14T19:09:15.422 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout 2023-07-14T19:09:15.422 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout [osd] 2023-07-14T19:09:15.423 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2023-07-14T19:09:15.423 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2023-07-14T19:09:15.423 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2023-07-14T19:09:15.423 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2023-07-14T19:09:15.423 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2023-07-14T19:09:15.423 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:14.714+0000 7f627deaa700 1 Processor -- start 2023-07-14T19:09:15.423 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:14.714+0000 7f627deaa700 1 -- start start 2023-07-14T19:09:15.423 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:14.714+0000 7f627deaa700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6278079cc0 0x7f627810a3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:09:15.424 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:14.714+0000 7f627deaa700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6278105da0 con 0x7f6278079cc0 2023-07-14T19:09:15.424 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:14.714+0000 7f62777fe700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6278079cc0 0x7f627810a3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:09:15.424 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:14.714+0000 7f62777fe700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6278079cc0 0x7f627810a3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:59806/0 (socket says 172.21.15.40:59806) 2023-07-14T19:09:15.424 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:14.714+0000 7f62777fe700 1 -- 172.21.15.40:0/3566740005 learned_addr learned my addr 172.21.15.40:0/3566740005 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-07-14T19:09:15.424 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:14.714+0000 7f62777fe700 1 -- 172.21.15.40:0/3566740005 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f627810a8f0 con 0x7f6278079cc0 2023-07-14T19:09:15.424 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:14.715+0000 7f62777fe700 1 --2- 172.21.15.40:0/3566740005 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6278079cc0 0x7f627810a3b0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f62680098a0 tx=0x7f6268009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2a838f2a1c4a8f0c server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:09:15.424 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:14.715+0000 7f62767fc700 1 -- 172.21.15.40:0/3566740005 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6268004160 con 0x7f6278079cc0 2023-07-14T19:09:15.425 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:14.715+0000 7f62767fc700 1 -- 172.21.15.40:0/3566740005 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f62680042c0 con 0x7f6278079cc0 2023-07-14T19:09:15.425 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:14.715+0000 7f62767fc700 1 -- 172.21.15.40:0/3566740005 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6268011a00 con 0x7f6278079cc0 2023-07-14T19:09:15.425 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:14.715+0000 7f627deaa700 1 -- 172.21.15.40:0/3566740005 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6278079cc0 msgr2=0x7f627810a3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:09:15.425 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:14.715+0000 7f627deaa700 1 --2- 172.21.15.40:0/3566740005 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6278079cc0 0x7f627810a3b0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f62680098a0 tx=0x7f6268009bb0 comp rx=0 tx=0).stop 2023-07-14T19:09:15.425 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:14.715+0000 7f627deaa700 1 -- 172.21.15.40:0/3566740005 shutdown_connections 2023-07-14T19:09:15.425 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:14.715+0000 7f627deaa700 1 --2- 172.21.15.40:0/3566740005 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6278079cc0 0x7f627810a3b0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:09:15.425 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:14.715+0000 7f627deaa700 1 -- 172.21.15.40:0/3566740005 >> 172.21.15.40:0/3566740005 conn(0x7f62780ff9a0 msgr2=0x7f6278101db0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:09:15.425 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:14.716+0000 7f627deaa700 1 -- 172.21.15.40:0/3566740005 shutdown_connections 2023-07-14T19:09:15.426 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:14.716+0000 7f627deaa700 1 -- 172.21.15.40:0/3566740005 wait complete. 2023-07-14T19:09:15.426 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:14.716+0000 7f627deaa700 1 Processor -- start 2023-07-14T19:09:15.426 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:14.716+0000 7f627deaa700 1 -- start start 2023-07-14T19:09:15.426 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:14.716+0000 7f627deaa700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6278079cc0 0x7f627819b310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:09:15.426 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:14.716+0000 7f627deaa700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f627819b850 con 0x7f6278079cc0 2023-07-14T19:09:15.426 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:14.717+0000 7f62777fe700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6278079cc0 0x7f627819b310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:09:15.426 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:14.717+0000 7f62777fe700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6278079cc0 0x7f627819b310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:59808/0 (socket says 172.21.15.40:59808) 2023-07-14T19:09:15.427 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:14.717+0000 7f62777fe700 1 -- 172.21.15.40:0/2232970840 learned_addr learned my addr 172.21.15.40:0/2232970840 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-07-14T19:09:15.427 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:14.717+0000 7f62777fe700 1 -- 172.21.15.40:0/2232970840 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6268009550 con 0x7f6278079cc0 2023-07-14T19:09:15.427 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:14.717+0000 7f62777fe700 1 --2- 172.21.15.40:0/2232970840 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6278079cc0 0x7f627819b310 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f6268009130 tx=0x7f6268011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:09:15.427 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:14.717+0000 7f6274ff9700 1 -- 172.21.15.40:0/2232970840 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6268003b70 con 0x7f6278079cc0 2023-07-14T19:09:15.427 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:14.717+0000 7f6274ff9700 1 -- 172.21.15.40:0/2232970840 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6268003cd0 con 0x7f6278079cc0 2023-07-14T19:09:15.427 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:14.717+0000 7f627deaa700 1 -- 172.21.15.40:0/2232970840 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f627819ba50 con 0x7f6278079cc0 2023-07-14T19:09:15.427 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:14.717+0000 7f6274ff9700 1 -- 172.21.15.40:0/2232970840 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f62680197e0 con 0x7f6278079cc0 2023-07-14T19:09:15.427 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:14.717+0000 7f627deaa700 1 -- 172.21.15.40:0/2232970840 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f627819be70 con 0x7f6278079cc0 2023-07-14T19:09:15.428 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:14.718+0000 7f6274ff9700 1 -- 172.21.15.40:0/2232970840 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 4) v1 ==== 45472+0+0 (secure 0 0 0) 0x7f626801a430 con 0x7f6278079cc0 2023-07-14T19:09:15.428 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:14.718+0000 7f6274ff9700 1 --2- 172.21.15.40:0/2232970840 >> [v2:172.21.15.40:6800/2569015667,v1:172.21.15.40:6801/2569015667] conn(0x7f6260038710 0x7f626003abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:09:15.428 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:14.718+0000 7f6274ff9700 1 -- 172.21.15.40:0/2232970840 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f626804c870 con 0x7f6278079cc0 2023-07-14T19:09:15.428 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:14.718+0000 7f627deaa700 1 -- 172.21.15.40:0/2232970840 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6278061c70 con 0x7f6278079cc0 2023-07-14T19:09:15.428 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:14.719+0000 7f6276ffd700 1 --2- 172.21.15.40:0/2232970840 >> [v2:172.21.15.40:6800/2569015667,v1:172.21.15.40:6801/2569015667] conn(0x7f6260038710 0x7f626003abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:09:15.428 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:14.719+0000 7f6276ffd700 1 --2- 172.21.15.40:0/2232970840 >> [v2:172.21.15.40:6800/2569015667,v1:172.21.15.40:6801/2569015667] conn(0x7f6260038710 0x7f626003abc0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f626c006fd0 tx=0x7f626c006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:09:15.428 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:14.723+0000 7f6274ff9700 1 -- 172.21.15.40:0/2232970840 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f626802dd60 con 0x7f6278079cc0 2023-07-14T19:09:15.429 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:15.078+0000 7f627deaa700 1 -- 172.21.15.40:0/2232970840 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) v1 -- 0x7f62781a0080 con 0x7f6278079cc0 2023-07-14T19:09:15.429 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:15.079+0000 7f6274ff9700 1 -- 172.21.15.40:0/2232970840 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v3) v1 ==== 70+0+373 (secure 0 0 0) 0x7f626802d350 con 0x7f6278079cc0 2023-07-14T19:09:15.429 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:15.081+0000 7f627deaa700 1 -- 172.21.15.40:0/2232970840 >> [v2:172.21.15.40:6800/2569015667,v1:172.21.15.40:6801/2569015667] conn(0x7f6260038710 msgr2=0x7f626003abc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:09:15.429 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:15.081+0000 7f627deaa700 1 --2- 172.21.15.40:0/2232970840 >> [v2:172.21.15.40:6800/2569015667,v1:172.21.15.40:6801/2569015667] conn(0x7f6260038710 0x7f626003abc0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f626c006fd0 tx=0x7f626c006e40 comp rx=0 tx=0).stop 2023-07-14T19:09:15.429 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:15.081+0000 7f627deaa700 1 -- 172.21.15.40:0/2232970840 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6278079cc0 msgr2=0x7f627819b310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:09:15.429 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:15.081+0000 7f627deaa700 1 --2- 172.21.15.40:0/2232970840 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6278079cc0 0x7f627819b310 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f6268009130 tx=0x7f6268011ad0 comp rx=0 tx=0).stop 2023-07-14T19:09:15.429 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:15.081+0000 7f627deaa700 1 -- 172.21.15.40:0/2232970840 shutdown_connections 2023-07-14T19:09:15.429 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:15.081+0000 7f627deaa700 1 --2- 172.21.15.40:0/2232970840 >> [v2:172.21.15.40:6800/2569015667,v1:172.21.15.40:6801/2569015667] conn(0x7f6260038710 0x7f626003abc0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:09:15.430 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:15.081+0000 7f627deaa700 1 --2- 172.21.15.40:0/2232970840 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6278079cc0 0x7f627819b310 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:09:15.430 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:15.081+0000 7f627deaa700 1 -- 172.21.15.40:0/2232970840 >> 172.21.15.40:0/2232970840 conn(0x7f62780ff9a0 msgr2=0x7f6278100780 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:09:15.430 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:15.081+0000 7f627deaa700 1 -- 172.21.15.40:0/2232970840 shutdown_connections 2023-07-14T19:09:15.430 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:15.081+0000 7f627deaa700 1 -- 172.21.15.40:0/2232970840 wait complete. 2023-07-14T19:09:15.430 INFO:teuthology.orchestra.run.smithi040.stdout:Enabling cephadm module... 2023-07-14T19:09:15.721 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:09:15 smithi040 ceph-mon[96518]: mgrmap e4: x(active, since 3s) 2023-07-14T19:09:15.722 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:09:15 smithi040 ceph-mon[96518]: from='client.? 172.21.15.40:0/2232970840' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2023-07-14T19:09:16.762 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:09:16 smithi040 ceph-mon[96518]: from='client.? 172.21.15.40:0/3845873168' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2023-07-14T19:09:16.762 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Jul 14 19:09:16 smithi040 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-mgr-x[96762]: ignoring --setuser ceph since I am not root 2023-07-14T19:09:16.763 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Jul 14 19:09:16 smithi040 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-mgr-x[96762]: ignoring --setgroup ceph since I am not root 2023-07-14T19:09:16.763 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Jul 14 19:09:16 smithi040 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-mgr-x[96762]: 2023-07-14T19:09:16.664+0000 7fc08d126200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-07-14T19:09:16.966 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:15.930+0000 7f34e5568700 1 Processor -- start 2023-07-14T19:09:16.966 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:15.930+0000 7f34e5568700 1 -- start start 2023-07-14T19:09:16.966 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:15.930+0000 7f34e5568700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f34e0079cc0 0x7f34e010a3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:09:16.966 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:15.930+0000 7f34e5568700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34e0105da0 con 0x7f34e0079cc0 2023-07-14T19:09:16.967 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:15.931+0000 7f34deffd700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f34e0079cc0 0x7f34e010a3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:09:16.967 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:15.931+0000 7f34deffd700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f34e0079cc0 0x7f34e010a3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:59812/0 (socket says 172.21.15.40:59812) 2023-07-14T19:09:16.967 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:15.931+0000 7f34deffd700 1 -- 172.21.15.40:0/990580403 learned_addr learned my addr 172.21.15.40:0/990580403 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-07-14T19:09:16.967 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:15.931+0000 7f34deffd700 1 -- 172.21.15.40:0/990580403 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34e010a8f0 con 0x7f34e0079cc0 2023-07-14T19:09:16.967 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:15.931+0000 7f34deffd700 1 --2- 172.21.15.40:0/990580403 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f34e0079cc0 0x7f34e010a3b0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f34d00098a0 tx=0x7f34d0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6065767bc8f71d52 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:09:16.968 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:15.931+0000 7f34ddffb700 1 -- 172.21.15.40:0/990580403 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f34d0004160 con 0x7f34e0079cc0 2023-07-14T19:09:16.968 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:15.931+0000 7f34ddffb700 1 -- 172.21.15.40:0/990580403 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f34d00042c0 con 0x7f34e0079cc0 2023-07-14T19:09:16.968 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:15.931+0000 7f34ddffb700 1 -- 172.21.15.40:0/990580403 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f34d0011a00 con 0x7f34e0079cc0 2023-07-14T19:09:16.968 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:15.932+0000 7f34e5568700 1 -- 172.21.15.40:0/990580403 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f34e0079cc0 msgr2=0x7f34e010a3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:09:16.968 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:15.932+0000 7f34e5568700 1 --2- 172.21.15.40:0/990580403 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f34e0079cc0 0x7f34e010a3b0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f34d00098a0 tx=0x7f34d0009bb0 comp rx=0 tx=0).stop 2023-07-14T19:09:16.968 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:15.932+0000 7f34e5568700 1 -- 172.21.15.40:0/990580403 shutdown_connections 2023-07-14T19:09:16.968 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:15.932+0000 7f34e5568700 1 --2- 172.21.15.40:0/990580403 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f34e0079cc0 0x7f34e010a3b0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:09:16.969 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:15.932+0000 7f34e5568700 1 -- 172.21.15.40:0/990580403 >> 172.21.15.40:0/990580403 conn(0x7f34e00ff9a0 msgr2=0x7f34e0101db0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:09:16.969 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:15.932+0000 7f34e5568700 1 -- 172.21.15.40:0/990580403 shutdown_connections 2023-07-14T19:09:16.969 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:15.932+0000 7f34e5568700 1 -- 172.21.15.40:0/990580403 wait complete. 2023-07-14T19:09:16.969 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:15.933+0000 7f34e5568700 1 Processor -- start 2023-07-14T19:09:16.969 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:15.933+0000 7f34e5568700 1 -- start start 2023-07-14T19:09:16.969 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:15.933+0000 7f34e5568700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f34e0079cc0 0x7f34e019b340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:09:16.969 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:15.933+0000 7f34e5568700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34e019b880 con 0x7f34e0079cc0 2023-07-14T19:09:16.970 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:15.933+0000 7f34deffd700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f34e0079cc0 0x7f34e019b340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:09:16.970 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:15.933+0000 7f34deffd700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f34e0079cc0 0x7f34e019b340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:59814/0 (socket says 172.21.15.40:59814) 2023-07-14T19:09:16.970 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:15.933+0000 7f34deffd700 1 -- 172.21.15.40:0/3845873168 learned_addr learned my addr 172.21.15.40:0/3845873168 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-07-14T19:09:16.970 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:15.934+0000 7f34deffd700 1 -- 172.21.15.40:0/3845873168 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34d0009550 con 0x7f34e0079cc0 2023-07-14T19:09:16.970 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:15.934+0000 7f34deffd700 1 --2- 172.21.15.40:0/3845873168 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f34e0079cc0 0x7f34e019b340 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f34d0009130 tx=0x7f34d0011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:09:16.970 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:15.934+0000 7f34c7fff700 1 -- 172.21.15.40:0/3845873168 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f34d0003b70 con 0x7f34e0079cc0 2023-07-14T19:09:16.970 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:15.934+0000 7f34c7fff700 1 -- 172.21.15.40:0/3845873168 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f34d0003cd0 con 0x7f34e0079cc0 2023-07-14T19:09:16.970 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:15.934+0000 7f34e5568700 1 -- 172.21.15.40:0/3845873168 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f34e019ba80 con 0x7f34e0079cc0 2023-07-14T19:09:16.971 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:15.934+0000 7f34c7fff700 1 -- 172.21.15.40:0/3845873168 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f34d00197e0 con 0x7f34e0079cc0 2023-07-14T19:09:16.971 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:15.934+0000 7f34e5568700 1 -- 172.21.15.40:0/3845873168 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f34e019bea0 con 0x7f34e0079cc0 2023-07-14T19:09:16.971 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:15.935+0000 7f34c7fff700 1 -- 172.21.15.40:0/3845873168 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 4) v1 ==== 45472+0+0 (secure 0 0 0) 0x7f34d0019a20 con 0x7f34e0079cc0 2023-07-14T19:09:16.971 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:15.935+0000 7f34c7fff700 1 --2- 172.21.15.40:0/3845873168 >> [v2:172.21.15.40:6800/2569015667,v1:172.21.15.40:6801/2569015667] conn(0x7f34c80386c0 0x7f34c803ab70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:09:16.971 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:15.935+0000 7f34c7fff700 1 -- 172.21.15.40:0/3845873168 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f34d004c870 con 0x7f34e0079cc0 2023-07-14T19:09:16.971 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:15.935+0000 7f34e5568700 1 -- 172.21.15.40:0/3845873168 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f34e0061c70 con 0x7f34e0079cc0 2023-07-14T19:09:16.971 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:15.935+0000 7f34de7fc700 1 --2- 172.21.15.40:0/3845873168 >> [v2:172.21.15.40:6800/2569015667,v1:172.21.15.40:6801/2569015667] conn(0x7f34c80386c0 0x7f34c803ab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:09:16.972 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:15.936+0000 7f34de7fc700 1 --2- 172.21.15.40:0/3845873168 >> [v2:172.21.15.40:6800/2569015667,v1:172.21.15.40:6801/2569015667] conn(0x7f34c80386c0 0x7f34c803ab70 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f34d4006fd0 tx=0x7f34d4006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:09:16.972 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:15.940+0000 7f34c7fff700 1 -- 172.21.15.40:0/3845873168 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f34d0018db0 con 0x7f34e0079cc0 2023-07-14T19:09:16.972 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:16.404+0000 7f34e5568700 1 -- 172.21.15.40:0/3845873168 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "cephadm"} v 0) v1 -- 0x7f34e019e9d0 con 0x7f34e0079cc0 2023-07-14T19:09:16.972 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:16.478+0000 7f34c7fff700 1 -- 172.21.15.40:0/3845873168 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mgrmap(e 5) v1 ==== 45483+0+0 (secure 0 0 0) 0x7f34d002bcd0 con 0x7f34e0079cc0 2023-07-14T19:09:16.972 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:16.479+0000 7f34c7fff700 1 -- 172.21.15.40:0/3845873168 <== mon.0 v2:172.21.15.40:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "cephadm"}]=0 v5) v1 ==== 86+0+0 (secure 0 0 0) 0x7f34d0023c00 con 0x7f34e0079cc0 2023-07-14T19:09:16.972 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:16.482+0000 7f34e5568700 1 -- 172.21.15.40:0/3845873168 >> [v2:172.21.15.40:6800/2569015667,v1:172.21.15.40:6801/2569015667] conn(0x7f34c80386c0 msgr2=0x7f34c803ab70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:09:16.972 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:16.482+0000 7f34e5568700 1 --2- 172.21.15.40:0/3845873168 >> [v2:172.21.15.40:6800/2569015667,v1:172.21.15.40:6801/2569015667] conn(0x7f34c80386c0 0x7f34c803ab70 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f34d4006fd0 tx=0x7f34d4006e40 comp rx=0 tx=0).stop 2023-07-14T19:09:16.973 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:16.482+0000 7f34e5568700 1 -- 172.21.15.40:0/3845873168 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f34e0079cc0 msgr2=0x7f34e019b340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:09:16.973 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:16.482+0000 7f34e5568700 1 --2- 172.21.15.40:0/3845873168 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f34e0079cc0 0x7f34e019b340 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f34d0009130 tx=0x7f34d0011ad0 comp rx=0 tx=0).stop 2023-07-14T19:09:16.973 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:16.482+0000 7f34e5568700 1 -- 172.21.15.40:0/3845873168 shutdown_connections 2023-07-14T19:09:16.973 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:16.482+0000 7f34e5568700 1 --2- 172.21.15.40:0/3845873168 >> [v2:172.21.15.40:6800/2569015667,v1:172.21.15.40:6801/2569015667] conn(0x7f34c80386c0 0x7f34c803ab70 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:09:16.973 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:16.482+0000 7f34e5568700 1 --2- 172.21.15.40:0/3845873168 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f34e0079cc0 0x7f34e019b340 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:09:16.973 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:16.482+0000 7f34e5568700 1 -- 172.21.15.40:0/3845873168 >> 172.21.15.40:0/3845873168 conn(0x7f34e00ff9a0 msgr2=0x7f34e0100ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:09:16.973 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:16.482+0000 7f34e5568700 1 -- 172.21.15.40:0/3845873168 shutdown_connections 2023-07-14T19:09:16.974 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:16.482+0000 7f34e5568700 1 -- 172.21.15.40:0/3845873168 wait complete. 2023-07-14T19:09:17.199 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Jul 14 19:09:16 smithi040 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-mgr-x[96762]: 2023-07-14T19:09:16.759+0000 7fc08d126200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-07-14T19:09:17.199 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Jul 14 19:09:16 smithi040 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-mgr-x[96762]: 2023-07-14T19:09:16.868+0000 7fc08d126200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-07-14T19:09:17.200 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Jul 14 19:09:16 smithi040 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-mgr-x[96762]: 2023-07-14T19:09:16.981+0000 7fc08d126200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-07-14T19:09:17.472 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Jul 14 19:09:17 smithi040 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-mgr-x[96762]: 2023-07-14T19:09:17.196+0000 7fc08d126200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-07-14T19:09:17.863 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:09:17 smithi040 ceph-mon[96518]: from='client.? 172.21.15.40:0/3845873168' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2023-07-14T19:09:17.864 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:09:17 smithi040 ceph-mon[96518]: mgrmap e5: x(active, since 5s) 2023-07-14T19:09:17.864 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Jul 14 19:09:17 smithi040 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-mgr-x[96762]: 2023-07-14T19:09:17.600+0000 7fc08d126200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-07-14T19:09:17.864 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Jul 14 19:09:17 smithi040 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-mgr-x[96762]: 2023-07-14T19:09:17.709+0000 7fc08d126200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-07-14T19:09:18.177 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Jul 14 19:09:17 smithi040 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-mgr-x[96762]: 2023-07-14T19:09:17.860+0000 7fc08d126200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-07-14T19:09:18.178 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Jul 14 19:09:17 smithi040 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-mgr-x[96762]: 2023-07-14T19:09:17.975+0000 7fc08d126200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-07-14T19:09:18.415 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout { 2023-07-14T19:09:18.416 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "epoch": 5, 2023-07-14T19:09:18.416 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "available": true, 2023-07-14T19:09:18.416 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "active_name": "x", 2023-07-14T19:09:18.416 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "num_standby": 0 2023-07-14T19:09:18.416 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout } 2023-07-14T19:09:18.417 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:17.523+0000 7f7ed48f6700 1 Processor -- start 2023-07-14T19:09:18.417 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:17.523+0000 7f7ed48f6700 1 -- start start 2023-07-14T19:09:18.417 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:17.524+0000 7f7ed48f6700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7ed01042f0 0x7f7ed0104700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:09:18.417 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:17.524+0000 7f7ed48f6700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ed0104c40 con 0x7f7ed01042f0 2023-07-14T19:09:18.417 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:17.524+0000 7f7ece59c700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7ed01042f0 0x7f7ed0104700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:09:18.417 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:17.524+0000 7f7ece59c700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7ed01042f0 0x7f7ed0104700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:59828/0 (socket says 172.21.15.40:59828) 2023-07-14T19:09:18.417 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:17.524+0000 7f7ece59c700 1 -- 172.21.15.40:0/2737555189 learned_addr learned my addr 172.21.15.40:0/2737555189 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-07-14T19:09:18.418 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:17.524+0000 7f7ece59c700 1 -- 172.21.15.40:0/2737555189 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ed0104d80 con 0x7f7ed01042f0 2023-07-14T19:09:18.418 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:17.524+0000 7f7ece59c700 1 --2- 172.21.15.40:0/2737555189 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7ed01042f0 0x7f7ed0104700 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f7ec00098a0 tx=0x7f7ec0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=420ab57b11f7b0e5 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:09:18.418 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:17.525+0000 7f7ecd59a700 1 -- 172.21.15.40:0/2737555189 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7ec0004160 con 0x7f7ed01042f0 2023-07-14T19:09:18.418 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:17.525+0000 7f7ecd59a700 1 -- 172.21.15.40:0/2737555189 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7ec00042c0 con 0x7f7ed01042f0 2023-07-14T19:09:18.418 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:17.525+0000 7f7ecd59a700 1 -- 172.21.15.40:0/2737555189 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7ec0011a00 con 0x7f7ed01042f0 2023-07-14T19:09:18.418 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:17.525+0000 7f7ed48f6700 1 -- 172.21.15.40:0/2737555189 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7ed01042f0 msgr2=0x7f7ed0104700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:09:18.419 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:17.525+0000 7f7ed48f6700 1 --2- 172.21.15.40:0/2737555189 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7ed01042f0 0x7f7ed0104700 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f7ec00098a0 tx=0x7f7ec0009bb0 comp rx=0 tx=0).stop 2023-07-14T19:09:18.419 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:17.525+0000 7f7ed48f6700 1 -- 172.21.15.40:0/2737555189 shutdown_connections 2023-07-14T19:09:18.419 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:17.525+0000 7f7ed48f6700 1 --2- 172.21.15.40:0/2737555189 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7ed01042f0 0x7f7ed0104700 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:09:18.419 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:17.525+0000 7f7ed48f6700 1 -- 172.21.15.40:0/2737555189 >> 172.21.15.40:0/2737555189 conn(0x7f7ed00ff9a0 msgr2=0x7f7ed0101db0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:09:18.419 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:17.526+0000 7f7ed48f6700 1 -- 172.21.15.40:0/2737555189 shutdown_connections 2023-07-14T19:09:18.419 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:17.526+0000 7f7ed48f6700 1 -- 172.21.15.40:0/2737555189 wait complete. 2023-07-14T19:09:18.419 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:17.526+0000 7f7ed48f6700 1 Processor -- start 2023-07-14T19:09:18.419 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:17.527+0000 7f7ed48f6700 1 -- start start 2023-07-14T19:09:18.420 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:17.527+0000 7f7ed48f6700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7ed01042f0 0x7f7ed0197fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:09:18.420 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:17.527+0000 7f7ed48f6700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ed0198500 con 0x7f7ed01042f0 2023-07-14T19:09:18.420 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:17.527+0000 7f7ece59c700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7ed01042f0 0x7f7ed0197fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:09:18.420 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:17.527+0000 7f7ece59c700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7ed01042f0 0x7f7ed0197fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:59830/0 (socket says 172.21.15.40:59830) 2023-07-14T19:09:18.420 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:17.527+0000 7f7ece59c700 1 -- 172.21.15.40:0/118061912 learned_addr learned my addr 172.21.15.40:0/118061912 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-07-14T19:09:18.420 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:17.527+0000 7f7ece59c700 1 -- 172.21.15.40:0/118061912 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ec0009550 con 0x7f7ed01042f0 2023-07-14T19:09:18.420 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:17.528+0000 7f7ece59c700 1 --2- 172.21.15.40:0/118061912 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7ed01042f0 0x7f7ed0197fc0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f7ec0006af0 tx=0x7f7ec0011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:09:18.421 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:17.528+0000 7f7ebf7fe700 1 -- 172.21.15.40:0/118061912 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7ec0003a40 con 0x7f7ed01042f0 2023-07-14T19:09:18.421 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:17.528+0000 7f7ebf7fe700 1 -- 172.21.15.40:0/118061912 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7ec0003ba0 con 0x7f7ed01042f0 2023-07-14T19:09:18.421 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:17.528+0000 7f7ebf7fe700 1 -- 172.21.15.40:0/118061912 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7ec001a6e0 con 0x7f7ed01042f0 2023-07-14T19:09:18.421 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:17.528+0000 7f7ed48f6700 1 -- 172.21.15.40:0/118061912 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7ed0198700 con 0x7f7ed01042f0 2023-07-14T19:09:18.421 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:17.528+0000 7f7ed48f6700 1 -- 172.21.15.40:0/118061912 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ed0198b20 con 0x7f7ed01042f0 2023-07-14T19:09:18.421 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:17.529+0000 7f7ed48f6700 1 -- 172.21.15.40:0/118061912 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7ed0061c70 con 0x7f7ed01042f0 2023-07-14T19:09:18.422 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:17.529+0000 7f7ebf7fe700 1 -- 172.21.15.40:0/118061912 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 45483+0+0 (secure 0 0 0) 0x7f7ec0024a60 con 0x7f7ed01042f0 2023-07-14T19:09:18.422 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:17.529+0000 7f7ebf7fe700 1 --2- 172.21.15.40:0/118061912 >> [v2:172.21.15.40:6800/2569015667,v1:172.21.15.40:6801/2569015667] conn(0x7f7eb8038760 0x7f7eb803ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:09:18.422 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:17.529+0000 7f7ebf7fe700 1 -- 172.21.15.40:0/118061912 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f7ec004c6f0 con 0x7f7ed01042f0 2023-07-14T19:09:18.423 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:17.529+0000 7f7ecdd9b700 1 -- 172.21.15.40:0/118061912 >> [v2:172.21.15.40:6800/2569015667,v1:172.21.15.40:6801/2569015667] conn(0x7f7eb8038760 msgr2=0x7f7eb803ac10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.40:6800/2569015667 2023-07-14T19:09:18.423 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:17.530+0000 7f7ecdd9b700 1 --2- 172.21.15.40:0/118061912 >> [v2:172.21.15.40:6800/2569015667,v1:172.21.15.40:6801/2569015667] conn(0x7f7eb8038760 0x7f7eb803ac10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-07-14T19:09:18.423 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:17.534+0000 7f7ebf7fe700 1 -- 172.21.15.40:0/118061912 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f7ec0024430 con 0x7f7ed01042f0 2023-07-14T19:09:18.423 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:17.731+0000 7f7ecdd9b700 1 -- 172.21.15.40:0/118061912 >> [v2:172.21.15.40:6800/2569015667,v1:172.21.15.40:6801/2569015667] conn(0x7f7eb8038760 msgr2=0x7f7eb803ac10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.40:6800/2569015667 2023-07-14T19:09:18.424 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:17.731+0000 7f7ecdd9b700 1 --2- 172.21.15.40:0/118061912 >> [v2:172.21.15.40:6800/2569015667,v1:172.21.15.40:6801/2569015667] conn(0x7f7eb8038760 0x7f7eb803ac10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-07-14T19:09:18.424 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:17.999+0000 7f7ed48f6700 1 -- 172.21.15.40:0/118061912 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7f7ed019b980 con 0x7f7ed01042f0 2023-07-14T19:09:18.424 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:17.999+0000 7f7ebf7fe700 1 -- 172.21.15.40:0/118061912 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v5) v1 ==== 56+0+88 (secure 0 0 0) 0x7f7ec0018350 con 0x7f7ed01042f0 2023-07-14T19:09:18.425 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:18.002+0000 7f7ed48f6700 1 -- 172.21.15.40:0/118061912 >> [v2:172.21.15.40:6800/2569015667,v1:172.21.15.40:6801/2569015667] conn(0x7f7eb8038760 msgr2=0x7f7eb803ac10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-07-14T19:09:18.425 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:18.002+0000 7f7ed48f6700 1 --2- 172.21.15.40:0/118061912 >> [v2:172.21.15.40:6800/2569015667,v1:172.21.15.40:6801/2569015667] conn(0x7f7eb8038760 0x7f7eb803ac10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:09:18.425 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:18.002+0000 7f7ed48f6700 1 -- 172.21.15.40:0/118061912 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7ed01042f0 msgr2=0x7f7ed0197fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:09:18.425 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:18.002+0000 7f7ed48f6700 1 --2- 172.21.15.40:0/118061912 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7ed01042f0 0x7f7ed0197fc0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f7ec0006af0 tx=0x7f7ec0011ad0 comp rx=0 tx=0).stop 2023-07-14T19:09:18.426 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:18.002+0000 7f7ed48f6700 1 -- 172.21.15.40:0/118061912 shutdown_connections 2023-07-14T19:09:18.426 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:18.002+0000 7f7ed48f6700 1 --2- 172.21.15.40:0/118061912 >> [v2:172.21.15.40:6800/2569015667,v1:172.21.15.40:6801/2569015667] conn(0x7f7eb8038760 0x7f7eb803ac10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:09:18.426 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:18.002+0000 7f7ed48f6700 1 --2- 172.21.15.40:0/118061912 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7ed01042f0 0x7f7ed0197fc0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:09:18.426 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:18.002+0000 7f7ed48f6700 1 -- 172.21.15.40:0/118061912 >> 172.21.15.40:0/118061912 conn(0x7f7ed00ff9a0 msgr2=0x7f7ed0100780 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:09:18.427 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:18.002+0000 7f7ed48f6700 1 -- 172.21.15.40:0/118061912 shutdown_connections 2023-07-14T19:09:18.427 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:18.002+0000 7f7ed48f6700 1 -- 172.21.15.40:0/118061912 wait complete. 2023-07-14T19:09:18.427 INFO:teuthology.orchestra.run.smithi040.stdout:Waiting for the mgr to restart... 2023-07-14T19:09:18.427 INFO:teuthology.orchestra.run.smithi040.stdout:Waiting for mgr epoch 5... 2023-07-14T19:09:18.971 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:09:18 smithi040 ceph-mon[96518]: from='client.? 172.21.15.40:0/118061912' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2023-07-14T19:09:18.977 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Jul 14 19:09:18 smithi040 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-mgr-x[96762]: 2023-07-14T19:09:18.497+0000 7fc08d126200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-07-14T19:09:19.471 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Jul 14 19:09:19 smithi040 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-mgr-x[96762]: 2023-07-14T19:09:19.185+0000 7fc08d126200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-07-14T19:09:19.472 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Jul 14 19:09:19 smithi040 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-mgr-x[96762]: 2023-07-14T19:09:19.376+0000 7fc08d126200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-07-14T19:09:20.663 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Jul 14 19:09:20 smithi040 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-mgr-x[96762]: 2023-07-14T19:09:20.332+0000 7fc08d126200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-07-14T19:09:20.663 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Jul 14 19:09:20 smithi040 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-mgr-x[96762]: 2023-07-14T19:09:20.442+0000 7fc08d126200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-07-14T19:09:20.664 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Jul 14 19:09:20 smithi040 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-mgr-x[96762]: 2023-07-14T19:09:20.660+0000 7fc08d126200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-07-14T19:09:21.262 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Jul 14 19:09:21 smithi040 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-mgr-x[96762]: 2023-07-14T19:09:21.005+0000 7fc08d126200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-07-14T19:09:21.721 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Jul 14 19:09:21 smithi040 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-mgr-x[96762]: 2023-07-14T19:09:21.259+0000 7fc08d126200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-07-14T19:09:21.722 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Jul 14 19:09:21 smithi040 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-mgr-x[96762]: 2023-07-14T19:09:21.440+0000 7fc08d126200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-07-14T19:09:22.222 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Jul 14 19:09:21 smithi040 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-mgr-x[96762]: 2023-07-14T19:09:21.767+0000 7fc08d126200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-07-14T19:09:22.972 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Jul 14 19:09:22 smithi040 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-mgr-x[96762]: 2023-07-14T19:09:22.585+0000 7fc08d126200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-07-14T19:09:22.973 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Jul 14 19:09:22 smithi040 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-mgr-x[96762]: 2023-07-14T19:09:22.742+0000 7fc08d126200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-07-14T19:09:23.393 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Jul 14 19:09:22 smithi040 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-mgr-x[96762]: 2023-07-14T19:09:22.984+0000 7fc08d126200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-07-14T19:09:23.393 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Jul 14 19:09:23 smithi040 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-mgr-x[96762]: 2023-07-14T19:09:23.120+0000 7fc08d126200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-07-14T19:09:23.721 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Jul 14 19:09:23 smithi040 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-mgr-x[96762]: 2023-07-14T19:09:23.391+0000 7fc08d126200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-07-14T19:09:24.721 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Jul 14 19:09:24 smithi040 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-mgr-x[96762]: 2023-07-14T19:09:24.426+0000 7fc08d126200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-07-14T19:09:24.722 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:09:24 smithi040 ceph-mon[96518]: Active manager daemon x restarted 2023-07-14T19:09:24.722 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:09:24 smithi040 ceph-mon[96518]: Activating manager daemon x 2023-07-14T19:09:24.722 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:09:24 smithi040 ceph-mon[96518]: osdmap e2: 0 total, 0 up, 0 in 2023-07-14T19:09:24.722 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:09:24 smithi040 ceph-mon[96518]: mgrmap e6: x(active, starting, since 0.00316451s) 2023-07-14T19:09:24.723 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:09:24 smithi040 ceph-mon[96518]: from='mgr.14120 172.21.15.40:0/1993184375' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-07-14T19:09:24.723 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:09:24 smithi040 ceph-mon[96518]: from='mgr.14120 172.21.15.40:0/1993184375' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2023-07-14T19:09:24.723 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:09:24 smithi040 ceph-mon[96518]: from='mgr.14120 172.21.15.40:0/1993184375' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2023-07-14T19:09:24.723 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:09:24 smithi040 ceph-mon[96518]: from='mgr.14120 172.21.15.40:0/1993184375' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2023-07-14T19:09:24.723 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:09:24 smithi040 ceph-mon[96518]: from='mgr.14120 172.21.15.40:0/1993184375' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2023-07-14T19:09:24.724 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:09:24 smithi040 ceph-mon[96518]: Manager daemon x is now available 2023-07-14T19:09:24.724 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:09:24 smithi040 ceph-mon[96518]: from='mgr.14120 172.21.15.40:0/1993184375' entity='mgr.x' 2023-07-14T19:09:24.724 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:09:24 smithi040 ceph-mon[96518]: from='mgr.14120 172.21.15.40:0/1993184375' entity='mgr.x' 2023-07-14T19:09:24.724 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:09:24 smithi040 ceph-mon[96518]: from='mgr.14120 172.21.15.40:0/1993184375' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T19:09:24.724 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:09:24 smithi040 ceph-mon[96518]: from='mgr.14120 172.21.15.40:0/1993184375' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T19:09:25.776 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout { 2023-07-14T19:09:25.776 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 7, 2023-07-14T19:09:25.776 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "initialized": true 2023-07-14T19:09:25.776 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout } 2023-07-14T19:09:25.776 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:18.922+0000 7fd12574d700 1 Processor -- start 2023-07-14T19:09:25.776 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:18.922+0000 7fd12574d700 1 -- start start 2023-07-14T19:09:25.777 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:18.922+0000 7fd12574d700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd120105410 0x7fd120105820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:09:25.777 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:18.922+0000 7fd12574d700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd120105d60 con 0x7fd120105410 2023-07-14T19:09:25.777 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:18.923+0000 7fd11effd700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd120105410 0x7fd120105820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:09:25.777 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:18.923+0000 7fd11effd700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd120105410 0x7fd120105820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:59838/0 (socket says 172.21.15.40:59838) 2023-07-14T19:09:25.777 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:18.923+0000 7fd11effd700 1 -- 172.21.15.40:0/3129992074 learned_addr learned my addr 172.21.15.40:0/3129992074 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-07-14T19:09:25.777 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:18.923+0000 7fd11effd700 1 -- 172.21.15.40:0/3129992074 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd120105ea0 con 0x7fd120105410 2023-07-14T19:09:25.778 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:18.923+0000 7fd11effd700 1 --2- 172.21.15.40:0/3129992074 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd120105410 0x7fd120105820 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fd1100098a0 tx=0x7fd110009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=68ba9e3dc4ecea0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:09:25.778 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:18.923+0000 7fd11dffb700 1 -- 172.21.15.40:0/3129992074 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd110004160 con 0x7fd120105410 2023-07-14T19:09:25.778 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:18.923+0000 7fd11dffb700 1 -- 172.21.15.40:0/3129992074 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd1100042c0 con 0x7fd120105410 2023-07-14T19:09:25.778 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:18.923+0000 7fd11dffb700 1 -- 172.21.15.40:0/3129992074 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd110011a00 con 0x7fd120105410 2023-07-14T19:09:25.778 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:18.924+0000 7fd12574d700 1 -- 172.21.15.40:0/3129992074 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd120105410 msgr2=0x7fd120105820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:09:25.778 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:18.924+0000 7fd12574d700 1 --2- 172.21.15.40:0/3129992074 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd120105410 0x7fd120105820 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fd1100098a0 tx=0x7fd110009bb0 comp rx=0 tx=0).stop 2023-07-14T19:09:25.778 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:18.924+0000 7fd12574d700 1 -- 172.21.15.40:0/3129992074 shutdown_connections 2023-07-14T19:09:25.779 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:18.924+0000 7fd12574d700 1 --2- 172.21.15.40:0/3129992074 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd120105410 0x7fd120105820 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:09:25.779 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:18.924+0000 7fd12574d700 1 -- 172.21.15.40:0/3129992074 >> 172.21.15.40:0/3129992074 conn(0x7fd120100ac0 msgr2=0x7fd120102ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:09:25.779 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:18.924+0000 7fd12574d700 1 -- 172.21.15.40:0/3129992074 shutdown_connections 2023-07-14T19:09:25.779 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:18.924+0000 7fd12574d700 1 -- 172.21.15.40:0/3129992074 wait complete. 2023-07-14T19:09:25.779 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:18.925+0000 7fd12574d700 1 Processor -- start 2023-07-14T19:09:25.779 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:18.925+0000 7fd12574d700 1 -- start start 2023-07-14T19:09:25.779 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:18.925+0000 7fd12574d700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd120105410 0x7fd12019b360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:09:25.779 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:18.925+0000 7fd12574d700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd12019b8a0 con 0x7fd120105410 2023-07-14T19:09:25.780 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:18.925+0000 7fd11effd700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd120105410 0x7fd12019b360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:09:25.780 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:18.925+0000 7fd11effd700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd120105410 0x7fd12019b360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:59840/0 (socket says 172.21.15.40:59840) 2023-07-14T19:09:25.780 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:18.925+0000 7fd11effd700 1 -- 172.21.15.40:0/2595428231 learned_addr learned my addr 172.21.15.40:0/2595428231 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-07-14T19:09:25.780 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:18.925+0000 7fd11effd700 1 -- 172.21.15.40:0/2595428231 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd110009550 con 0x7fd120105410 2023-07-14T19:09:25.780 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:18.926+0000 7fd11effd700 1 --2- 172.21.15.40:0/2595428231 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd120105410 0x7fd12019b360 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fd110009fb0 tx=0x7fd110011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:09:25.780 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:18.926+0000 7fd107fff700 1 -- 172.21.15.40:0/2595428231 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd110003a40 con 0x7fd120105410 2023-07-14T19:09:25.780 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:18.926+0000 7fd107fff700 1 -- 172.21.15.40:0/2595428231 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd110003ba0 con 0x7fd120105410 2023-07-14T19:09:25.781 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:18.926+0000 7fd12574d700 1 -- 172.21.15.40:0/2595428231 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd12019baa0 con 0x7fd120105410 2023-07-14T19:09:25.781 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:18.926+0000 7fd107fff700 1 -- 172.21.15.40:0/2595428231 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd11001a750 con 0x7fd120105410 2023-07-14T19:09:25.781 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:18.926+0000 7fd12574d700 1 -- 172.21.15.40:0/2595428231 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd12019bec0 con 0x7fd120105410 2023-07-14T19:09:25.781 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:18.927+0000 7fd107fff700 1 -- 172.21.15.40:0/2595428231 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 45483+0+0 (secure 0 0 0) 0x7fd11001a980 con 0x7fd120105410 2023-07-14T19:09:25.781 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:18.927+0000 7fd107fff700 1 --2- 172.21.15.40:0/2595428231 >> [v2:172.21.15.40:6800/2569015667,v1:172.21.15.40:6801/2569015667] conn(0x7fd108038710 0x7fd10803abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:09:25.781 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:18.927+0000 7fd107fff700 1 -- 172.21.15.40:0/2595428231 --> [v2:172.21.15.40:6800/2569015667,v1:172.21.15.40:6801/2569015667] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd10803b2d0 con 0x7fd108038710 2023-07-14T19:09:25.781 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:18.927+0000 7fd107fff700 1 -- 172.21.15.40:0/2595428231 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7fd11004c400 con 0x7fd120105410 2023-07-14T19:09:25.782 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:18.927+0000 7fd11e7fc700 1 -- 172.21.15.40:0/2595428231 >> [v2:172.21.15.40:6800/2569015667,v1:172.21.15.40:6801/2569015667] conn(0x7fd108038710 msgr2=0x7fd10803abc0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.40:6800/2569015667 2023-07-14T19:09:25.782 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:18.927+0000 7fd11e7fc700 1 --2- 172.21.15.40:0/2595428231 >> [v2:172.21.15.40:6800/2569015667,v1:172.21.15.40:6801/2569015667] conn(0x7fd108038710 0x7fd10803abc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-07-14T19:09:25.782 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:19.128+0000 7fd11e7fc700 1 -- 172.21.15.40:0/2595428231 >> [v2:172.21.15.40:6800/2569015667,v1:172.21.15.40:6801/2569015667] conn(0x7fd108038710 msgr2=0x7fd10803abc0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.40:6800/2569015667 2023-07-14T19:09:25.782 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:19.128+0000 7fd11e7fc700 1 --2- 172.21.15.40:0/2595428231 >> [v2:172.21.15.40:6800/2569015667,v1:172.21.15.40:6801/2569015667] conn(0x7fd108038710 0x7fd10803abc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-07-14T19:09:25.782 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:19.529+0000 7fd11e7fc700 1 -- 172.21.15.40:0/2595428231 >> [v2:172.21.15.40:6800/2569015667,v1:172.21.15.40:6801/2569015667] conn(0x7fd108038710 msgr2=0x7fd10803abc0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.40:6800/2569015667 2023-07-14T19:09:25.782 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:19.529+0000 7fd11e7fc700 1 --2- 172.21.15.40:0/2595428231 >> [v2:172.21.15.40:6800/2569015667,v1:172.21.15.40:6801/2569015667] conn(0x7fd108038710 0x7fd10803abc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-07-14T19:09:25.783 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:20.330+0000 7fd11e7fc700 1 -- 172.21.15.40:0/2595428231 >> [v2:172.21.15.40:6800/2569015667,v1:172.21.15.40:6801/2569015667] conn(0x7fd108038710 msgr2=0x7fd10803abc0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.40:6800/2569015667 2023-07-14T19:09:25.783 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:20.330+0000 7fd11e7fc700 1 --2- 172.21.15.40:0/2595428231 >> [v2:172.21.15.40:6800/2569015667,v1:172.21.15.40:6801/2569015667] conn(0x7fd108038710 0x7fd10803abc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-07-14T19:09:25.783 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:21.931+0000 7fd11e7fc700 1 -- 172.21.15.40:0/2595428231 >> [v2:172.21.15.40:6800/2569015667,v1:172.21.15.40:6801/2569015667] conn(0x7fd108038710 msgr2=0x7fd10803abc0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.40:6800/2569015667 2023-07-14T19:09:25.783 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:21.931+0000 7fd11e7fc700 1 --2- 172.21.15.40:0/2595428231 >> [v2:172.21.15.40:6800/2569015667,v1:172.21.15.40:6801/2569015667] conn(0x7fd108038710 0x7fd10803abc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2023-07-14T19:09:25.783 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:24.442+0000 7fd107fff700 1 -- 172.21.15.40:0/2595428231 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mgrmap(e 6) v1 ==== 45250+0+0 (secure 0 0 0) 0x7fd11002d910 con 0x7fd120105410 2023-07-14T19:09:25.783 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:24.442+0000 7fd107fff700 1 -- 172.21.15.40:0/2595428231 >> [v2:172.21.15.40:6800/2569015667,v1:172.21.15.40:6801/2569015667] conn(0x7fd108038710 msgr2=0x7fd10803abc0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-07-14T19:09:25.783 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:24.442+0000 7fd107fff700 1 --2- 172.21.15.40:0/2595428231 >> [v2:172.21.15.40:6800/2569015667,v1:172.21.15.40:6801/2569015667] conn(0x7fd108038710 0x7fd10803abc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:09:25.784 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:25.443+0000 7fd107fff700 1 -- 172.21.15.40:0/2595428231 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mgrmap(e 7) v1 ==== 45377+0+0 (secure 0 0 0) 0x7fd11001cda0 con 0x7fd120105410 2023-07-14T19:09:25.784 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:25.443+0000 7fd107fff700 1 --2- 172.21.15.40:0/2595428231 >> [v2:172.21.15.40:6800/3698757264,v1:172.21.15.40:6801/3698757264] conn(0x7fd10803bfe0 0x7fd10803e3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:09:25.784 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:25.443+0000 7fd107fff700 1 -- 172.21.15.40:0/2595428231 --> [v2:172.21.15.40:6800/3698757264,v1:172.21.15.40:6801/3698757264] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd10803b2d0 con 0x7fd10803bfe0 2023-07-14T19:09:25.784 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:25.444+0000 7fd11e7fc700 1 --2- 172.21.15.40:0/2595428231 >> [v2:172.21.15.40:6800/3698757264,v1:172.21.15.40:6801/3698757264] conn(0x7fd10803bfe0 0x7fd10803e3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:09:25.784 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:25.444+0000 7fd11e7fc700 1 --2- 172.21.15.40:0/2595428231 >> [v2:172.21.15.40:6800/3698757264,v1:172.21.15.40:6801/3698757264] conn(0x7fd10803bfe0 0x7fd10803e3c0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fd114003a10 tx=0x7fd1140092b0 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:09:25.785 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:25.445+0000 7fd107fff700 1 -- 172.21.15.40:0/2595428231 <== mgr.14120 v2:172.21.15.40:6800/3698757264 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+6910 (secure 0 0 0) 0x7fd10803b2d0 con 0x7fd10803bfe0 2023-07-14T19:09:25.785 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:25.456+0000 7fd12574d700 1 -- 172.21.15.40:0/2595428231 --> [v2:172.21.15.40:6800/3698757264,v1:172.21.15.40:6801/3698757264] -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7fd12019c6a0 con 0x7fd10803bfe0 2023-07-14T19:09:25.785 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:25.456+0000 7fd107fff700 1 -- 172.21.15.40:0/2595428231 <== mgr.14120 v2:172.21.15.40:6800/3698757264 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+51 (secure 0 0 0) 0x7fd12019c6a0 con 0x7fd10803bfe0 2023-07-14T19:09:25.785 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:25.456+0000 7fd12574d700 1 -- 172.21.15.40:0/2595428231 >> [v2:172.21.15.40:6800/3698757264,v1:172.21.15.40:6801/3698757264] conn(0x7fd10803bfe0 msgr2=0x7fd10803e3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:09:25.785 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:25.456+0000 7fd12574d700 1 --2- 172.21.15.40:0/2595428231 >> [v2:172.21.15.40:6800/3698757264,v1:172.21.15.40:6801/3698757264] conn(0x7fd10803bfe0 0x7fd10803e3c0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fd114003a10 tx=0x7fd1140092b0 comp rx=0 tx=0).stop 2023-07-14T19:09:25.785 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:25.456+0000 7fd12574d700 1 -- 172.21.15.40:0/2595428231 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd120105410 msgr2=0x7fd12019b360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:09:25.785 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:25.456+0000 7fd12574d700 1 --2- 172.21.15.40:0/2595428231 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd120105410 0x7fd12019b360 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fd110009fb0 tx=0x7fd110011ad0 comp rx=0 tx=0).stop 2023-07-14T19:09:25.786 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:25.456+0000 7fd12574d700 1 -- 172.21.15.40:0/2595428231 shutdown_connections 2023-07-14T19:09:25.786 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:25.456+0000 7fd12574d700 1 --2- 172.21.15.40:0/2595428231 >> [v2:172.21.15.40:6800/3698757264,v1:172.21.15.40:6801/3698757264] conn(0x7fd10803bfe0 0x7fd10803e3c0 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:09:25.786 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:25.456+0000 7fd12574d700 1 --2- 172.21.15.40:0/2595428231 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd120105410 0x7fd12019b360 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:09:25.786 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:25.456+0000 7fd12574d700 1 --2- 172.21.15.40:0/2595428231 >> [v2:172.21.15.40:6800/2569015667,v1:172.21.15.40:6801/2569015667] conn(0x7fd108038710 0x7fd10803abc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:09:25.786 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:25.456+0000 7fd12574d700 1 -- 172.21.15.40:0/2595428231 >> 172.21.15.40:0/2595428231 conn(0x7fd120100ac0 msgr2=0x7fd120102160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:09:25.786 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:25.456+0000 7fd12574d700 1 -- 172.21.15.40:0/2595428231 shutdown_connections 2023-07-14T19:09:25.786 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:25.457+0000 7fd12574d700 1 -- 172.21.15.40:0/2595428231 wait complete. 2023-07-14T19:09:25.787 INFO:teuthology.orchestra.run.smithi040.stdout:mgr epoch 5 is available 2023-07-14T19:09:25.787 INFO:teuthology.orchestra.run.smithi040.stdout:Setting orchestrator backend to cephadm... 2023-07-14T19:09:25.798 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:09:25 smithi040 ceph-mon[96518]: Found migration_current of "None". Setting to last migration. 2023-07-14T19:09:25.798 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:09:25 smithi040 ceph-mon[96518]: from='mgr.14120 172.21.15.40:0/1993184375' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T19:09:25.799 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:09:25 smithi040 ceph-mon[96518]: from='mgr.14120 172.21.15.40:0/1993184375' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2023-07-14T19:09:25.799 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:09:25 smithi040 ceph-mon[96518]: from='mgr.14120 172.21.15.40:0/1993184375' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2023-07-14T19:09:25.799 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:09:25 smithi040 ceph-mon[96518]: mgrmap e7: x(active, since 1.00541s) 2023-07-14T19:09:26.843 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:09:26 smithi040 ceph-mon[96518]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2023-07-14T19:09:26.843 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:09:26 smithi040 ceph-mon[96518]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2023-07-14T19:09:26.843 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:09:26 smithi040 ceph-mon[96518]: from='mgr.14120 172.21.15.40:0/1993184375' entity='mgr.x' 2023-07-14T19:09:26.843 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:09:26 smithi040 ceph-mon[96518]: from='mgr.14120 172.21.15.40:0/1993184375' entity='mgr.x' 2023-07-14T19:09:27.118 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:26.271+0000 7fc72db6b700 1 Processor -- start 2023-07-14T19:09:27.121 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:26.271+0000 7fc72db6b700 1 -- start start 2023-07-14T19:09:27.122 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:26.271+0000 7fc72db6b700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc728079ce0 0x7fc728107f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:09:27.122 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:26.271+0000 7fc72db6b700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7281084d0 con 0x7fc728079ce0 2023-07-14T19:09:27.122 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:26.272+0000 7fc7277fe700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc728079ce0 0x7fc728107f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:09:27.122 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:26.272+0000 7fc7277fe700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc728079ce0 0x7fc728107f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:59884/0 (socket says 172.21.15.40:59884) 2023-07-14T19:09:27.123 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:26.272+0000 7fc7277fe700 1 -- 172.21.15.40:0/1943658328 learned_addr learned my addr 172.21.15.40:0/1943658328 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-07-14T19:09:27.123 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:26.272+0000 7fc7277fe700 1 -- 172.21.15.40:0/1943658328 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc728108610 con 0x7fc728079ce0 2023-07-14T19:09:27.123 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:26.272+0000 7fc7277fe700 1 --2- 172.21.15.40:0/1943658328 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc728079ce0 0x7fc728107f90 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fc7180098a0 tx=0x7fc718009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7c7d434a15a65afa server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:09:27.123 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:26.272+0000 7fc7267fc700 1 -- 172.21.15.40:0/1943658328 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fc718004160 con 0x7fc728079ce0 2023-07-14T19:09:27.124 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:26.272+0000 7fc7267fc700 1 -- 172.21.15.40:0/1943658328 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc7180042c0 con 0x7fc728079ce0 2023-07-14T19:09:27.124 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:26.272+0000 7fc7267fc700 1 -- 172.21.15.40:0/1943658328 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fc718011a00 con 0x7fc728079ce0 2023-07-14T19:09:27.124 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:26.273+0000 7fc72db6b700 1 -- 172.21.15.40:0/1943658328 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc728079ce0 msgr2=0x7fc728107f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:09:27.124 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:26.273+0000 7fc72db6b700 1 --2- 172.21.15.40:0/1943658328 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc728079ce0 0x7fc728107f90 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fc7180098a0 tx=0x7fc718009bb0 comp rx=0 tx=0).stop 2023-07-14T19:09:27.125 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:26.273+0000 7fc72db6b700 1 -- 172.21.15.40:0/1943658328 shutdown_connections 2023-07-14T19:09:27.125 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:26.273+0000 7fc72db6b700 1 --2- 172.21.15.40:0/1943658328 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc728079ce0 0x7fc728107f90 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:09:27.125 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:26.273+0000 7fc72db6b700 1 -- 172.21.15.40:0/1943658328 >> 172.21.15.40:0/1943658328 conn(0x7fc7280ff9a0 msgr2=0x7fc728101db0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:09:27.126 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:26.273+0000 7fc72db6b700 1 -- 172.21.15.40:0/1943658328 shutdown_connections 2023-07-14T19:09:27.126 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:26.273+0000 7fc72db6b700 1 -- 172.21.15.40:0/1943658328 wait complete. 2023-07-14T19:09:27.126 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:26.274+0000 7fc72db6b700 1 Processor -- start 2023-07-14T19:09:27.127 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:26.274+0000 7fc72db6b700 1 -- start start 2023-07-14T19:09:27.127 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:26.274+0000 7fc72db6b700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc7281990d0 0x7fc7281994e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:09:27.127 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:26.274+0000 7fc72db6b700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc728199a20 con 0x7fc7281990d0 2023-07-14T19:09:27.128 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:26.274+0000 7fc7277fe700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc7281990d0 0x7fc7281994e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:09:27.128 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:26.274+0000 7fc7277fe700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc7281990d0 0x7fc7281994e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:59886/0 (socket says 172.21.15.40:59886) 2023-07-14T19:09:27.128 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:26.274+0000 7fc7277fe700 1 -- 172.21.15.40:0/1911705653 learned_addr learned my addr 172.21.15.40:0/1911705653 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-07-14T19:09:27.128 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:26.274+0000 7fc7277fe700 1 -- 172.21.15.40:0/1911705653 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc718009550 con 0x7fc7281990d0 2023-07-14T19:09:27.129 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:26.275+0000 7fc7277fe700 1 --2- 172.21.15.40:0/1911705653 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc7281990d0 0x7fc7281994e0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fc718009fb0 tx=0x7fc718011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:09:27.130 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:26.275+0000 7fc724ff9700 1 -- 172.21.15.40:0/1911705653 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fc718003930 con 0x7fc7281990d0 2023-07-14T19:09:27.130 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:26.275+0000 7fc72db6b700 1 -- 172.21.15.40:0/1911705653 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc728199c20 con 0x7fc7281990d0 2023-07-14T19:09:27.130 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:26.275+0000 7fc724ff9700 1 -- 172.21.15.40:0/1911705653 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc718003a90 con 0x7fc7281990d0 2023-07-14T19:09:27.130 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:26.275+0000 7fc724ff9700 1 -- 172.21.15.40:0/1911705653 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fc7180195e0 con 0x7fc7281990d0 2023-07-14T19:09:27.131 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:26.275+0000 7fc72db6b700 1 -- 172.21.15.40:0/1911705653 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc72819c880 con 0x7fc7281990d0 2023-07-14T19:09:27.131 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:26.276+0000 7fc724ff9700 1 -- 172.21.15.40:0/1911705653 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 7) v1 ==== 45377+0+0 (secure 0 0 0) 0x7fc7180198c0 con 0x7fc7281990d0 2023-07-14T19:09:27.131 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:26.276+0000 7fc724ff9700 1 --2- 172.21.15.40:0/1911705653 >> [v2:172.21.15.40:6800/3698757264,v1:172.21.15.40:6801/3698757264] conn(0x7fc7100385a0 0x7fc71003aa50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:09:27.132 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:26.276+0000 7fc724ff9700 1 -- 172.21.15.40:0/1911705653 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7fc71804c630 con 0x7fc7281990d0 2023-07-14T19:09:27.132 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:26.276+0000 7fc72db6b700 1 -- 172.21.15.40:0/1911705653 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc728079ce0 con 0x7fc7281990d0 2023-07-14T19:09:27.132 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:26.276+0000 7fc726ffd700 1 --2- 172.21.15.40:0/1911705653 >> [v2:172.21.15.40:6800/3698757264,v1:172.21.15.40:6801/3698757264] conn(0x7fc7100385a0 0x7fc71003aa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:09:27.132 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:26.277+0000 7fc726ffd700 1 --2- 172.21.15.40:0/1911705653 >> [v2:172.21.15.40:6800/3698757264,v1:172.21.15.40:6801/3698757264] conn(0x7fc7100385a0 0x7fc71003aa50 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fc71c006fd0 tx=0x7fc71c006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:09:27.133 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:26.281+0000 7fc724ff9700 1 -- 172.21.15.40:0/1911705653 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fc718019bc0 con 0x7fc7281990d0 2023-07-14T19:09:27.133 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:26.669+0000 7fc72db6b700 1 -- 172.21.15.40:0/1911705653 --> [v2:172.21.15.40:6800/3698757264,v1:172.21.15.40:6801/3698757264] -- mgr_command(tid 0: {"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}) v1 -- 0x7fc72819ca70 con 0x7fc7100385a0 2023-07-14T19:09:27.133 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:26.674+0000 7fc724ff9700 1 -- 172.21.15.40:0/1911705653 <== mgr.14120 v2:172.21.15.40:6800/3698757264 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fc72819ca70 con 0x7fc7100385a0 2023-07-14T19:09:27.133 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:26.677+0000 7fc72db6b700 1 -- 172.21.15.40:0/1911705653 >> [v2:172.21.15.40:6800/3698757264,v1:172.21.15.40:6801/3698757264] conn(0x7fc7100385a0 msgr2=0x7fc71003aa50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:09:27.134 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:26.677+0000 7fc72db6b700 1 --2- 172.21.15.40:0/1911705653 >> [v2:172.21.15.40:6800/3698757264,v1:172.21.15.40:6801/3698757264] conn(0x7fc7100385a0 0x7fc71003aa50 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fc71c006fd0 tx=0x7fc71c006e40 comp rx=0 tx=0).stop 2023-07-14T19:09:27.134 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:26.677+0000 7fc72db6b700 1 -- 172.21.15.40:0/1911705653 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc7281990d0 msgr2=0x7fc7281994e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:09:27.134 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:26.677+0000 7fc72db6b700 1 --2- 172.21.15.40:0/1911705653 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc7281990d0 0x7fc7281994e0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fc718009fb0 tx=0x7fc718011ad0 comp rx=0 tx=0).stop 2023-07-14T19:09:27.134 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:26.677+0000 7fc72db6b700 1 -- 172.21.15.40:0/1911705653 shutdown_connections 2023-07-14T19:09:27.135 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:26.677+0000 7fc72db6b700 1 --2- 172.21.15.40:0/1911705653 >> [v2:172.21.15.40:6800/3698757264,v1:172.21.15.40:6801/3698757264] conn(0x7fc7100385a0 0x7fc71003aa50 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:09:27.135 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:26.677+0000 7fc72db6b700 1 --2- 172.21.15.40:0/1911705653 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc7281990d0 0x7fc7281994e0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:09:27.135 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:26.677+0000 7fc72db6b700 1 -- 172.21.15.40:0/1911705653 >> 172.21.15.40:0/1911705653 conn(0x7fc7280ff9a0 msgr2=0x7fc728100780 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:09:27.135 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:26.677+0000 7fc72db6b700 1 -- 172.21.15.40:0/1911705653 shutdown_connections 2023-07-14T19:09:27.136 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:26.677+0000 7fc72db6b700 1 -- 172.21.15.40:0/1911705653 wait complete. 2023-07-14T19:09:27.972 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:09:27 smithi040 ceph-mon[96518]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T19:09:27.972 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:09:27 smithi040 ceph-mon[96518]: from='mgr.14120 172.21.15.40:0/1993184375' entity='mgr.x' 2023-07-14T19:09:27.972 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:09:27 smithi040 ceph-mon[96518]: from='mgr.14120 172.21.15.40:0/1993184375' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T19:09:27.972 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:09:27 smithi040 ceph-mon[96518]: mgrmap e8: x(active, since 2s) 2023-07-14T19:09:27.973 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:09:27 smithi040 ceph-mon[96518]: [14/Jul/2023:19:09:27] ENGINE Bus STARTING 2023-07-14T19:09:27.973 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:09:27 smithi040 ceph-mon[96518]: [14/Jul/2023:19:09:27] ENGINE Serving on https://172.21.15.40:7150 2023-07-14T19:09:27.973 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:09:27 smithi040 ceph-mon[96518]: [14/Jul/2023:19:09:27] ENGINE Serving on http://172.21.15.40:8765 2023-07-14T19:09:27.973 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:09:27 smithi040 ceph-mon[96518]: [14/Jul/2023:19:09:27] ENGINE Bus STARTED 2023-07-14T19:09:27.973 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:09:27 smithi040 ceph-mon[96518]: from='mgr.14120 172.21.15.40:0/1993184375' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T19:09:28.566 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout value unchanged 2023-07-14T19:09:28.567 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:27.637+0000 7f0ae4eae700 1 Processor -- start 2023-07-14T19:09:28.567 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:27.637+0000 7f0ae4eae700 1 -- start start 2023-07-14T19:09:28.567 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:27.637+0000 7f0ae4eae700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0ae01042f0 0x7f0ae0104700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:09:28.567 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:27.637+0000 7f0ae4eae700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ae0104c40 con 0x7f0ae01042f0 2023-07-14T19:09:28.567 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:27.637+0000 7f0ade59c700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0ae01042f0 0x7f0ae0104700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:09:28.568 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:27.637+0000 7f0ade59c700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0ae01042f0 0x7f0ae0104700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:59898/0 (socket says 172.21.15.40:59898) 2023-07-14T19:09:28.568 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:27.637+0000 7f0ade59c700 1 -- 172.21.15.40:0/282110250 learned_addr learned my addr 172.21.15.40:0/282110250 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-07-14T19:09:28.568 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:27.637+0000 7f0ade59c700 1 -- 172.21.15.40:0/282110250 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ae0104d80 con 0x7f0ae01042f0 2023-07-14T19:09:28.568 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:27.638+0000 7f0ade59c700 1 --2- 172.21.15.40:0/282110250 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0ae01042f0 0x7f0ae0104700 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f0ad00098a0 tx=0x7f0ad0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d62dc25dd3b3b55b server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:09:28.568 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:27.638+0000 7f0add59a700 1 -- 172.21.15.40:0/282110250 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0ad0004160 con 0x7f0ae01042f0 2023-07-14T19:09:28.569 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:27.638+0000 7f0add59a700 1 -- 172.21.15.40:0/282110250 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0ad00042c0 con 0x7f0ae01042f0 2023-07-14T19:09:28.569 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:27.638+0000 7f0add59a700 1 -- 172.21.15.40:0/282110250 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0ad0011a00 con 0x7f0ae01042f0 2023-07-14T19:09:28.569 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:27.638+0000 7f0ae4eae700 1 -- 172.21.15.40:0/282110250 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0ae01042f0 msgr2=0x7f0ae0104700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:09:28.569 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:27.638+0000 7f0ae4eae700 1 --2- 172.21.15.40:0/282110250 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0ae01042f0 0x7f0ae0104700 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f0ad00098a0 tx=0x7f0ad0009bb0 comp rx=0 tx=0).stop 2023-07-14T19:09:28.569 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:27.639+0000 7f0ae4eae700 1 -- 172.21.15.40:0/282110250 shutdown_connections 2023-07-14T19:09:28.569 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:27.639+0000 7f0ae4eae700 1 --2- 172.21.15.40:0/282110250 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0ae01042f0 0x7f0ae0104700 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:09:28.570 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:27.639+0000 7f0ae4eae700 1 -- 172.21.15.40:0/282110250 >> 172.21.15.40:0/282110250 conn(0x7f0ae00ff9a0 msgr2=0x7f0ae0101db0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:09:28.572 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:27.639+0000 7f0ae4eae700 1 -- 172.21.15.40:0/282110250 shutdown_connections 2023-07-14T19:09:28.572 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:27.639+0000 7f0ae4eae700 1 -- 172.21.15.40:0/282110250 wait complete. 2023-07-14T19:09:28.572 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:27.639+0000 7f0ae4eae700 1 Processor -- start 2023-07-14T19:09:28.573 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:27.639+0000 7f0ae4eae700 1 -- start start 2023-07-14T19:09:28.573 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:27.640+0000 7f0ae4eae700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0ae01042f0 0x7f0ae0196ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:09:28.573 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:27.640+0000 7f0ae4eae700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ae01973e0 con 0x7f0ae01042f0 2023-07-14T19:09:28.573 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:27.640+0000 7f0ade59c700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0ae01042f0 0x7f0ae0196ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:09:28.573 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:27.640+0000 7f0ade59c700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0ae01042f0 0x7f0ae0196ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:59900/0 (socket says 172.21.15.40:59900) 2023-07-14T19:09:28.573 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:27.640+0000 7f0ade59c700 1 -- 172.21.15.40:0/3366706252 learned_addr learned my addr 172.21.15.40:0/3366706252 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-07-14T19:09:28.574 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:27.640+0000 7f0ade59c700 1 -- 172.21.15.40:0/3366706252 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ad0009550 con 0x7f0ae01042f0 2023-07-14T19:09:28.574 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:27.640+0000 7f0ade59c700 1 --2- 172.21.15.40:0/3366706252 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0ae01042f0 0x7f0ae0196ea0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f0ad0009fb0 tx=0x7f0ad0011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:09:28.574 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:27.640+0000 7f0acf7fe700 1 -- 172.21.15.40:0/3366706252 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0ad0003930 con 0x7f0ae01042f0 2023-07-14T19:09:28.574 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:27.640+0000 7f0acf7fe700 1 -- 172.21.15.40:0/3366706252 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0ad0003a90 con 0x7f0ae01042f0 2023-07-14T19:09:28.574 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:27.640+0000 7f0ae4eae700 1 -- 172.21.15.40:0/3366706252 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ae01975e0 con 0x7f0ae01042f0 2023-07-14T19:09:28.574 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:27.640+0000 7f0acf7fe700 1 -- 172.21.15.40:0/3366706252 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0ad0019610 con 0x7f0ae01042f0 2023-07-14T19:09:28.575 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:27.641+0000 7f0ae4eae700 1 -- 172.21.15.40:0/3366706252 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ae0197a00 con 0x7f0ae01042f0 2023-07-14T19:09:28.575 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:27.642+0000 7f0acf7fe700 1 -- 172.21.15.40:0/3366706252 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 8) v1 ==== 45483+0+0 (secure 0 0 0) 0x7f0ad001a430 con 0x7f0ae01042f0 2023-07-14T19:09:28.575 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:27.642+0000 7f0ae4eae700 1 -- 172.21.15.40:0/3366706252 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0ae0190c20 con 0x7f0ae01042f0 2023-07-14T19:09:28.575 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:27.642+0000 7f0acf7fe700 1 --2- 172.21.15.40:0/3366706252 >> [v2:172.21.15.40:6800/3698757264,v1:172.21.15.40:6801/3698757264] conn(0x7f0ac803a910 0x7f0ac803cdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:09:28.575 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:27.642+0000 7f0acf7fe700 1 -- 172.21.15.40:0/3366706252 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f0ad004c7e0 con 0x7f0ae01042f0 2023-07-14T19:09:28.575 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:27.642+0000 7f0addd9b700 1 --2- 172.21.15.40:0/3366706252 >> [v2:172.21.15.40:6800/3698757264,v1:172.21.15.40:6801/3698757264] conn(0x7f0ac803a910 0x7f0ac803cdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:09:28.576 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:27.642+0000 7f0addd9b700 1 --2- 172.21.15.40:0/3366706252 >> [v2:172.21.15.40:6800/3698757264,v1:172.21.15.40:6801/3698757264] conn(0x7f0ac803a910 0x7f0ac803cdc0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f0ad4006fd0 tx=0x7f0ad4006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:09:28.576 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:27.647+0000 7f0acf7fe700 1 -- 172.21.15.40:0/3366706252 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f0ad0023b80 con 0x7f0ae01042f0 2023-07-14T19:09:28.576 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:28.003+0000 7f0ae4eae700 1 -- 172.21.15.40:0/3366706252 --> [v2:172.21.15.40:6800/3698757264,v1:172.21.15.40:6801/3698757264] -- mgr_command(tid 0: {"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}) v1 -- 0x7f0ae0060a80 con 0x7f0ac803a910 2023-07-14T19:09:28.576 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:28.004+0000 7f0acf7fe700 1 -- 172.21.15.40:0/3366706252 <== mgr.14120 v2:172.21.15.40:6800/3698757264 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+16 (secure 0 0 0) 0x7f0ae0060a80 con 0x7f0ac803a910 2023-07-14T19:09:28.576 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:28.006+0000 7f0ae4eae700 1 -- 172.21.15.40:0/3366706252 >> [v2:172.21.15.40:6800/3698757264,v1:172.21.15.40:6801/3698757264] conn(0x7f0ac803a910 msgr2=0x7f0ac803cdc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:09:28.576 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:28.006+0000 7f0ae4eae700 1 --2- 172.21.15.40:0/3366706252 >> [v2:172.21.15.40:6800/3698757264,v1:172.21.15.40:6801/3698757264] conn(0x7f0ac803a910 0x7f0ac803cdc0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f0ad4006fd0 tx=0x7f0ad4006e40 comp rx=0 tx=0).stop 2023-07-14T19:09:28.577 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:28.006+0000 7f0ae4eae700 1 -- 172.21.15.40:0/3366706252 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0ae01042f0 msgr2=0x7f0ae0196ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:09:28.577 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:28.006+0000 7f0ae4eae700 1 --2- 172.21.15.40:0/3366706252 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0ae01042f0 0x7f0ae0196ea0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f0ad0009fb0 tx=0x7f0ad0011ad0 comp rx=0 tx=0).stop 2023-07-14T19:09:28.577 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:28.006+0000 7f0ae4eae700 1 -- 172.21.15.40:0/3366706252 shutdown_connections 2023-07-14T19:09:28.577 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:28.006+0000 7f0ae4eae700 1 --2- 172.21.15.40:0/3366706252 >> [v2:172.21.15.40:6800/3698757264,v1:172.21.15.40:6801/3698757264] conn(0x7f0ac803a910 0x7f0ac803cdc0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:09:28.577 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:28.006+0000 7f0ae4eae700 1 --2- 172.21.15.40:0/3366706252 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0ae01042f0 0x7f0ae0196ea0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:09:28.578 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:28.006+0000 7f0ae4eae700 1 -- 172.21.15.40:0/3366706252 >> 172.21.15.40:0/3366706252 conn(0x7f0ae00ff9a0 msgr2=0x7f0ae018dd60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:09:28.578 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:28.006+0000 7f0ae4eae700 1 -- 172.21.15.40:0/3366706252 shutdown_connections 2023-07-14T19:09:28.578 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:28.006+0000 7f0ae4eae700 1 -- 172.21.15.40:0/3366706252 wait complete. 2023-07-14T19:09:28.578 INFO:teuthology.orchestra.run.smithi040.stdout:Generating ssh key... 2023-07-14T19:09:28.971 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:09:28 smithi040 ceph-mon[96518]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T19:09:29.928 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:09:29 smithi040 ceph-mon[96518]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T19:09:29.929 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:09:29 smithi040 ceph-mon[96518]: Generating ssh key... 2023-07-14T19:09:30.222 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Jul 14 19:09:29 smithi040 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-mgr-x[96762]: Generating public/private rsa key pair. 2023-07-14T19:09:30.222 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Jul 14 19:09:29 smithi040 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-mgr-x[96762]: Your identification has been saved in /tmp/tmp2uu41otf/key. 2023-07-14T19:09:30.222 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Jul 14 19:09:29 smithi040 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-mgr-x[96762]: Your public key has been saved in /tmp/tmp2uu41otf/key.pub. 2023-07-14T19:09:30.222 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Jul 14 19:09:29 smithi040 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-mgr-x[96762]: The key fingerprint is: 2023-07-14T19:09:30.222 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Jul 14 19:09:29 smithi040 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-mgr-x[96762]: SHA256:XNuOuFxAcOZsST0mbJb+OTBi27kcmvUyN0UAE1b1vK4 ceph-adc52f04-2279-11ee-9b33-001a4aab830c 2023-07-14T19:09:30.223 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Jul 14 19:09:29 smithi040 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-mgr-x[96762]: The key's randomart image is: 2023-07-14T19:09:30.223 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Jul 14 19:09:29 smithi040 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-mgr-x[96762]: +---[RSA 3072]----+ 2023-07-14T19:09:30.223 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Jul 14 19:09:29 smithi040 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-mgr-x[96762]: | ..O*... | 2023-07-14T19:09:30.223 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Jul 14 19:09:29 smithi040 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-mgr-x[96762]: | O*o= o | 2023-07-14T19:09:30.224 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Jul 14 19:09:29 smithi040 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-mgr-x[96762]: | +*o.o o | 2023-07-14T19:09:30.224 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Jul 14 19:09:29 smithi040 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-mgr-x[96762]: | o++. o. . | 2023-07-14T19:09:30.224 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Jul 14 19:09:29 smithi040 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-mgr-x[96762]: | . +S=.o. . | 2023-07-14T19:09:30.224 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Jul 14 19:09:29 smithi040 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-mgr-x[96762]: | . =o+o.. | 2023-07-14T19:09:30.225 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Jul 14 19:09:29 smithi040 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-mgr-x[96762]: | =.+oo. . | 2023-07-14T19:09:30.225 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Jul 14 19:09:29 smithi040 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-mgr-x[96762]: | o.=o+ . | 2023-07-14T19:09:30.225 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Jul 14 19:09:29 smithi040 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-mgr-x[96762]: | o+ .E | 2023-07-14T19:09:30.226 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Jul 14 19:09:29 smithi040 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-mgr-x[96762]: +----[SHA256]-----+ 2023-07-14T19:09:30.313 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:29.085+0000 7f22646d6700 1 Processor -- start 2023-07-14T19:09:30.313 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:29.085+0000 7f22646d6700 1 -- start start 2023-07-14T19:09:30.313 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:29.085+0000 7f22646d6700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f225c105f00 0x7f225c1082e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:09:30.313 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:29.085+0000 7f22646d6700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f225c079d60 con 0x7f225c105f00 2023-07-14T19:09:30.314 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:29.086+0000 7f2262472700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f225c105f00 0x7f225c1082e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:09:30.314 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:29.086+0000 7f2262472700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f225c105f00 0x7f225c1082e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:59904/0 (socket says 172.21.15.40:59904) 2023-07-14T19:09:30.314 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:29.086+0000 7f2262472700 1 -- 172.21.15.40:0/2426331993 learned_addr learned my addr 172.21.15.40:0/2426331993 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-07-14T19:09:30.314 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:29.086+0000 7f2262472700 1 -- 172.21.15.40:0/2426331993 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f225c079ea0 con 0x7f225c105f00 2023-07-14T19:09:30.314 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:29.086+0000 7f2262472700 1 --2- 172.21.15.40:0/2426331993 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f225c105f00 0x7f225c1082e0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f224c0098a0 tx=0x7f224c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a46350b4570425d7 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:09:30.314 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:29.086+0000 7f2261470700 1 -- 172.21.15.40:0/2426331993 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f224c004160 con 0x7f225c105f00 2023-07-14T19:09:30.314 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:29.086+0000 7f2261470700 1 -- 172.21.15.40:0/2426331993 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f224c0042c0 con 0x7f225c105f00 2023-07-14T19:09:30.315 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:29.086+0000 7f2261470700 1 -- 172.21.15.40:0/2426331993 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f224c011a00 con 0x7f225c105f00 2023-07-14T19:09:30.315 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:29.087+0000 7f22646d6700 1 -- 172.21.15.40:0/2426331993 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f225c105f00 msgr2=0x7f225c1082e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:09:30.315 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:29.087+0000 7f22646d6700 1 --2- 172.21.15.40:0/2426331993 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f225c105f00 0x7f225c1082e0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f224c0098a0 tx=0x7f224c009bb0 comp rx=0 tx=0).stop 2023-07-14T19:09:30.315 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:29.087+0000 7f22646d6700 1 -- 172.21.15.40:0/2426331993 shutdown_connections 2023-07-14T19:09:30.315 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:29.087+0000 7f22646d6700 1 --2- 172.21.15.40:0/2426331993 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f225c105f00 0x7f225c1082e0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:09:30.315 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:29.087+0000 7f22646d6700 1 -- 172.21.15.40:0/2426331993 >> 172.21.15.40:0/2426331993 conn(0x7f225c0ff9a0 msgr2=0x7f225c101db0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:09:30.315 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:29.087+0000 7f22646d6700 1 -- 172.21.15.40:0/2426331993 shutdown_connections 2023-07-14T19:09:30.316 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:29.087+0000 7f22646d6700 1 -- 172.21.15.40:0/2426331993 wait complete. 2023-07-14T19:09:30.316 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:29.088+0000 7f22646d6700 1 Processor -- start 2023-07-14T19:09:30.316 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:29.088+0000 7f22646d6700 1 -- start start 2023-07-14T19:09:30.316 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:29.088+0000 7f22646d6700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f225c105f00 0x7f225c105860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:09:30.316 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:29.088+0000 7f22646d6700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f225c105da0 con 0x7f225c105f00 2023-07-14T19:09:30.316 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:29.088+0000 7f2262472700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f225c105f00 0x7f225c105860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:09:30.316 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:29.088+0000 7f2262472700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f225c105f00 0x7f225c105860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:59906/0 (socket says 172.21.15.40:59906) 2023-07-14T19:09:30.317 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:29.088+0000 7f2262472700 1 -- 172.21.15.40:0/645234698 learned_addr learned my addr 172.21.15.40:0/645234698 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-07-14T19:09:30.317 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:29.088+0000 7f2262472700 1 -- 172.21.15.40:0/645234698 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f224c009550 con 0x7f225c105f00 2023-07-14T19:09:30.317 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:29.088+0000 7f2262472700 1 --2- 172.21.15.40:0/645234698 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f225c105f00 0x7f225c105860 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f224c006af0 tx=0x7f224c011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:09:30.317 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:29.089+0000 7f22537fe700 1 -- 172.21.15.40:0/645234698 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f224c003a40 con 0x7f225c105f00 2023-07-14T19:09:30.317 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:29.089+0000 7f22537fe700 1 -- 172.21.15.40:0/645234698 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f224c003ba0 con 0x7f225c105f00 2023-07-14T19:09:30.318 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:29.089+0000 7f22646d6700 1 -- 172.21.15.40:0/645234698 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f225c103eb0 con 0x7f225c105f00 2023-07-14T19:09:30.318 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:29.089+0000 7f22537fe700 1 -- 172.21.15.40:0/645234698 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f224c01a700 con 0x7f225c105f00 2023-07-14T19:09:30.318 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:29.089+0000 7f22646d6700 1 -- 172.21.15.40:0/645234698 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f225c1042d0 con 0x7f225c105f00 2023-07-14T19:09:30.318 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:29.090+0000 7f22646d6700 1 -- 172.21.15.40:0/645234698 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f225c1094f0 con 0x7f225c105f00 2023-07-14T19:09:30.318 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:29.090+0000 7f22537fe700 1 -- 172.21.15.40:0/645234698 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 8) v1 ==== 45483+0+0 (secure 0 0 0) 0x7f224c024ad0 con 0x7f225c105f00 2023-07-14T19:09:30.318 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:29.090+0000 7f22537fe700 1 --2- 172.21.15.40:0/645234698 >> [v2:172.21.15.40:6800/3698757264,v1:172.21.15.40:6801/3698757264] conn(0x7f2248038760 0x7f224803ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:09:30.318 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:29.090+0000 7f22537fe700 1 -- 172.21.15.40:0/645234698 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f224c04cbc0 con 0x7f225c105f00 2023-07-14T19:09:30.318 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:29.091+0000 7f2261c71700 1 --2- 172.21.15.40:0/645234698 >> [v2:172.21.15.40:6800/3698757264,v1:172.21.15.40:6801/3698757264] conn(0x7f2248038760 0x7f224803ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:09:30.319 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:29.091+0000 7f2261c71700 1 --2- 172.21.15.40:0/645234698 >> [v2:172.21.15.40:6800/3698757264,v1:172.21.15.40:6801/3698757264] conn(0x7f2248038760 0x7f224803ac10 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f2258006fd0 tx=0x7f2258006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:09:30.319 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:29.095+0000 7f22537fe700 1 -- 172.21.15.40:0/645234698 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f224c01a930 con 0x7f225c105f00 2023-07-14T19:09:30.319 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:29.453+0000 7f22646d6700 1 -- 172.21.15.40:0/645234698 --> [v2:172.21.15.40:6800/3698757264,v1:172.21.15.40:6801/3698757264] -- mgr_command(tid 0: {"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}) v1 -- 0x7f225c100e90 con 0x7f2248038760 2023-07-14T19:09:30.319 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:29.931+0000 7f22537fe700 1 -- 172.21.15.40:0/645234698 <== mgr.14120 v2:172.21.15.40:6800/3698757264 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f225c100e90 con 0x7f2248038760 2023-07-14T19:09:30.319 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:29.933+0000 7f22646d6700 1 -- 172.21.15.40:0/645234698 >> [v2:172.21.15.40:6800/3698757264,v1:172.21.15.40:6801/3698757264] conn(0x7f2248038760 msgr2=0x7f224803ac10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:09:30.320 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:29.933+0000 7f22646d6700 1 --2- 172.21.15.40:0/645234698 >> [v2:172.21.15.40:6800/3698757264,v1:172.21.15.40:6801/3698757264] conn(0x7f2248038760 0x7f224803ac10 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f2258006fd0 tx=0x7f2258006e40 comp rx=0 tx=0).stop 2023-07-14T19:09:30.320 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:29.933+0000 7f22646d6700 1 -- 172.21.15.40:0/645234698 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f225c105f00 msgr2=0x7f225c105860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:09:30.320 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:29.933+0000 7f22646d6700 1 --2- 172.21.15.40:0/645234698 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f225c105f00 0x7f225c105860 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f224c006af0 tx=0x7f224c011ad0 comp rx=0 tx=0).stop 2023-07-14T19:09:30.320 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:29.933+0000 7f22646d6700 1 -- 172.21.15.40:0/645234698 shutdown_connections 2023-07-14T19:09:30.320 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:29.933+0000 7f22646d6700 1 --2- 172.21.15.40:0/645234698 >> [v2:172.21.15.40:6800/3698757264,v1:172.21.15.40:6801/3698757264] conn(0x7f2248038760 0x7f224803ac10 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:09:30.321 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:29.933+0000 7f22646d6700 1 --2- 172.21.15.40:0/645234698 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f225c105f00 0x7f225c105860 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:09:30.321 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:29.933+0000 7f22646d6700 1 -- 172.21.15.40:0/645234698 >> 172.21.15.40:0/645234698 conn(0x7f225c0ff9a0 msgr2=0x7f225c100780 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:09:30.321 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:29.934+0000 7f22646d6700 1 -- 172.21.15.40:0/645234698 shutdown_connections 2023-07-14T19:09:30.321 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:29.934+0000 7f22646d6700 1 -- 172.21.15.40:0/645234698 wait complete. 2023-07-14T19:09:31.199 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:09:30 smithi040 ceph-mon[96518]: from='mgr.14120 172.21.15.40:0/1993184375' entity='mgr.x' 2023-07-14T19:09:31.199 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:09:30 smithi040 ceph-mon[96518]: from='mgr.14120 172.21.15.40:0/1993184375' entity='mgr.x' 2023-07-14T19:09:31.558 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCx4XHNpeOaW8LCJ1HelEIxHYdFfJK2jKnOC8RvOyvlpn71Ync460tNVCQJXb3RCoqlxh61OhMdqTjl2siDjQyHPkGNKKl/tOSQMsrLi+lIJ7HgISudmy7A3mm87JxOXHs0imEP2Muq5gGZFE9TfDnfezULGMtW97ztYiegJb3//fVTtfezL0JXdpFDaWE34GYoEb7CAnUIlNce+UZtxXCp28xJwyiJHIoBpHFmxL4QJ9BueRBKrvWMdFWAg1WMyK9XH1hXCK74PLLFa3uMpAa/Qq2R/h4Fr5gkPGUr5JoJEu4+Wy20hOHHOpC3nFCrRzPi+SKSWtbPWiS+eHRuTwY3sQGyU57tdtgclKRjj4J49Ckiw0nWggstdKE4EFqCskeOzT4kJuAu5yKqMNcQ4zhPw58/zZD9Ipag3i7TphF1j4QlvcQ4V57ovBNwkEHnyD0yI/C0JwU8B+Ko5GQLGMderXrKxFM78mrHdCwh9ok+kqWfLZME+IzqNkO/40g0HxE= ceph-adc52f04-2279-11ee-9b33-001a4aab830c 2023-07-14T19:09:31.559 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:30.802+0000 7fa9c9119700 1 Processor -- start 2023-07-14T19:09:31.559 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:30.802+0000 7fa9c9119700 1 -- start start 2023-07-14T19:09:31.559 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:30.803+0000 7fa9c9119700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa9c4105410 0x7fa9c4105820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:09:31.559 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:30.803+0000 7fa9c9119700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9c4105d60 con 0x7fa9c4105410 2023-07-14T19:09:31.559 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:30.803+0000 7fa9c2d9d700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa9c4105410 0x7fa9c4105820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:09:31.560 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:30.803+0000 7fa9c2d9d700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa9c4105410 0x7fa9c4105820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:59910/0 (socket says 172.21.15.40:59910) 2023-07-14T19:09:31.560 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:30.803+0000 7fa9c2d9d700 1 -- 172.21.15.40:0/625326914 learned_addr learned my addr 172.21.15.40:0/625326914 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-07-14T19:09:31.560 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:30.803+0000 7fa9c2d9d700 1 -- 172.21.15.40:0/625326914 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9c4105ea0 con 0x7fa9c4105410 2023-07-14T19:09:31.560 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:30.803+0000 7fa9c2d9d700 1 --2- 172.21.15.40:0/625326914 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa9c4105410 0x7fa9c4105820 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fa9b40098a0 tx=0x7fa9b4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d5cfb9c01a1c5d4e server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:09:31.560 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:30.803+0000 7fa9c1d9b700 1 -- 172.21.15.40:0/625326914 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa9b4004160 con 0x7fa9c4105410 2023-07-14T19:09:31.561 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:30.804+0000 7fa9c1d9b700 1 -- 172.21.15.40:0/625326914 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa9b40042c0 con 0x7fa9c4105410 2023-07-14T19:09:31.561 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:30.804+0000 7fa9c1d9b700 1 -- 172.21.15.40:0/625326914 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa9b4011a00 con 0x7fa9c4105410 2023-07-14T19:09:31.561 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:30.804+0000 7fa9c9119700 1 -- 172.21.15.40:0/625326914 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa9c4105410 msgr2=0x7fa9c4105820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:09:31.561 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:30.804+0000 7fa9c9119700 1 --2- 172.21.15.40:0/625326914 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa9c4105410 0x7fa9c4105820 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fa9b40098a0 tx=0x7fa9b4009bb0 comp rx=0 tx=0).stop 2023-07-14T19:09:31.561 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:30.804+0000 7fa9c9119700 1 -- 172.21.15.40:0/625326914 shutdown_connections 2023-07-14T19:09:31.561 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:30.804+0000 7fa9c9119700 1 --2- 172.21.15.40:0/625326914 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa9c4105410 0x7fa9c4105820 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:09:31.561 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:30.804+0000 7fa9c9119700 1 -- 172.21.15.40:0/625326914 >> 172.21.15.40:0/625326914 conn(0x7fa9c4100a80 msgr2=0x7fa9c4102e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:09:31.562 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:30.804+0000 7fa9c9119700 1 -- 172.21.15.40:0/625326914 shutdown_connections 2023-07-14T19:09:31.562 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:30.804+0000 7fa9c9119700 1 -- 172.21.15.40:0/625326914 wait complete. 2023-07-14T19:09:31.562 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:30.805+0000 7fa9c9119700 1 Processor -- start 2023-07-14T19:09:31.562 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:30.805+0000 7fa9c9119700 1 -- start start 2023-07-14T19:09:31.562 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:30.805+0000 7fa9c9119700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa9c4105410 0x7fa9c4199160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:09:31.562 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:30.805+0000 7fa9c9119700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9c41996a0 con 0x7fa9c4105410 2023-07-14T19:09:31.563 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:30.805+0000 7fa9c2d9d700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa9c4105410 0x7fa9c4199160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:09:31.563 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:30.805+0000 7fa9c2d9d700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa9c4105410 0x7fa9c4199160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:59912/0 (socket says 172.21.15.40:59912) 2023-07-14T19:09:31.563 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:30.805+0000 7fa9c2d9d700 1 -- 172.21.15.40:0/3809942249 learned_addr learned my addr 172.21.15.40:0/3809942249 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-07-14T19:09:31.563 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:30.806+0000 7fa9c2d9d700 1 -- 172.21.15.40:0/3809942249 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9b4009550 con 0x7fa9c4105410 2023-07-14T19:09:31.563 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:30.806+0000 7fa9c2d9d700 1 --2- 172.21.15.40:0/3809942249 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa9c4105410 0x7fa9c4199160 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fa9b4009fb0 tx=0x7fa9b4011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:09:31.563 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:30.806+0000 7fa9abfff700 1 -- 172.21.15.40:0/3809942249 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa9b4003a40 con 0x7fa9c4105410 2023-07-14T19:09:31.564 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:30.806+0000 7fa9c9119700 1 -- 172.21.15.40:0/3809942249 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa9c41998a0 con 0x7fa9c4105410 2023-07-14T19:09:31.564 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:30.806+0000 7fa9abfff700 1 -- 172.21.15.40:0/3809942249 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa9b4003ba0 con 0x7fa9c4105410 2023-07-14T19:09:31.564 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:30.806+0000 7fa9abfff700 1 -- 172.21.15.40:0/3809942249 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa9b401a750 con 0x7fa9c4105410 2023-07-14T19:09:31.564 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:30.806+0000 7fa9c9119700 1 -- 172.21.15.40:0/3809942249 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa9c4199cc0 con 0x7fa9c4105410 2023-07-14T19:09:31.564 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:30.807+0000 7fa9abfff700 1 -- 172.21.15.40:0/3809942249 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 8) v1 ==== 45483+0+0 (secure 0 0 0) 0x7fa9b401a9d0 con 0x7fa9c4105410 2023-07-14T19:09:31.564 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:30.807+0000 7fa9c9119700 1 -- 172.21.15.40:0/3809942249 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa9c4061c70 con 0x7fa9c4105410 2023-07-14T19:09:31.564 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:30.808+0000 7fa9abfff700 1 --2- 172.21.15.40:0/3809942249 >> [v2:172.21.15.40:6800/3698757264,v1:172.21.15.40:6801/3698757264] conn(0x7fa9ac0386c0 0x7fa9ac03ab70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:09:31.564 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:30.808+0000 7fa9abfff700 1 -- 172.21.15.40:0/3809942249 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7fa9b404c9b0 con 0x7fa9c4105410 2023-07-14T19:09:31.565 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:30.808+0000 7fa9c259c700 1 --2- 172.21.15.40:0/3809942249 >> [v2:172.21.15.40:6800/3698757264,v1:172.21.15.40:6801/3698757264] conn(0x7fa9ac0386c0 0x7fa9ac03ab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:09:31.565 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:30.808+0000 7fa9c259c700 1 --2- 172.21.15.40:0/3809942249 >> [v2:172.21.15.40:6800/3698757264,v1:172.21.15.40:6801/3698757264] conn(0x7fa9ac0386c0 0x7fa9ac03ab70 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fa9b8006fd0 tx=0x7fa9b8006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:09:31.565 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:30.812+0000 7fa9abfff700 1 -- 172.21.15.40:0/3809942249 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fa9b4018d20 con 0x7fa9c4105410 2023-07-14T19:09:31.565 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:31.173+0000 7fa9c9119700 1 -- 172.21.15.40:0/3809942249 --> [v2:172.21.15.40:6800/3698757264,v1:172.21.15.40:6801/3698757264] -- mgr_command(tid 0: {"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}) v1 -- 0x7fa9c4102d40 con 0x7fa9ac0386c0 2023-07-14T19:09:31.565 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:31.174+0000 7fa9abfff700 1 -- 172.21.15.40:0/3809942249 <== mgr.14120 v2:172.21.15.40:6800/3698757264 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+595 (secure 0 0 0) 0x7fa9c4102d40 con 0x7fa9ac0386c0 2023-07-14T19:09:31.565 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:31.176+0000 7fa9c9119700 1 -- 172.21.15.40:0/3809942249 >> [v2:172.21.15.40:6800/3698757264,v1:172.21.15.40:6801/3698757264] conn(0x7fa9ac0386c0 msgr2=0x7fa9ac03ab70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:09:31.565 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:31.176+0000 7fa9c9119700 1 --2- 172.21.15.40:0/3809942249 >> [v2:172.21.15.40:6800/3698757264,v1:172.21.15.40:6801/3698757264] conn(0x7fa9ac0386c0 0x7fa9ac03ab70 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fa9b8006fd0 tx=0x7fa9b8006e40 comp rx=0 tx=0).stop 2023-07-14T19:09:31.566 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:31.176+0000 7fa9c9119700 1 -- 172.21.15.40:0/3809942249 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa9c4105410 msgr2=0x7fa9c4199160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:09:31.566 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:31.176+0000 7fa9c9119700 1 --2- 172.21.15.40:0/3809942249 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa9c4105410 0x7fa9c4199160 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fa9b4009fb0 tx=0x7fa9b4011ad0 comp rx=0 tx=0).stop 2023-07-14T19:09:31.566 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:31.176+0000 7fa9c9119700 1 -- 172.21.15.40:0/3809942249 shutdown_connections 2023-07-14T19:09:31.566 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:31.176+0000 7fa9c9119700 1 --2- 172.21.15.40:0/3809942249 >> [v2:172.21.15.40:6800/3698757264,v1:172.21.15.40:6801/3698757264] conn(0x7fa9ac0386c0 0x7fa9ac03ab70 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:09:31.566 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:31.176+0000 7fa9c9119700 1 --2- 172.21.15.40:0/3809942249 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa9c4105410 0x7fa9c4199160 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:09:31.566 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:31.176+0000 7fa9c9119700 1 -- 172.21.15.40:0/3809942249 >> 172.21.15.40:0/3809942249 conn(0x7fa9c4100a80 msgr2=0x7fa9c41020e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:09:31.566 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:31.176+0000 7fa9c9119700 1 -- 172.21.15.40:0/3809942249 shutdown_connections 2023-07-14T19:09:31.567 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:31.176+0000 7fa9c9119700 1 -- 172.21.15.40:0/3809942249 wait complete. 2023-07-14T19:09:31.567 INFO:teuthology.orchestra.run.smithi040.stdout:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2023-07-14T19:09:31.567 INFO:teuthology.orchestra.run.smithi040.stdout:Adding key to root@localhost authorized_keys... 2023-07-14T19:09:31.567 INFO:teuthology.orchestra.run.smithi040.stdout:Adding host smithi040... 2023-07-14T19:09:32.688 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:09:32 smithi040 ceph-mon[96518]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T19:09:33.721 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:09:33 smithi040 ceph-mon[96518]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi040", "addr": "172.21.15.40", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T19:09:34.721 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:09:34 smithi040 ceph-mon[96518]: Deploying cephadm binary to smithi040 2023-07-14T19:09:35.512 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout Added host 'smithi040' with addr '172.21.15.40' 2023-07-14T19:09:35.512 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:32.021+0000 7fb115ee1700 1 Processor -- start 2023-07-14T19:09:35.512 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:32.021+0000 7fb115ee1700 1 -- start start 2023-07-14T19:09:35.512 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:32.022+0000 7fb115ee1700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb110105f00 0x7fb1101082e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:09:35.512 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:32.022+0000 7fb115ee1700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb110079d60 con 0x7fb110105f00 2023-07-14T19:09:35.513 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:32.022+0000 7fb10f7fe700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb110105f00 0x7fb1101082e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:09:35.513 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:32.022+0000 7fb10f7fe700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb110105f00 0x7fb1101082e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:59916/0 (socket says 172.21.15.40:59916) 2023-07-14T19:09:35.513 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:32.022+0000 7fb10f7fe700 1 -- 172.21.15.40:0/202060034 learned_addr learned my addr 172.21.15.40:0/202060034 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-07-14T19:09:35.513 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:32.022+0000 7fb10f7fe700 1 -- 172.21.15.40:0/202060034 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb110079ea0 con 0x7fb110105f00 2023-07-14T19:09:35.513 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:32.022+0000 7fb10f7fe700 1 --2- 172.21.15.40:0/202060034 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb110105f00 0x7fb1101082e0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fb1000098a0 tx=0x7fb100009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=819774b3f681e6db server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:09:35.513 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:32.022+0000 7fb10e7fc700 1 -- 172.21.15.40:0/202060034 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb100004160 con 0x7fb110105f00 2023-07-14T19:09:35.513 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:32.022+0000 7fb10e7fc700 1 -- 172.21.15.40:0/202060034 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb1000042c0 con 0x7fb110105f00 2023-07-14T19:09:35.514 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:32.022+0000 7fb10e7fc700 1 -- 172.21.15.40:0/202060034 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb100011a00 con 0x7fb110105f00 2023-07-14T19:09:35.514 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:32.023+0000 7fb115ee1700 1 -- 172.21.15.40:0/202060034 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb110105f00 msgr2=0x7fb1101082e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:09:35.514 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:32.023+0000 7fb115ee1700 1 --2- 172.21.15.40:0/202060034 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb110105f00 0x7fb1101082e0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fb1000098a0 tx=0x7fb100009bb0 comp rx=0 tx=0).stop 2023-07-14T19:09:35.514 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:32.023+0000 7fb115ee1700 1 -- 172.21.15.40:0/202060034 shutdown_connections 2023-07-14T19:09:35.514 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:32.023+0000 7fb115ee1700 1 --2- 172.21.15.40:0/202060034 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb110105f00 0x7fb1101082e0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:09:35.514 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:32.023+0000 7fb115ee1700 1 -- 172.21.15.40:0/202060034 >> 172.21.15.40:0/202060034 conn(0x7fb1100ff9a0 msgr2=0x7fb110101db0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:09:35.514 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:32.023+0000 7fb115ee1700 1 -- 172.21.15.40:0/202060034 shutdown_connections 2023-07-14T19:09:35.515 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:32.023+0000 7fb115ee1700 1 -- 172.21.15.40:0/202060034 wait complete. 2023-07-14T19:09:35.515 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:32.024+0000 7fb115ee1700 1 Processor -- start 2023-07-14T19:09:35.515 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:32.024+0000 7fb115ee1700 1 -- start start 2023-07-14T19:09:35.515 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:32.024+0000 7fb115ee1700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb110105f00 0x7fb110105860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:09:35.515 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:32.024+0000 7fb115ee1700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb110105da0 con 0x7fb110105f00 2023-07-14T19:09:35.515 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:32.024+0000 7fb10f7fe700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb110105f00 0x7fb110105860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:09:35.515 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:32.024+0000 7fb10f7fe700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb110105f00 0x7fb110105860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:59918/0 (socket says 172.21.15.40:59918) 2023-07-14T19:09:35.515 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:32.024+0000 7fb10f7fe700 1 -- 172.21.15.40:0/1389094775 learned_addr learned my addr 172.21.15.40:0/1389094775 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-07-14T19:09:35.516 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:32.024+0000 7fb10f7fe700 1 -- 172.21.15.40:0/1389094775 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb100009550 con 0x7fb110105f00 2023-07-14T19:09:35.516 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:32.025+0000 7fb10f7fe700 1 --2- 172.21.15.40:0/1389094775 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb110105f00 0x7fb110105860 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fb100006af0 tx=0x7fb100011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:09:35.516 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:32.025+0000 7fb10cff9700 1 -- 172.21.15.40:0/1389094775 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb100003a40 con 0x7fb110105f00 2023-07-14T19:09:35.516 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:32.025+0000 7fb10cff9700 1 -- 172.21.15.40:0/1389094775 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb100003ba0 con 0x7fb110105f00 2023-07-14T19:09:35.516 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:32.025+0000 7fb115ee1700 1 -- 172.21.15.40:0/1389094775 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb110103eb0 con 0x7fb110105f00 2023-07-14T19:09:35.516 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:32.025+0000 7fb10cff9700 1 -- 172.21.15.40:0/1389094775 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb10001a700 con 0x7fb110105f00 2023-07-14T19:09:35.516 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:32.025+0000 7fb115ee1700 1 -- 172.21.15.40:0/1389094775 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb1101042d0 con 0x7fb110105f00 2023-07-14T19:09:35.517 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:32.026+0000 7fb10cff9700 1 -- 172.21.15.40:0/1389094775 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 8) v1 ==== 45483+0+0 (secure 0 0 0) 0x7fb10001a980 con 0x7fb110105f00 2023-07-14T19:09:35.517 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:32.026+0000 7fb10cff9700 1 --2- 172.21.15.40:0/1389094775 >> [v2:172.21.15.40:6800/3698757264,v1:172.21.15.40:6801/3698757264] conn(0x7fb0fc038710 0x7fb0fc03abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:09:35.517 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:32.026+0000 7fb10cff9700 1 -- 172.21.15.40:0/1389094775 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7fb10004c8d0 con 0x7fb110105f00 2023-07-14T19:09:35.517 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:32.026+0000 7fb115ee1700 1 -- 172.21.15.40:0/1389094775 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb110061ce0 con 0x7fb110105f00 2023-07-14T19:09:35.517 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:32.026+0000 7fb10effd700 1 --2- 172.21.15.40:0/1389094775 >> [v2:172.21.15.40:6800/3698757264,v1:172.21.15.40:6801/3698757264] conn(0x7fb0fc038710 0x7fb0fc03abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:09:35.517 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:32.027+0000 7fb10effd700 1 --2- 172.21.15.40:0/1389094775 >> [v2:172.21.15.40:6800/3698757264,v1:172.21.15.40:6801/3698757264] conn(0x7fb0fc038710 0x7fb0fc03abc0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fb0f8006fd0 tx=0x7fb0f8006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:09:35.517 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:32.031+0000 7fb10cff9700 1 -- 172.21.15.40:0/1389094775 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fb110061ce0 con 0x7fb110105f00 2023-07-14T19:09:35.518 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:32.397+0000 7fb115ee1700 1 -- 172.21.15.40:0/1389094775 --> [v2:172.21.15.40:6800/3698757264,v1:172.21.15.40:6801/3698757264] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi040", "addr": "172.21.15.40", "target": ["mon-mgr", ""]}) v1 -- 0x7fb110104da0 con 0x7fb0fc038710 2023-07-14T19:09:35.518 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:35.155+0000 7fb10cff9700 1 -- 172.21.15.40:0/1389094775 <== mgr.14120 v2:172.21.15.40:6800/3698757264 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+48 (secure 0 0 0) 0x7fb0f8009e20 con 0x7fb0fc038710 2023-07-14T19:09:35.518 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:35.158+0000 7fb115ee1700 1 -- 172.21.15.40:0/1389094775 >> [v2:172.21.15.40:6800/3698757264,v1:172.21.15.40:6801/3698757264] conn(0x7fb0fc038710 msgr2=0x7fb0fc03abc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:09:35.518 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:35.158+0000 7fb115ee1700 1 --2- 172.21.15.40:0/1389094775 >> [v2:172.21.15.40:6800/3698757264,v1:172.21.15.40:6801/3698757264] conn(0x7fb0fc038710 0x7fb0fc03abc0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fb0f8006fd0 tx=0x7fb0f8006e40 comp rx=0 tx=0).stop 2023-07-14T19:09:35.518 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:35.158+0000 7fb115ee1700 1 -- 172.21.15.40:0/1389094775 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb110105f00 msgr2=0x7fb110105860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:09:35.518 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:35.158+0000 7fb115ee1700 1 --2- 172.21.15.40:0/1389094775 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb110105f00 0x7fb110105860 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fb100006af0 tx=0x7fb100011ad0 comp rx=0 tx=0).stop 2023-07-14T19:09:35.518 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:35.158+0000 7fb115ee1700 1 -- 172.21.15.40:0/1389094775 shutdown_connections 2023-07-14T19:09:35.518 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:35.158+0000 7fb115ee1700 1 --2- 172.21.15.40:0/1389094775 >> [v2:172.21.15.40:6800/3698757264,v1:172.21.15.40:6801/3698757264] conn(0x7fb0fc038710 0x7fb0fc03abc0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:09:35.519 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:35.158+0000 7fb115ee1700 1 --2- 172.21.15.40:0/1389094775 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb110105f00 0x7fb110105860 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:09:35.519 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:35.158+0000 7fb115ee1700 1 -- 172.21.15.40:0/1389094775 >> 172.21.15.40:0/1389094775 conn(0x7fb1100ff9a0 msgr2=0x7fb110100780 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:09:35.519 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:35.159+0000 7fb115ee1700 1 -- 172.21.15.40:0/1389094775 shutdown_connections 2023-07-14T19:09:35.519 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:35.159+0000 7fb115ee1700 1 -- 172.21.15.40:0/1389094775 wait complete. 2023-07-14T19:09:35.519 INFO:teuthology.orchestra.run.smithi040.stdout:Deploying unmanaged mon service... 2023-07-14T19:09:36.446 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:09:36 smithi040 ceph-mon[96518]: from='mgr.14120 172.21.15.40:0/1993184375' entity='mgr.x' 2023-07-14T19:09:36.446 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:09:36 smithi040 ceph-mon[96518]: Added host smithi040 2023-07-14T19:09:36.447 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:09:36 smithi040 ceph-mon[96518]: from='mgr.14120 172.21.15.40:0/1993184375' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T19:09:36.859 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout Scheduled mon update... 2023-07-14T19:09:36.859 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:36.020+0000 7f9882a88700 1 Processor -- start 2023-07-14T19:09:36.859 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:36.020+0000 7f9882a88700 1 -- start start 2023-07-14T19:09:36.859 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:36.020+0000 7f9882a88700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f987c1036e0 0x7f987c103af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:09:36.859 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:36.020+0000 7f9882a88700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f987c104030 con 0x7f987c1036e0 2023-07-14T19:09:36.860 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:36.020+0000 7f9880824700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f987c1036e0 0x7f987c103af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:09:36.860 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:36.020+0000 7f9880824700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f987c1036e0 0x7f987c103af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:59952/0 (socket says 172.21.15.40:59952) 2023-07-14T19:09:36.860 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:36.020+0000 7f9880824700 1 -- 172.21.15.40:0/3397173940 learned_addr learned my addr 172.21.15.40:0/3397173940 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-07-14T19:09:36.860 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:36.021+0000 7f9880824700 1 -- 172.21.15.40:0/3397173940 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f987c104170 con 0x7f987c1036e0 2023-07-14T19:09:36.860 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:36.021+0000 7f9880824700 1 --2- 172.21.15.40:0/3397173940 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f987c1036e0 0x7f987c103af0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f98700098a0 tx=0x7f9870009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=12583af673ad8339 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:09:36.860 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:36.021+0000 7f987b7fe700 1 -- 172.21.15.40:0/3397173940 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f9870004160 con 0x7f987c1036e0 2023-07-14T19:09:36.861 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:36.021+0000 7f987b7fe700 1 -- 172.21.15.40:0/3397173940 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f98700042c0 con 0x7f987c1036e0 2023-07-14T19:09:36.861 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:36.021+0000 7f987b7fe700 1 -- 172.21.15.40:0/3397173940 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f9870011a00 con 0x7f987c1036e0 2023-07-14T19:09:36.861 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:36.022+0000 7f9882a88700 1 -- 172.21.15.40:0/3397173940 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f987c1036e0 msgr2=0x7f987c103af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:09:36.861 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:36.022+0000 7f9882a88700 1 --2- 172.21.15.40:0/3397173940 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f987c1036e0 0x7f987c103af0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f98700098a0 tx=0x7f9870009bb0 comp rx=0 tx=0).stop 2023-07-14T19:09:36.861 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:36.022+0000 7f9882a88700 1 -- 172.21.15.40:0/3397173940 shutdown_connections 2023-07-14T19:09:36.861 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:36.022+0000 7f9882a88700 1 --2- 172.21.15.40:0/3397173940 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f987c1036e0 0x7f987c103af0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:09:36.861 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:36.022+0000 7f9882a88700 1 -- 172.21.15.40:0/3397173940 >> 172.21.15.40:0/3397173940 conn(0x7f987c0ff120 msgr2=0x7f987c101530 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:09:36.861 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:36.022+0000 7f9882a88700 1 -- 172.21.15.40:0/3397173940 shutdown_connections 2023-07-14T19:09:36.862 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:36.022+0000 7f9882a88700 1 -- 172.21.15.40:0/3397173940 wait complete. 2023-07-14T19:09:36.862 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:36.022+0000 7f9882a88700 1 Processor -- start 2023-07-14T19:09:36.862 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:36.022+0000 7f9882a88700 1 -- start start 2023-07-14T19:09:36.862 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:36.022+0000 7f9882a88700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f987c1036e0 0x7f987c199110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:09:36.862 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:36.022+0000 7f9882a88700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f987c199650 con 0x7f987c1036e0 2023-07-14T19:09:36.862 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:36.023+0000 7f9880824700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f987c1036e0 0x7f987c199110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:09:36.862 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:36.023+0000 7f9880824700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f987c1036e0 0x7f987c199110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:59954/0 (socket says 172.21.15.40:59954) 2023-07-14T19:09:36.863 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:36.023+0000 7f9880824700 1 -- 172.21.15.40:0/1949477001 learned_addr learned my addr 172.21.15.40:0/1949477001 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-07-14T19:09:36.863 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:36.023+0000 7f9880824700 1 -- 172.21.15.40:0/1949477001 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9870009550 con 0x7f987c1036e0 2023-07-14T19:09:36.863 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:36.023+0000 7f9880824700 1 --2- 172.21.15.40:0/1949477001 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f987c1036e0 0x7f987c199110 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f9870006af0 tx=0x7f9870011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:09:36.863 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:36.023+0000 7f9879ffb700 1 -- 172.21.15.40:0/1949477001 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f9870003a40 con 0x7f987c1036e0 2023-07-14T19:09:36.863 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:36.023+0000 7f9879ffb700 1 -- 172.21.15.40:0/1949477001 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9870003ba0 con 0x7f987c1036e0 2023-07-14T19:09:36.863 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:36.023+0000 7f9882a88700 1 -- 172.21.15.40:0/1949477001 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f987c199850 con 0x7f987c1036e0 2023-07-14T19:09:36.863 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:36.023+0000 7f9879ffb700 1 -- 172.21.15.40:0/1949477001 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f987001a700 con 0x7f987c1036e0 2023-07-14T19:09:36.864 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:36.023+0000 7f9882a88700 1 -- 172.21.15.40:0/1949477001 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f987c199c70 con 0x7f987c1036e0 2023-07-14T19:09:36.864 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:36.024+0000 7f9879ffb700 1 -- 172.21.15.40:0/1949477001 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 8) v1 ==== 45483+0+0 (secure 0 0 0) 0x7f987001a930 con 0x7f987c1036e0 2023-07-14T19:09:36.864 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:36.025+0000 7f9882a88700 1 -- 172.21.15.40:0/1949477001 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f987c061c70 con 0x7f987c1036e0 2023-07-14T19:09:36.864 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:36.025+0000 7f9879ffb700 1 --2- 172.21.15.40:0/1949477001 >> [v2:172.21.15.40:6800/3698757264,v1:172.21.15.40:6801/3698757264] conn(0x7f986c038760 0x7f986c03ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:09:36.864 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:36.025+0000 7f9879ffb700 1 -- 172.21.15.40:0/1949477001 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f987004c8b0 con 0x7f987c1036e0 2023-07-14T19:09:36.864 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:36.026+0000 7f987bfff700 1 --2- 172.21.15.40:0/1949477001 >> [v2:172.21.15.40:6800/3698757264,v1:172.21.15.40:6801/3698757264] conn(0x7f986c038760 0x7f986c03ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:09:36.864 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:36.026+0000 7f987bfff700 1 --2- 172.21.15.40:0/1949477001 >> [v2:172.21.15.40:6800/3698757264,v1:172.21.15.40:6801/3698757264] conn(0x7f986c038760 0x7f986c03ac10 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f9868006fd0 tx=0x7f9868006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:09:36.865 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:36.030+0000 7f9879ffb700 1 -- 172.21.15.40:0/1949477001 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f9870018b80 con 0x7f987c1036e0 2023-07-14T19:09:36.865 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:36.410+0000 7f9882a88700 1 -- 172.21.15.40:0/1949477001 --> [v2:172.21.15.40:6800/3698757264,v1:172.21.15.40:6801/3698757264] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}) v1 -- 0x7f987c1005d0 con 0x7f986c038760 2023-07-14T19:09:36.865 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:36.417+0000 7f9879ffb700 1 -- 172.21.15.40:0/1949477001 <== mgr.14120 v2:172.21.15.40:6800/3698757264 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f987c1005d0 con 0x7f986c038760 2023-07-14T19:09:36.865 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:36.420+0000 7f9882a88700 1 -- 172.21.15.40:0/1949477001 >> [v2:172.21.15.40:6800/3698757264,v1:172.21.15.40:6801/3698757264] conn(0x7f986c038760 msgr2=0x7f986c03ac10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:09:36.865 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:36.420+0000 7f9882a88700 1 --2- 172.21.15.40:0/1949477001 >> [v2:172.21.15.40:6800/3698757264,v1:172.21.15.40:6801/3698757264] conn(0x7f986c038760 0x7f986c03ac10 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f9868006fd0 tx=0x7f9868006e40 comp rx=0 tx=0).stop 2023-07-14T19:09:36.865 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:36.420+0000 7f9882a88700 1 -- 172.21.15.40:0/1949477001 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f987c1036e0 msgr2=0x7f987c199110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:09:36.865 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:36.420+0000 7f9882a88700 1 --2- 172.21.15.40:0/1949477001 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f987c1036e0 0x7f987c199110 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f9870006af0 tx=0x7f9870011ad0 comp rx=0 tx=0).stop 2023-07-14T19:09:36.866 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:36.420+0000 7f9882a88700 1 -- 172.21.15.40:0/1949477001 shutdown_connections 2023-07-14T19:09:36.866 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:36.420+0000 7f9882a88700 1 --2- 172.21.15.40:0/1949477001 >> [v2:172.21.15.40:6800/3698757264,v1:172.21.15.40:6801/3698757264] conn(0x7f986c038760 0x7f986c03ac10 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:09:36.866 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:36.420+0000 7f9882a88700 1 --2- 172.21.15.40:0/1949477001 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f987c1036e0 0x7f987c199110 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:09:36.866 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:36.420+0000 7f9882a88700 1 -- 172.21.15.40:0/1949477001 >> 172.21.15.40:0/1949477001 conn(0x7f987c0ff120 msgr2=0x7f987c0ffec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:09:36.866 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:36.420+0000 7f9882a88700 1 -- 172.21.15.40:0/1949477001 shutdown_connections 2023-07-14T19:09:36.866 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:36.420+0000 7f9882a88700 1 -- 172.21.15.40:0/1949477001 wait complete. 2023-07-14T19:09:36.866 INFO:teuthology.orchestra.run.smithi040.stdout:Deploying unmanaged mgr service... 2023-07-14T19:09:37.721 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:09:37 smithi040 ceph-mon[96518]: from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T19:09:37.722 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:09:37 smithi040 ceph-mon[96518]: Saving service mon spec with placement count:5 2023-07-14T19:09:37.722 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:09:37 smithi040 ceph-mon[96518]: from='mgr.14120 172.21.15.40:0/1993184375' entity='mgr.x' 2023-07-14T19:09:38.306 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout Scheduled mgr update... 2023-07-14T19:09:38.306 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:37.484+0000 7f9abea99700 1 Processor -- start 2023-07-14T19:09:38.306 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:37.484+0000 7f9abea99700 1 -- start start 2023-07-14T19:09:38.306 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:37.485+0000 7f9abea99700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9ab81047f0 0x7f9ab8104c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:09:38.306 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:37.485+0000 7f9abea99700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ab8105140 con 0x7f9ab81047f0 2023-07-14T19:09:38.307 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:37.485+0000 7f9abc835700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9ab81047f0 0x7f9ab8104c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:09:38.307 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:37.485+0000 7f9abc835700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9ab81047f0 0x7f9ab8104c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:59958/0 (socket says 172.21.15.40:59958) 2023-07-14T19:09:38.307 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:37.485+0000 7f9abc835700 1 -- 172.21.15.40:0/88532573 learned_addr learned my addr 172.21.15.40:0/88532573 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-07-14T19:09:38.307 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:37.485+0000 7f9abc835700 1 -- 172.21.15.40:0/88532573 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ab8105280 con 0x7f9ab81047f0 2023-07-14T19:09:38.307 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:37.485+0000 7f9abc835700 1 --2- 172.21.15.40:0/88532573 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9ab81047f0 0x7f9ab8104c00 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f9aac0098a0 tx=0x7f9aac009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=96bfed7eca0509e3 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:09:38.307 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:37.485+0000 7f9ab77fe700 1 -- 172.21.15.40:0/88532573 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f9aac004160 con 0x7f9ab81047f0 2023-07-14T19:09:38.308 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:37.486+0000 7f9ab77fe700 1 -- 172.21.15.40:0/88532573 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9aac0042c0 con 0x7f9ab81047f0 2023-07-14T19:09:38.308 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:37.486+0000 7f9ab77fe700 1 -- 172.21.15.40:0/88532573 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f9aac011a00 con 0x7f9ab81047f0 2023-07-14T19:09:38.308 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:37.486+0000 7f9abea99700 1 -- 172.21.15.40:0/88532573 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9ab81047f0 msgr2=0x7f9ab8104c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:09:38.308 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:37.486+0000 7f9abea99700 1 --2- 172.21.15.40:0/88532573 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9ab81047f0 0x7f9ab8104c00 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f9aac0098a0 tx=0x7f9aac009bb0 comp rx=0 tx=0).stop 2023-07-14T19:09:38.308 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:37.486+0000 7f9abea99700 1 -- 172.21.15.40:0/88532573 shutdown_connections 2023-07-14T19:09:38.308 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:37.486+0000 7f9abea99700 1 --2- 172.21.15.40:0/88532573 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9ab81047f0 0x7f9ab8104c00 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:09:38.308 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:37.486+0000 7f9abea99700 1 -- 172.21.15.40:0/88532573 >> 172.21.15.40:0/88532573 conn(0x7f9ab80ff120 msgr2=0x7f9ab8101530 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:09:38.309 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:37.486+0000 7f9abea99700 1 -- 172.21.15.40:0/88532573 shutdown_connections 2023-07-14T19:09:38.309 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:37.486+0000 7f9abea99700 1 -- 172.21.15.40:0/88532573 wait complete. 2023-07-14T19:09:38.309 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:37.487+0000 7f9abea99700 1 Processor -- start 2023-07-14T19:09:38.309 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:37.487+0000 7f9abea99700 1 -- start start 2023-07-14T19:09:38.309 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:37.487+0000 7f9abea99700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9ab81047f0 0x7f9ab818f460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:09:38.310 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:37.487+0000 7f9abea99700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ab818f9a0 con 0x7f9ab81047f0 2023-07-14T19:09:38.310 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:37.487+0000 7f9abc835700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9ab81047f0 0x7f9ab818f460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:09:38.310 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:37.487+0000 7f9abc835700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9ab81047f0 0x7f9ab818f460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:59960/0 (socket says 172.21.15.40:59960) 2023-07-14T19:09:38.310 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:37.487+0000 7f9abc835700 1 -- 172.21.15.40:0/1172466810 learned_addr learned my addr 172.21.15.40:0/1172466810 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-07-14T19:09:38.310 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:37.488+0000 7f9abc835700 1 -- 172.21.15.40:0/1172466810 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9aac009550 con 0x7f9ab81047f0 2023-07-14T19:09:38.310 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:37.488+0000 7f9abc835700 1 --2- 172.21.15.40:0/1172466810 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9ab81047f0 0x7f9ab818f460 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f9aac009fb0 tx=0x7f9aac011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:09:38.310 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:37.488+0000 7f9ab5ffb700 1 -- 172.21.15.40:0/1172466810 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f9aac003930 con 0x7f9ab81047f0 2023-07-14T19:09:38.310 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:37.488+0000 7f9ab5ffb700 1 -- 172.21.15.40:0/1172466810 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9aac003a90 con 0x7f9ab81047f0 2023-07-14T19:09:38.311 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:37.488+0000 7f9abea99700 1 -- 172.21.15.40:0/1172466810 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9ab818dab0 con 0x7f9ab81047f0 2023-07-14T19:09:38.311 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:37.488+0000 7f9ab5ffb700 1 -- 172.21.15.40:0/1172466810 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f9aac0195e0 con 0x7f9ab81047f0 2023-07-14T19:09:38.311 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:37.488+0000 7f9abea99700 1 -- 172.21.15.40:0/1172466810 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ab818ded0 con 0x7f9ab81047f0 2023-07-14T19:09:38.311 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:37.489+0000 7f9ab5ffb700 1 -- 172.21.15.40:0/1172466810 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 8) v1 ==== 45483+0+0 (secure 0 0 0) 0x7f9aac019850 con 0x7f9ab81047f0 2023-07-14T19:09:38.311 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:37.489+0000 7f9abea99700 1 -- 172.21.15.40:0/1172466810 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9ab8061c70 con 0x7f9ab81047f0 2023-07-14T19:09:38.311 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:37.490+0000 7f9ab5ffb700 1 --2- 172.21.15.40:0/1172466810 >> [v2:172.21.15.40:6800/3698757264,v1:172.21.15.40:6801/3698757264] conn(0x7f9aa8038710 0x7f9aa803abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:09:38.311 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:37.490+0000 7f9ab5ffb700 1 -- 172.21.15.40:0/1172466810 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f9aac04c7b0 con 0x7f9ab81047f0 2023-07-14T19:09:38.312 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:37.490+0000 7f9ab7fff700 1 --2- 172.21.15.40:0/1172466810 >> [v2:172.21.15.40:6800/3698757264,v1:172.21.15.40:6801/3698757264] conn(0x7f9aa8038710 0x7f9aa803abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:09:38.312 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:37.490+0000 7f9ab7fff700 1 --2- 172.21.15.40:0/1172466810 >> [v2:172.21.15.40:6800/3698757264,v1:172.21.15.40:6801/3698757264] conn(0x7f9aa8038710 0x7f9aa803abc0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f9aa4006fd0 tx=0x7f9aa4006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:09:38.312 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:37.495+0000 7f9ab5ffb700 1 -- 172.21.15.40:0/1172466810 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f9aac02b340 con 0x7f9ab81047f0 2023-07-14T19:09:38.312 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:37.597+0000 7f9ab5ffb700 1 -- 172.21.15.40:0/1172466810 <== mon.0 v2:172.21.15.40:3300/0 7 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f9aac04c1c0 con 0x7f9ab81047f0 2023-07-14T19:09:38.312 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:37.891+0000 7f9abea99700 1 -- 172.21.15.40:0/1172466810 --> [v2:172.21.15.40:6800/3698757264,v1:172.21.15.40:6801/3698757264] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}) v1 -- 0x7f9a98000d90 con 0x7f9aa8038710 2023-07-14T19:09:38.312 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:37.898+0000 7f9ab5ffb700 1 -- 172.21.15.40:0/1172466810 <== mgr.14120 v2:172.21.15.40:6800/3698757264 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f9a98000d90 con 0x7f9aa8038710 2023-07-14T19:09:38.312 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:37.900+0000 7f9abea99700 1 -- 172.21.15.40:0/1172466810 >> [v2:172.21.15.40:6800/3698757264,v1:172.21.15.40:6801/3698757264] conn(0x7f9aa8038710 msgr2=0x7f9aa803abc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:09:38.313 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:37.900+0000 7f9abea99700 1 --2- 172.21.15.40:0/1172466810 >> [v2:172.21.15.40:6800/3698757264,v1:172.21.15.40:6801/3698757264] conn(0x7f9aa8038710 0x7f9aa803abc0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f9aa4006fd0 tx=0x7f9aa4006e40 comp rx=0 tx=0).stop 2023-07-14T19:09:38.313 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:37.900+0000 7f9abea99700 1 -- 172.21.15.40:0/1172466810 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9ab81047f0 msgr2=0x7f9ab818f460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:09:38.313 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:37.900+0000 7f9abea99700 1 --2- 172.21.15.40:0/1172466810 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9ab81047f0 0x7f9ab818f460 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f9aac009fb0 tx=0x7f9aac011ad0 comp rx=0 tx=0).stop 2023-07-14T19:09:38.313 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:37.900+0000 7f9abea99700 1 -- 172.21.15.40:0/1172466810 shutdown_connections 2023-07-14T19:09:38.313 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:37.900+0000 7f9abea99700 1 --2- 172.21.15.40:0/1172466810 >> [v2:172.21.15.40:6800/3698757264,v1:172.21.15.40:6801/3698757264] conn(0x7f9aa8038710 0x7f9aa803abc0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:09:38.313 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:37.900+0000 7f9abea99700 1 --2- 172.21.15.40:0/1172466810 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9ab81047f0 0x7f9ab818f460 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:09:38.313 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:37.900+0000 7f9abea99700 1 -- 172.21.15.40:0/1172466810 >> 172.21.15.40:0/1172466810 conn(0x7f9ab80ff120 msgr2=0x7f9ab8103c80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:09:38.313 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:37.901+0000 7f9abea99700 1 -- 172.21.15.40:0/1172466810 shutdown_connections 2023-07-14T19:09:38.314 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:37.901+0000 7f9abea99700 1 -- 172.21.15.40:0/1172466810 wait complete. 2023-07-14T19:09:38.971 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:09:38 smithi040 ceph-mon[96518]: from='mgr.14120 172.21.15.40:0/1993184375' entity='mgr.x' 2023-07-14T19:09:38.972 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:09:38 smithi040 ceph-mon[96518]: from='client.14144 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T19:09:38.972 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:09:38 smithi040 ceph-mon[96518]: Saving service mgr spec with placement count:2 2023-07-14T19:09:38.972 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:09:38 smithi040 ceph-mon[96518]: from='mgr.14120 172.21.15.40:0/1993184375' entity='mgr.x' 2023-07-14T19:09:38.973 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:09:38 smithi040 ceph-mon[96518]: from='mgr.14120 172.21.15.40:0/1993184375' entity='mgr.x' 2023-07-14T19:09:39.648 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:38.798+0000 7f395c2d1700 1 Processor -- start 2023-07-14T19:09:39.649 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:38.798+0000 7f395c2d1700 1 -- start start 2023-07-14T19:09:39.649 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:38.798+0000 7f395c2d1700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f39541058d0 0x7f3954105ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:09:39.649 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:38.798+0000 7f395c2d1700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3954106220 con 0x7f39541058d0 2023-07-14T19:09:39.649 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:38.798+0000 7f395a06d700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f39541058d0 0x7f3954105ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:09:39.649 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:38.798+0000 7f395a06d700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f39541058d0 0x7f3954105ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:59964/0 (socket says 172.21.15.40:59964) 2023-07-14T19:09:39.649 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:38.798+0000 7f395a06d700 1 -- 172.21.15.40:0/1185355647 learned_addr learned my addr 172.21.15.40:0/1185355647 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-07-14T19:09:39.649 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:38.799+0000 7f395a06d700 1 -- 172.21.15.40:0/1185355647 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3954106360 con 0x7f39541058d0 2023-07-14T19:09:39.650 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:38.799+0000 7f395a06d700 1 --2- 172.21.15.40:0/1185355647 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f39541058d0 0x7f3954105ce0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f39500098a0 tx=0x7f3950009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=22869457d8d04e81 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:09:39.650 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:38.799+0000 7f395906b700 1 -- 172.21.15.40:0/1185355647 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3950004160 con 0x7f39541058d0 2023-07-14T19:09:39.650 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:38.799+0000 7f395906b700 1 -- 172.21.15.40:0/1185355647 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f39500042c0 con 0x7f39541058d0 2023-07-14T19:09:39.650 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:38.799+0000 7f395906b700 1 -- 172.21.15.40:0/1185355647 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f39500119b0 con 0x7f39541058d0 2023-07-14T19:09:39.650 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:38.800+0000 7f395c2d1700 1 -- 172.21.15.40:0/1185355647 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f39541058d0 msgr2=0x7f3954105ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:09:39.650 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:38.800+0000 7f395c2d1700 1 --2- 172.21.15.40:0/1185355647 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f39541058d0 0x7f3954105ce0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f39500098a0 tx=0x7f3950009bb0 comp rx=0 tx=0).stop 2023-07-14T19:09:39.650 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:38.800+0000 7f395c2d1700 1 -- 172.21.15.40:0/1185355647 shutdown_connections 2023-07-14T19:09:39.651 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:38.800+0000 7f395c2d1700 1 --2- 172.21.15.40:0/1185355647 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f39541058d0 0x7f3954105ce0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:09:39.651 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:38.800+0000 7f395c2d1700 1 -- 172.21.15.40:0/1185355647 >> 172.21.15.40:0/1185355647 conn(0x7f39540ff120 msgr2=0x7f3954101530 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:09:39.651 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:38.800+0000 7f395c2d1700 1 -- 172.21.15.40:0/1185355647 shutdown_connections 2023-07-14T19:09:39.651 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:38.800+0000 7f395c2d1700 1 -- 172.21.15.40:0/1185355647 wait complete. 2023-07-14T19:09:39.651 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:38.801+0000 7f395c2d1700 1 Processor -- start 2023-07-14T19:09:39.651 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:38.801+0000 7f395c2d1700 1 -- start start 2023-07-14T19:09:39.651 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:38.801+0000 7f395c2d1700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f39541058d0 0x7f39541990e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:09:39.652 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:38.801+0000 7f395c2d1700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3954199620 con 0x7f39541058d0 2023-07-14T19:09:39.652 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:38.801+0000 7f395a06d700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f39541058d0 0x7f39541990e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:09:39.652 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:38.801+0000 7f395a06d700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f39541058d0 0x7f39541990e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:59966/0 (socket says 172.21.15.40:59966) 2023-07-14T19:09:39.652 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:38.801+0000 7f395a06d700 1 -- 172.21.15.40:0/2267584748 learned_addr learned my addr 172.21.15.40:0/2267584748 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-07-14T19:09:39.652 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:38.801+0000 7f395a06d700 1 -- 172.21.15.40:0/2267584748 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3950009550 con 0x7f39541058d0 2023-07-14T19:09:39.652 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:38.802+0000 7f395a06d700 1 --2- 172.21.15.40:0/2267584748 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f39541058d0 0x7f39541990e0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f3950003e60 tx=0x7f3950011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:09:39.652 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:38.802+0000 7f394b7fe700 1 -- 172.21.15.40:0/2267584748 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3950003970 con 0x7f39541058d0 2023-07-14T19:09:39.652 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:38.802+0000 7f395c2d1700 1 -- 172.21.15.40:0/2267584748 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3954199820 con 0x7f39541058d0 2023-07-14T19:09:39.653 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:38.802+0000 7f394b7fe700 1 -- 172.21.15.40:0/2267584748 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3950003ad0 con 0x7f39541058d0 2023-07-14T19:09:39.653 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:38.802+0000 7f394b7fe700 1 -- 172.21.15.40:0/2267584748 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f39500195e0 con 0x7f39541058d0 2023-07-14T19:09:39.653 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:38.802+0000 7f395c2d1700 1 -- 172.21.15.40:0/2267584748 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3954199c40 con 0x7f39541058d0 2023-07-14T19:09:39.653 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:38.803+0000 7f394b7fe700 1 -- 172.21.15.40:0/2267584748 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 8) v1 ==== 45483+0+0 (secure 0 0 0) 0x7f39500198a0 con 0x7f39541058d0 2023-07-14T19:09:39.653 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:38.803+0000 7f395c2d1700 1 -- 172.21.15.40:0/2267584748 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3954061c70 con 0x7f39541058d0 2023-07-14T19:09:39.653 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:38.803+0000 7f394b7fe700 1 --2- 172.21.15.40:0/2267584748 >> [v2:172.21.15.40:6800/3698757264,v1:172.21.15.40:6801/3698757264] conn(0x7f39400386c0 0x7f394003ab70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:09:39.653 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:38.803+0000 7f394b7fe700 1 -- 172.21.15.40:0/2267584748 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f395004c8f0 con 0x7f39541058d0 2023-07-14T19:09:39.654 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:38.803+0000 7f395986c700 1 --2- 172.21.15.40:0/2267584748 >> [v2:172.21.15.40:6800/3698757264,v1:172.21.15.40:6801/3698757264] conn(0x7f39400386c0 0x7f394003ab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:09:39.654 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:38.804+0000 7f395986c700 1 --2- 172.21.15.40:0/2267584748 >> [v2:172.21.15.40:6800/3698757264,v1:172.21.15.40:6801/3698757264] conn(0x7f39400386c0 0x7f394003ab70 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f3944006fd0 tx=0x7f3944006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:09:39.654 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:38.808+0000 7f394b7fe700 1 -- 172.21.15.40:0/2267584748 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f395002b340 con 0x7f39541058d0 2023-07-14T19:09:39.654 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:39.167+0000 7f395c2d1700 1 -- 172.21.15.40:0/2267584748 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/container_init}] v 0) v1 -- 0x7f395419caa0 con 0x7f39541058d0 2023-07-14T19:09:39.654 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:39.170+0000 7f394b7fe700 1 -- 172.21.15.40:0/2267584748 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/container_init}]=0 v7)=0 v7) v1 ==== 142+0+0 (secure 0 0 0) 0x7f3950018b40 con 0x7f39541058d0 2023-07-14T19:09:39.654 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:39.172+0000 7f395c2d1700 1 -- 172.21.15.40:0/2267584748 >> [v2:172.21.15.40:6800/3698757264,v1:172.21.15.40:6801/3698757264] conn(0x7f39400386c0 msgr2=0x7f394003ab70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:09:39.654 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:39.172+0000 7f395c2d1700 1 --2- 172.21.15.40:0/2267584748 >> [v2:172.21.15.40:6800/3698757264,v1:172.21.15.40:6801/3698757264] conn(0x7f39400386c0 0x7f394003ab70 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f3944006fd0 tx=0x7f3944006e40 comp rx=0 tx=0).stop 2023-07-14T19:09:39.654 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:39.172+0000 7f395c2d1700 1 -- 172.21.15.40:0/2267584748 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f39541058d0 msgr2=0x7f39541990e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:09:39.655 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:39.172+0000 7f395c2d1700 1 --2- 172.21.15.40:0/2267584748 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f39541058d0 0x7f39541990e0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f3950003e60 tx=0x7f3950011aa0 comp rx=0 tx=0).stop 2023-07-14T19:09:39.655 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:39.172+0000 7f395c2d1700 1 -- 172.21.15.40:0/2267584748 shutdown_connections 2023-07-14T19:09:39.655 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:39.172+0000 7f395c2d1700 1 --2- 172.21.15.40:0/2267584748 >> [v2:172.21.15.40:6800/3698757264,v1:172.21.15.40:6801/3698757264] conn(0x7f39400386c0 0x7f394003ab70 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:09:39.655 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:39.172+0000 7f395c2d1700 1 --2- 172.21.15.40:0/2267584748 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f39541058d0 0x7f39541990e0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:09:39.655 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:39.172+0000 7f395c2d1700 1 -- 172.21.15.40:0/2267584748 >> 172.21.15.40:0/2267584748 conn(0x7f39540ff120 msgr2=0x7f3954103d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:09:39.655 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:39.173+0000 7f395c2d1700 1 -- 172.21.15.40:0/2267584748 shutdown_connections 2023-07-14T19:09:39.655 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:39.173+0000 7f395c2d1700 1 -- 172.21.15.40:0/2267584748 wait complete. 2023-07-14T19:09:40.221 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:09:40 smithi040 ceph-mon[96518]: from='client.? 172.21.15.40:0/2267584748' entity='client.admin' 2023-07-14T19:09:40.222 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:09:40 smithi040 ceph-mon[96518]: from='mgr.14120 172.21.15.40:0/1993184375' entity='mgr.x' 2023-07-14T19:09:41.098 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:40.275+0000 7f5271c41700 1 Processor -- start 2023-07-14T19:09:41.099 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:40.275+0000 7f5271c41700 1 -- start start 2023-07-14T19:09:41.099 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:40.275+0000 7f5271c41700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f526c105f40 0x7f526c108320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:09:41.099 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:40.275+0000 7f5271c41700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f526c079d20 con 0x7f526c105f40 2023-07-14T19:09:41.100 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:40.276+0000 7f526b7fe700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f526c105f40 0x7f526c108320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:09:41.100 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:40.276+0000 7f526b7fe700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f526c105f40 0x7f526c108320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:59970/0 (socket says 172.21.15.40:59970) 2023-07-14T19:09:41.100 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:40.276+0000 7f526b7fe700 1 -- 172.21.15.40:0/1321219393 learned_addr learned my addr 172.21.15.40:0/1321219393 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-07-14T19:09:41.101 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:40.276+0000 7f526b7fe700 1 -- 172.21.15.40:0/1321219393 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f526c079e60 con 0x7f526c105f40 2023-07-14T19:09:41.101 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:40.276+0000 7f526b7fe700 1 --2- 172.21.15.40:0/1321219393 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f526c105f40 0x7f526c108320 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f525c0098a0 tx=0x7f525c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f1d3d11c4b3fd1ee server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:09:41.102 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:40.276+0000 7f526a7fc700 1 -- 172.21.15.40:0/1321219393 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f525c004160 con 0x7f526c105f40 2023-07-14T19:09:41.102 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:40.276+0000 7f526a7fc700 1 -- 172.21.15.40:0/1321219393 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f525c0042c0 con 0x7f526c105f40 2023-07-14T19:09:41.102 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:40.276+0000 7f526a7fc700 1 -- 172.21.15.40:0/1321219393 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f525c0119b0 con 0x7f526c105f40 2023-07-14T19:09:41.102 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:40.277+0000 7f5271c41700 1 -- 172.21.15.40:0/1321219393 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f526c105f40 msgr2=0x7f526c108320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:09:41.102 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:40.277+0000 7f5271c41700 1 --2- 172.21.15.40:0/1321219393 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f526c105f40 0x7f526c108320 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f525c0098a0 tx=0x7f525c009bb0 comp rx=0 tx=0).stop 2023-07-14T19:09:41.103 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:40.277+0000 7f5271c41700 1 -- 172.21.15.40:0/1321219393 shutdown_connections 2023-07-14T19:09:41.103 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:40.277+0000 7f5271c41700 1 --2- 172.21.15.40:0/1321219393 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f526c105f40 0x7f526c108320 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:09:41.103 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:40.277+0000 7f5271c41700 1 -- 172.21.15.40:0/1321219393 >> 172.21.15.40:0/1321219393 conn(0x7f526c0ff9a0 msgr2=0x7f526c101db0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:09:41.103 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:40.277+0000 7f5271c41700 1 -- 172.21.15.40:0/1321219393 shutdown_connections 2023-07-14T19:09:41.103 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:40.277+0000 7f5271c41700 1 -- 172.21.15.40:0/1321219393 wait complete. 2023-07-14T19:09:41.103 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:40.278+0000 7f5271c41700 1 Processor -- start 2023-07-14T19:09:41.103 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:40.278+0000 7f5271c41700 1 -- start start 2023-07-14T19:09:41.104 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:40.278+0000 7f5271c41700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f526c105f40 0x7f526c199170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:09:41.104 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:40.278+0000 7f5271c41700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f526c1996b0 con 0x7f526c105f40 2023-07-14T19:09:41.104 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:40.279+0000 7f526b7fe700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f526c105f40 0x7f526c199170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:09:41.104 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:40.279+0000 7f526b7fe700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f526c105f40 0x7f526c199170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:59972/0 (socket says 172.21.15.40:59972) 2023-07-14T19:09:41.104 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:40.279+0000 7f526b7fe700 1 -- 172.21.15.40:0/351228473 learned_addr learned my addr 172.21.15.40:0/351228473 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-07-14T19:09:41.104 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:40.279+0000 7f526b7fe700 1 -- 172.21.15.40:0/351228473 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f525c009550 con 0x7f526c105f40 2023-07-14T19:09:41.105 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:40.279+0000 7f526b7fe700 1 --2- 172.21.15.40:0/351228473 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f526c105f40 0x7f526c199170 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f525c003e60 tx=0x7f525c011a30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:09:41.105 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:40.279+0000 7f5268ff9700 1 -- 172.21.15.40:0/351228473 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f525c003a60 con 0x7f526c105f40 2023-07-14T19:09:41.105 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:40.279+0000 7f5271c41700 1 -- 172.21.15.40:0/351228473 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f526c199910 con 0x7f526c105f40 2023-07-14T19:09:41.105 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:40.279+0000 7f5268ff9700 1 -- 172.21.15.40:0/351228473 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f525c003bc0 con 0x7f526c105f40 2023-07-14T19:09:41.105 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:40.279+0000 7f5268ff9700 1 -- 172.21.15.40:0/351228473 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f525c0196c0 con 0x7f526c105f40 2023-07-14T19:09:41.105 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:40.280+0000 7f5271c41700 1 -- 172.21.15.40:0/351228473 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f526c199d90 con 0x7f526c105f40 2023-07-14T19:09:41.105 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:40.280+0000 7f5268ff9700 1 -- 172.21.15.40:0/351228473 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 8) v1 ==== 45483+0+0 (secure 0 0 0) 0x7f525c019980 con 0x7f526c105f40 2023-07-14T19:09:41.106 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:40.281+0000 7f5271c41700 1 -- 172.21.15.40:0/351228473 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f526c061c70 con 0x7f526c105f40 2023-07-14T19:09:41.106 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:40.281+0000 7f5268ff9700 1 --2- 172.21.15.40:0/351228473 >> [v2:172.21.15.40:6800/3698757264,v1:172.21.15.40:6801/3698757264] conn(0x7f52540386f0 0x7f525403aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:09:41.106 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:40.281+0000 7f5268ff9700 1 -- 172.21.15.40:0/351228473 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f525c01a840 con 0x7f526c105f40 2023-07-14T19:09:41.106 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:40.281+0000 7f526affd700 1 --2- 172.21.15.40:0/351228473 >> [v2:172.21.15.40:6800/3698757264,v1:172.21.15.40:6801/3698757264] conn(0x7f52540386f0 0x7f525403aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:09:41.106 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:40.281+0000 7f526affd700 1 --2- 172.21.15.40:0/351228473 >> [v2:172.21.15.40:6800/3698757264,v1:172.21.15.40:6801/3698757264] conn(0x7f52540386f0 0x7f525403aba0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f5260006fd0 tx=0x7f5260006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:09:41.106 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:40.286+0000 7f5268ff9700 1 -- 172.21.15.40:0/351228473 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f525c0109d0 con 0x7f526c105f40 2023-07-14T19:09:41.106 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:40.661+0000 7f5271c41700 1 -- 172.21.15.40:0/351228473 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command([{prefix=config set, name=mgr/dashboard/ssl_server_port}] v 0) v1 -- 0x7f526c103d80 con 0x7f526c105f40 2023-07-14T19:09:41.107 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:40.667+0000 7f5268ff9700 1 -- 172.21.15.40:0/351228473 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/dashboard/ssl_server_port}]=0 v8)=0 v8) v1 ==== 130+0+0 (secure 0 0 0) 0x7f525c02be80 con 0x7f526c105f40 2023-07-14T19:09:41.107 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:40.669+0000 7f5271c41700 1 -- 172.21.15.40:0/351228473 >> [v2:172.21.15.40:6800/3698757264,v1:172.21.15.40:6801/3698757264] conn(0x7f52540386f0 msgr2=0x7f525403aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:09:41.107 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:40.669+0000 7f5271c41700 1 --2- 172.21.15.40:0/351228473 >> [v2:172.21.15.40:6800/3698757264,v1:172.21.15.40:6801/3698757264] conn(0x7f52540386f0 0x7f525403aba0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f5260006fd0 tx=0x7f5260006e40 comp rx=0 tx=0).stop 2023-07-14T19:09:41.107 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:40.669+0000 7f5271c41700 1 -- 172.21.15.40:0/351228473 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f526c105f40 msgr2=0x7f526c199170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:09:41.107 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:40.669+0000 7f5271c41700 1 --2- 172.21.15.40:0/351228473 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f526c105f40 0x7f526c199170 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f525c003e60 tx=0x7f525c011a30 comp rx=0 tx=0).stop 2023-07-14T19:09:41.107 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:40.669+0000 7f5271c41700 1 -- 172.21.15.40:0/351228473 shutdown_connections 2023-07-14T19:09:41.108 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:40.669+0000 7f5271c41700 1 --2- 172.21.15.40:0/351228473 >> [v2:172.21.15.40:6800/3698757264,v1:172.21.15.40:6801/3698757264] conn(0x7f52540386f0 0x7f525403aba0 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:09:41.108 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:40.669+0000 7f5271c41700 1 --2- 172.21.15.40:0/351228473 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f526c105f40 0x7f526c199170 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:09:41.108 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:40.669+0000 7f5271c41700 1 -- 172.21.15.40:0/351228473 >> 172.21.15.40:0/351228473 conn(0x7f526c0ff9a0 msgr2=0x7f526c101090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:09:41.108 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:40.670+0000 7f5271c41700 1 -- 172.21.15.40:0/351228473 shutdown_connections 2023-07-14T19:09:41.108 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:40.670+0000 7f5271c41700 1 -- 172.21.15.40:0/351228473 wait complete. 2023-07-14T19:09:41.108 INFO:teuthology.orchestra.run.smithi040.stdout:Enabling the dashboard module... 2023-07-14T19:09:41.675 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:09:41 smithi040 ceph-mon[96518]: from='client.? 172.21.15.40:0/351228473' entity='client.admin' 2023-07-14T19:09:41.676 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:09:41 smithi040 ceph-mon[96518]: from='mgr.14120 172.21.15.40:0/1993184375' entity='mgr.x' 2023-07-14T19:09:42.971 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:09:42 smithi040 ceph-mon[96518]: from='client.? 172.21.15.40:0/258159801' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2023-07-14T19:09:42.972 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Jul 14 19:09:42 smithi040 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-mgr-x[96762]: ignoring --setuser ceph since I am not root 2023-07-14T19:09:42.972 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Jul 14 19:09:42 smithi040 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-mgr-x[96762]: ignoring --setgroup ceph since I am not root 2023-07-14T19:09:42.972 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Jul 14 19:09:42 smithi040 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-mgr-x[96762]: 2023-07-14T19:09:42.875+0000 7f3030c92200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-07-14T19:09:43.143 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:41.657+0000 7f322f8c7700 1 Processor -- start 2023-07-14T19:09:43.147 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:41.657+0000 7f322f8c7700 1 -- start start 2023-07-14T19:09:43.148 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:41.657+0000 7f322f8c7700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f32281036e0 0x7f3228103af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:09:43.148 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:41.657+0000 7f322f8c7700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3228104030 con 0x7f32281036e0 2023-07-14T19:09:43.148 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:41.658+0000 7f322d663700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f32281036e0 0x7f3228103af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:09:43.148 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:41.658+0000 7f322d663700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f32281036e0 0x7f3228103af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:59976/0 (socket says 172.21.15.40:59976) 2023-07-14T19:09:43.148 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:41.658+0000 7f322d663700 1 -- 172.21.15.40:0/1720328434 learned_addr learned my addr 172.21.15.40:0/1720328434 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-07-14T19:09:43.148 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:41.658+0000 7f322d663700 1 -- 172.21.15.40:0/1720328434 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3228104170 con 0x7f32281036e0 2023-07-14T19:09:43.149 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:41.659+0000 7f322d663700 1 --2- 172.21.15.40:0/1720328434 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f32281036e0 0x7f3228103af0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f32240098a0 tx=0x7f3224009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=224b02374bd16f5f server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:09:43.149 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:41.659+0000 7f321ffff700 1 -- 172.21.15.40:0/1720328434 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3224004160 con 0x7f32281036e0 2023-07-14T19:09:43.149 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:41.659+0000 7f321ffff700 1 -- 172.21.15.40:0/1720328434 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f32240042c0 con 0x7f32281036e0 2023-07-14T19:09:43.149 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:41.659+0000 7f321ffff700 1 -- 172.21.15.40:0/1720328434 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3224011960 con 0x7f32281036e0 2023-07-14T19:09:43.149 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:41.660+0000 7f322f8c7700 1 -- 172.21.15.40:0/1720328434 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f32281036e0 msgr2=0x7f3228103af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:09:43.149 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:41.660+0000 7f322f8c7700 1 --2- 172.21.15.40:0/1720328434 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f32281036e0 0x7f3228103af0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f32240098a0 tx=0x7f3224009bb0 comp rx=0 tx=0).stop 2023-07-14T19:09:43.149 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:41.660+0000 7f322f8c7700 1 -- 172.21.15.40:0/1720328434 shutdown_connections 2023-07-14T19:09:43.149 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:41.660+0000 7f322f8c7700 1 --2- 172.21.15.40:0/1720328434 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f32281036e0 0x7f3228103af0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:09:43.150 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:41.660+0000 7f322f8c7700 1 -- 172.21.15.40:0/1720328434 >> 172.21.15.40:0/1720328434 conn(0x7f32280ff120 msgr2=0x7f3228101530 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:09:43.150 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:41.660+0000 7f322f8c7700 1 -- 172.21.15.40:0/1720328434 shutdown_connections 2023-07-14T19:09:43.150 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:41.660+0000 7f322f8c7700 1 -- 172.21.15.40:0/1720328434 wait complete. 2023-07-14T19:09:43.150 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:41.661+0000 7f322f8c7700 1 Processor -- start 2023-07-14T19:09:43.150 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:41.661+0000 7f322f8c7700 1 -- start start 2023-07-14T19:09:43.150 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:41.661+0000 7f322f8c7700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f32281036e0 0x7f3228106070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:09:43.150 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:41.661+0000 7f322f8c7700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32281065b0 con 0x7f32281036e0 2023-07-14T19:09:43.151 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:41.661+0000 7f322d663700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f32281036e0 0x7f3228106070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:09:43.151 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:41.661+0000 7f322d663700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f32281036e0 0x7f3228106070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:59978/0 (socket says 172.21.15.40:59978) 2023-07-14T19:09:43.151 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:41.661+0000 7f322d663700 1 -- 172.21.15.40:0/258159801 learned_addr learned my addr 172.21.15.40:0/258159801 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-07-14T19:09:43.151 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:41.662+0000 7f322d663700 1 -- 172.21.15.40:0/258159801 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3224009550 con 0x7f32281036e0 2023-07-14T19:09:43.151 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:41.662+0000 7f322d663700 1 --2- 172.21.15.40:0/258159801 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f32281036e0 0x7f3228106070 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f3224000c00 tx=0x7f3224011f20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:09:43.151 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:41.662+0000 7f321e7fc700 1 -- 172.21.15.40:0/258159801 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f32240036a0 con 0x7f32281036e0 2023-07-14T19:09:43.151 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:41.662+0000 7f321e7fc700 1 -- 172.21.15.40:0/258159801 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3224024440 con 0x7f32281036e0 2023-07-14T19:09:43.151 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:41.662+0000 7f322f8c7700 1 -- 172.21.15.40:0/258159801 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f32281046c0 con 0x7f32281036e0 2023-07-14T19:09:43.152 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:41.662+0000 7f321e7fc700 1 -- 172.21.15.40:0/258159801 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f322401a430 con 0x7f32281036e0 2023-07-14T19:09:43.152 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:41.662+0000 7f322f8c7700 1 -- 172.21.15.40:0/258159801 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3228104ae0 con 0x7f32281036e0 2023-07-14T19:09:43.152 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:41.663+0000 7f321e7fc700 1 -- 172.21.15.40:0/258159801 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 8) v1 ==== 45483+0+0 (secure 0 0 0) 0x7f322401a6f0 con 0x7f32281036e0 2023-07-14T19:09:43.152 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:41.663+0000 7f322f8c7700 1 -- 172.21.15.40:0/258159801 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3228061c70 con 0x7f32281036e0 2023-07-14T19:09:43.152 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:41.663+0000 7f321e7fc700 1 --2- 172.21.15.40:0/258159801 >> [v2:172.21.15.40:6800/3698757264,v1:172.21.15.40:6801/3698757264] conn(0x7f3214038740 0x7f321403abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:09:43.152 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:41.664+0000 7f321e7fc700 1 -- 172.21.15.40:0/258159801 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f322404c4c0 con 0x7f32281036e0 2023-07-14T19:09:43.153 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:41.664+0000 7f322ce62700 1 --2- 172.21.15.40:0/258159801 >> [v2:172.21.15.40:6800/3698757264,v1:172.21.15.40:6801/3698757264] conn(0x7f3214038740 0x7f321403abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:09:43.153 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:41.664+0000 7f322ce62700 1 --2- 172.21.15.40:0/258159801 >> [v2:172.21.15.40:6800/3698757264,v1:172.21.15.40:6801/3698757264] conn(0x7f3214038740 0x7f321403abf0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f3218006fd0 tx=0x7f3218006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:09:43.153 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:41.669+0000 7f321e7fc700 1 -- 172.21.15.40:0/258159801 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f3224018db0 con 0x7f32281036e0 2023-07-14T19:09:43.153 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:42.139+0000 7f322f8c7700 1 -- 172.21.15.40:0/258159801 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "dashboard"} v 0) v1 -- 0x7f3228107360 con 0x7f32281036e0 2023-07-14T19:09:43.153 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:42.676+0000 7f321e7fc700 1 -- 172.21.15.40:0/258159801 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mgrmap(e 9) v1 ==== 45496+0+0 (secure 0 0 0) 0x7f322402ccd0 con 0x7f32281036e0 2023-07-14T19:09:43.153 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:42.677+0000 7f321e7fc700 1 -- 172.21.15.40:0/258159801 <== mon.0 v2:172.21.15.40:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "dashboard"}]=0 v9) v1 ==== 88+0+0 (secure 0 0 0) 0x7f322400bc20 con 0x7f32281036e0 2023-07-14T19:09:43.153 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:42.680+0000 7f322f8c7700 1 -- 172.21.15.40:0/258159801 >> [v2:172.21.15.40:6800/3698757264,v1:172.21.15.40:6801/3698757264] conn(0x7f3214038740 msgr2=0x7f321403abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:09:43.153 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:42.680+0000 7f322f8c7700 1 --2- 172.21.15.40:0/258159801 >> [v2:172.21.15.40:6800/3698757264,v1:172.21.15.40:6801/3698757264] conn(0x7f3214038740 0x7f321403abf0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f3218006fd0 tx=0x7f3218006e40 comp rx=0 tx=0).stop 2023-07-14T19:09:43.154 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:42.680+0000 7f322f8c7700 1 -- 172.21.15.40:0/258159801 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f32281036e0 msgr2=0x7f3228106070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:09:43.154 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:42.680+0000 7f322f8c7700 1 --2- 172.21.15.40:0/258159801 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f32281036e0 0x7f3228106070 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f3224000c00 tx=0x7f3224011f20 comp rx=0 tx=0).stop 2023-07-14T19:09:43.154 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:42.680+0000 7f322f8c7700 1 -- 172.21.15.40:0/258159801 shutdown_connections 2023-07-14T19:09:43.154 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:42.680+0000 7f322f8c7700 1 --2- 172.21.15.40:0/258159801 >> [v2:172.21.15.40:6800/3698757264,v1:172.21.15.40:6801/3698757264] conn(0x7f3214038740 0x7f321403abf0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:09:43.154 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:42.680+0000 7f322f8c7700 1 --2- 172.21.15.40:0/258159801 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f32281036e0 0x7f3228106070 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:09:43.154 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:42.680+0000 7f322f8c7700 1 -- 172.21.15.40:0/258159801 >> 172.21.15.40:0/258159801 conn(0x7f32280ff120 msgr2=0x7f32280ffec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:09:43.154 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:42.680+0000 7f322f8c7700 1 -- 172.21.15.40:0/258159801 shutdown_connections 2023-07-14T19:09:43.155 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:42.680+0000 7f322f8c7700 1 -- 172.21.15.40:0/258159801 wait complete. 2023-07-14T19:09:43.364 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Jul 14 19:09:42 smithi040 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-mgr-x[96762]: 2023-07-14T19:09:42.972+0000 7f3030c92200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-07-14T19:09:43.364 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Jul 14 19:09:43 smithi040 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-mgr-x[96762]: 2023-07-14T19:09:43.080+0000 7f3030c92200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-07-14T19:09:43.364 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Jul 14 19:09:43 smithi040 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-mgr-x[96762]: 2023-07-14T19:09:43.196+0000 7f3030c92200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-07-14T19:09:43.678 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Jul 14 19:09:43 smithi040 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-mgr-x[96762]: 2023-07-14T19:09:43.409+0000 7f3030c92200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-07-14T19:09:43.971 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:09:43 smithi040 ceph-mon[96518]: from='client.? 172.21.15.40:0/258159801' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2023-07-14T19:09:43.972 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:09:43 smithi040 ceph-mon[96518]: mgrmap e9: x(active, since 18s) 2023-07-14T19:09:43.972 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Jul 14 19:09:43 smithi040 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-mgr-x[96762]: 2023-07-14T19:09:43.815+0000 7f3030c92200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-07-14T19:09:43.972 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Jul 14 19:09:43 smithi040 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-mgr-x[96762]: 2023-07-14T19:09:43.924+0000 7f3030c92200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-07-14T19:09:44.332 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Jul 14 19:09:44 smithi040 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-mgr-x[96762]: 2023-07-14T19:09:44.076+0000 7f3030c92200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-07-14T19:09:44.333 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Jul 14 19:09:44 smithi040 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-mgr-x[96762]: 2023-07-14T19:09:44.193+0000 7f3030c92200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-07-14T19:09:44.595 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout { 2023-07-14T19:09:44.596 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "epoch": 9, 2023-07-14T19:09:44.596 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "available": true, 2023-07-14T19:09:44.596 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "active_name": "x", 2023-07-14T19:09:44.596 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "num_standby": 0 2023-07-14T19:09:44.596 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout } 2023-07-14T19:09:44.597 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:43.683+0000 7ffa7f473700 1 Processor -- start 2023-07-14T19:09:44.597 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:43.683+0000 7ffa7f473700 1 -- start start 2023-07-14T19:09:44.597 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:43.683+0000 7ffa7f473700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ffa7807bdc0 0x7ffa7807a2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:09:44.597 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:43.684+0000 7ffa7f473700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa7807a800 con 0x7ffa7807bdc0 2023-07-14T19:09:44.597 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:43.684+0000 7ffa7d20f700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ffa7807bdc0 0x7ffa7807a2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:09:44.597 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:43.684+0000 7ffa7d20f700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ffa7807bdc0 0x7ffa7807a2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:59992/0 (socket says 172.21.15.40:59992) 2023-07-14T19:09:44.597 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:43.684+0000 7ffa7d20f700 1 -- 172.21.15.40:0/3972674190 learned_addr learned my addr 172.21.15.40:0/3972674190 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-07-14T19:09:44.598 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:43.684+0000 7ffa7d20f700 1 -- 172.21.15.40:0/3972674190 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa7807a940 con 0x7ffa7807bdc0 2023-07-14T19:09:44.598 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:43.684+0000 7ffa7d20f700 1 --2- 172.21.15.40:0/3972674190 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ffa7807bdc0 0x7ffa7807a2c0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7ffa740098a0 tx=0x7ffa74009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=49a9201b56526af5 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:09:44.598 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:43.684+0000 7ffa6ffff700 1 -- 172.21.15.40:0/3972674190 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ffa74004160 con 0x7ffa7807bdc0 2023-07-14T19:09:44.598 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:43.684+0000 7ffa6ffff700 1 -- 172.21.15.40:0/3972674190 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ffa740042c0 con 0x7ffa7807bdc0 2023-07-14T19:09:44.598 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:43.684+0000 7ffa6ffff700 1 -- 172.21.15.40:0/3972674190 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ffa740119b0 con 0x7ffa7807bdc0 2023-07-14T19:09:44.598 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:43.685+0000 7ffa7f473700 1 -- 172.21.15.40:0/3972674190 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ffa7807bdc0 msgr2=0x7ffa7807a2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:09:44.598 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:43.685+0000 7ffa7f473700 1 --2- 172.21.15.40:0/3972674190 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ffa7807bdc0 0x7ffa7807a2c0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7ffa740098a0 tx=0x7ffa74009bb0 comp rx=0 tx=0).stop 2023-07-14T19:09:44.599 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:43.685+0000 7ffa7f473700 1 -- 172.21.15.40:0/3972674190 shutdown_connections 2023-07-14T19:09:44.599 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:43.685+0000 7ffa7f473700 1 --2- 172.21.15.40:0/3972674190 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ffa7807bdc0 0x7ffa7807a2c0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:09:44.599 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:43.685+0000 7ffa7f473700 1 -- 172.21.15.40:0/3972674190 >> 172.21.15.40:0/3972674190 conn(0x7ffa78100ac0 msgr2=0x7ffa78102ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:09:44.599 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:43.685+0000 7ffa7f473700 1 -- 172.21.15.40:0/3972674190 shutdown_connections 2023-07-14T19:09:44.599 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:43.685+0000 7ffa7f473700 1 -- 172.21.15.40:0/3972674190 wait complete. 2023-07-14T19:09:44.599 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:43.686+0000 7ffa7f473700 1 Processor -- start 2023-07-14T19:09:44.600 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:43.686+0000 7ffa7f473700 1 -- start start 2023-07-14T19:09:44.600 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:43.686+0000 7ffa7f473700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ffa7807bdc0 0x7ffa7819b3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:09:44.600 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:43.686+0000 7ffa7f473700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa7819b8f0 con 0x7ffa7807bdc0 2023-07-14T19:09:44.600 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:43.687+0000 7ffa7d20f700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ffa7807bdc0 0x7ffa7819b3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:09:44.600 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:43.687+0000 7ffa7d20f700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ffa7807bdc0 0x7ffa7819b3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:59994/0 (socket says 172.21.15.40:59994) 2023-07-14T19:09:44.600 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:43.687+0000 7ffa7d20f700 1 -- 172.21.15.40:0/4245065822 learned_addr learned my addr 172.21.15.40:0/4245065822 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-07-14T19:09:44.600 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:43.687+0000 7ffa7d20f700 1 -- 172.21.15.40:0/4245065822 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa74009550 con 0x7ffa7807bdc0 2023-07-14T19:09:44.601 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:43.687+0000 7ffa7d20f700 1 --2- 172.21.15.40:0/4245065822 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ffa7807bdc0 0x7ffa7819b3b0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7ffa74006af0 tx=0x7ffa74011a10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:09:44.601 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:43.687+0000 7ffa6e7fc700 1 -- 172.21.15.40:0/4245065822 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ffa74003a40 con 0x7ffa7807bdc0 2023-07-14T19:09:44.601 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:43.687+0000 7ffa6e7fc700 1 -- 172.21.15.40:0/4245065822 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ffa74003ba0 con 0x7ffa7807bdc0 2023-07-14T19:09:44.601 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:43.687+0000 7ffa7f473700 1 -- 172.21.15.40:0/4245065822 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffa7819baf0 con 0x7ffa7807bdc0 2023-07-14T19:09:44.601 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:43.687+0000 7ffa6e7fc700 1 -- 172.21.15.40:0/4245065822 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ffa74019690 con 0x7ffa7807bdc0 2023-07-14T19:09:44.601 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:43.687+0000 7ffa7f473700 1 -- 172.21.15.40:0/4245065822 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffa7819bf10 con 0x7ffa7807bdc0 2023-07-14T19:09:44.601 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:43.688+0000 7ffa6e7fc700 1 -- 172.21.15.40:0/4245065822 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 9) v1 ==== 45496+0+0 (secure 0 0 0) 0x7ffa74019950 con 0x7ffa7807bdc0 2023-07-14T19:09:44.602 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:43.688+0000 7ffa7f473700 1 -- 172.21.15.40:0/4245065822 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffa78061c70 con 0x7ffa7807bdc0 2023-07-14T19:09:44.602 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:43.688+0000 7ffa6e7fc700 1 --2- 172.21.15.40:0/4245065822 >> [v2:172.21.15.40:6800/3698757264,v1:172.21.15.40:6801/3698757264] conn(0x7ffa64038710 0x7ffa6403abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:09:44.602 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:43.689+0000 7ffa6e7fc700 1 -- 172.21.15.40:0/4245065822 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7ffa7404c6d0 con 0x7ffa7807bdc0 2023-07-14T19:09:44.602 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:43.689+0000 7ffa7ca0e700 1 -- 172.21.15.40:0/4245065822 >> [v2:172.21.15.40:6800/3698757264,v1:172.21.15.40:6801/3698757264] conn(0x7ffa64038710 msgr2=0x7ffa6403abc0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.40:6800/3698757264 2023-07-14T19:09:44.602 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:43.689+0000 7ffa7ca0e700 1 --2- 172.21.15.40:0/4245065822 >> [v2:172.21.15.40:6800/3698757264,v1:172.21.15.40:6801/3698757264] conn(0x7ffa64038710 0x7ffa6403abc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-07-14T19:09:44.602 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:43.693+0000 7ffa6e7fc700 1 -- 172.21.15.40:0/4245065822 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7ffa7402dcd0 con 0x7ffa7807bdc0 2023-07-14T19:09:44.602 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:43.890+0000 7ffa7ca0e700 1 -- 172.21.15.40:0/4245065822 >> [v2:172.21.15.40:6800/3698757264,v1:172.21.15.40:6801/3698757264] conn(0x7ffa64038710 msgr2=0x7ffa6403abc0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.40:6800/3698757264 2023-07-14T19:09:44.602 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:43.890+0000 7ffa7ca0e700 1 --2- 172.21.15.40:0/4245065822 >> [v2:172.21.15.40:6800/3698757264,v1:172.21.15.40:6801/3698757264] conn(0x7ffa64038710 0x7ffa6403abc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-07-14T19:09:44.603 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:44.167+0000 7ffa7f473700 1 -- 172.21.15.40:0/4245065822 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7ffa7819f000 con 0x7ffa7807bdc0 2023-07-14T19:09:44.603 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:44.167+0000 7ffa6e7fc700 1 -- 172.21.15.40:0/4245065822 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v9) v1 ==== 56+0+88 (secure 0 0 0) 0x7ffa74023390 con 0x7ffa7807bdc0 2023-07-14T19:09:44.603 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:44.169+0000 7ffa7f473700 1 -- 172.21.15.40:0/4245065822 >> [v2:172.21.15.40:6800/3698757264,v1:172.21.15.40:6801/3698757264] conn(0x7ffa64038710 msgr2=0x7ffa6403abc0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-07-14T19:09:44.603 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:44.170+0000 7ffa7f473700 1 --2- 172.21.15.40:0/4245065822 >> [v2:172.21.15.40:6800/3698757264,v1:172.21.15.40:6801/3698757264] conn(0x7ffa64038710 0x7ffa6403abc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:09:44.603 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:44.170+0000 7ffa7f473700 1 -- 172.21.15.40:0/4245065822 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ffa7807bdc0 msgr2=0x7ffa7819b3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:09:44.603 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:44.170+0000 7ffa7f473700 1 --2- 172.21.15.40:0/4245065822 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ffa7807bdc0 0x7ffa7819b3b0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7ffa74006af0 tx=0x7ffa74011a10 comp rx=0 tx=0).stop 2023-07-14T19:09:44.603 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:44.170+0000 7ffa7f473700 1 -- 172.21.15.40:0/4245065822 shutdown_connections 2023-07-14T19:09:44.604 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:44.170+0000 7ffa7f473700 1 --2- 172.21.15.40:0/4245065822 >> [v2:172.21.15.40:6800/3698757264,v1:172.21.15.40:6801/3698757264] conn(0x7ffa64038710 0x7ffa6403abc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:09:44.604 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:44.170+0000 7ffa7f473700 1 --2- 172.21.15.40:0/4245065822 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ffa7807bdc0 0x7ffa7819b3b0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:09:44.604 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:44.170+0000 7ffa7f473700 1 -- 172.21.15.40:0/4245065822 >> 172.21.15.40:0/4245065822 conn(0x7ffa78100ac0 msgr2=0x7ffa78102140 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:09:44.604 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:44.170+0000 7ffa7f473700 1 -- 172.21.15.40:0/4245065822 shutdown_connections 2023-07-14T19:09:44.604 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:44.170+0000 7ffa7f473700 1 -- 172.21.15.40:0/4245065822 wait complete. 2023-07-14T19:09:44.604 INFO:teuthology.orchestra.run.smithi040.stdout:Waiting for the mgr to restart... 2023-07-14T19:09:44.604 INFO:teuthology.orchestra.run.smithi040.stdout:Waiting for mgr epoch 9... 2023-07-14T19:09:44.859 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:09:44 smithi040 ceph-mon[96518]: from='client.? 172.21.15.40:0/4245065822' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2023-07-14T19:09:44.859 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Jul 14 19:09:44 smithi040 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-mgr-x[96762]: 2023-07-14T19:09:44.725+0000 7f3030c92200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-07-14T19:09:45.721 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Jul 14 19:09:45 smithi040 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-mgr-x[96762]: 2023-07-14T19:09:45.423+0000 7f3030c92200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-07-14T19:09:45.722 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Jul 14 19:09:45 smithi040 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-mgr-x[96762]: 2023-07-14T19:09:45.615+0000 7f3030c92200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-07-14T19:09:46.909 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Jul 14 19:09:46 smithi040 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-mgr-x[96762]: 2023-07-14T19:09:46.577+0000 7f3030c92200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-07-14T19:09:46.910 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Jul 14 19:09:46 smithi040 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-mgr-x[96762]: 2023-07-14T19:09:46.688+0000 7f3030c92200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-07-14T19:09:47.221 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Jul 14 19:09:46 smithi040 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-mgr-x[96762]: 2023-07-14T19:09:46.907+0000 7f3030c92200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-07-14T19:09:47.527 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Jul 14 19:09:47 smithi040 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-mgr-x[96762]: 2023-07-14T19:09:47.256+0000 7f3030c92200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-07-14T19:09:47.971 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Jul 14 19:09:47 smithi040 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-mgr-x[96762]: 2023-07-14T19:09:47.525+0000 7f3030c92200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-07-14T19:09:47.972 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Jul 14 19:09:47 smithi040 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-mgr-x[96762]: 2023-07-14T19:09:47.708+0000 7f3030c92200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-07-14T19:09:48.471 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Jul 14 19:09:48 smithi040 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-mgr-x[96762]: 2023-07-14T19:09:48.040+0000 7f3030c92200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-07-14T19:09:49.222 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Jul 14 19:09:48 smithi040 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-mgr-x[96762]: 2023-07-14T19:09:48.863+0000 7f3030c92200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-07-14T19:09:49.222 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Jul 14 19:09:49 smithi040 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-mgr-x[96762]: 2023-07-14T19:09:49.021+0000 7f3030c92200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-07-14T19:09:49.678 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Jul 14 19:09:49 smithi040 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-mgr-x[96762]: 2023-07-14T19:09:49.266+0000 7f3030c92200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-07-14T19:09:49.679 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Jul 14 19:09:49 smithi040 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-mgr-x[96762]: 2023-07-14T19:09:49.404+0000 7f3030c92200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-07-14T19:09:49.972 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Jul 14 19:09:49 smithi040 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-mgr-x[96762]: 2023-07-14T19:09:49.675+0000 7f3030c92200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-07-14T19:09:50.971 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Jul 14 19:09:50 smithi040 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-mgr-x[96762]: 2023-07-14T19:09:50.717+0000 7f3030c92200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-07-14T19:09:50.972 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:09:50 smithi040 ceph-mon[96518]: Active manager daemon x restarted 2023-07-14T19:09:50.972 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:09:50 smithi040 ceph-mon[96518]: Activating manager daemon x 2023-07-14T19:09:50.972 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:09:50 smithi040 ceph-mon[96518]: osdmap e3: 0 total, 0 up, 0 in 2023-07-14T19:09:50.973 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:09:50 smithi040 ceph-mon[96518]: mgrmap e10: x(active, starting, since 0.00339232s) 2023-07-14T19:09:50.973 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:09:50 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-07-14T19:09:50.973 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:09:50 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2023-07-14T19:09:50.973 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:09:50 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2023-07-14T19:09:50.973 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:09:50 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2023-07-14T19:09:50.973 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:09:50 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2023-07-14T19:09:50.973 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:09:50 smithi040 ceph-mon[96518]: Manager daemon x is now available 2023-07-14T19:09:50.974 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:09:50 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T19:09:50.974 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:09:50 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T19:09:52.222 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:09:51 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2023-07-14T19:09:52.222 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:09:51 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2023-07-14T19:09:52.222 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:09:51 smithi040 ceph-mon[96518]: [14/Jul/2023:19:09:51] ENGINE Bus STARTING 2023-07-14T19:09:52.222 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:09:51 smithi040 ceph-mon[96518]: from='client.14156 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2023-07-14T19:09:52.223 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:09:51 smithi040 ceph-mon[96518]: mgrmap e11: x(active, since 1.00642s) 2023-07-14T19:09:52.454 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout { 2023-07-14T19:09:52.454 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 11, 2023-07-14T19:09:52.455 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "initialized": true 2023-07-14T19:09:52.455 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout } 2023-07-14T19:09:52.455 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:45.099+0000 7f0da0f97700 1 Processor -- start 2023-07-14T19:09:52.455 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:45.099+0000 7f0da0f97700 1 -- start start 2023-07-14T19:09:52.455 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:45.099+0000 7f0da0f97700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0d9c105f00 0x7f0d9c1082e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:09:52.460 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:45.099+0000 7f0da0f97700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d9c079d60 con 0x7f0d9c105f00 2023-07-14T19:09:52.461 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:45.100+0000 7f0d9a59c700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0d9c105f00 0x7f0d9c1082e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:09:52.461 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:45.100+0000 7f0d9a59c700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0d9c105f00 0x7f0d9c1082e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:60002/0 (socket says 172.21.15.40:60002) 2023-07-14T19:09:52.461 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:45.100+0000 7f0d9a59c700 1 -- 172.21.15.40:0/3903598740 learned_addr learned my addr 172.21.15.40:0/3903598740 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-07-14T19:09:52.462 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:45.100+0000 7f0d9a59c700 1 -- 172.21.15.40:0/3903598740 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d9c079ea0 con 0x7f0d9c105f00 2023-07-14T19:09:52.462 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:45.100+0000 7f0d9a59c700 1 --2- 172.21.15.40:0/3903598740 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0d9c105f00 0x7f0d9c1082e0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f0d8c0098a0 tx=0x7f0d8c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8619114d971b7ac server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:09:52.462 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:45.100+0000 7f0d9959a700 1 -- 172.21.15.40:0/3903598740 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0d8c004160 con 0x7f0d9c105f00 2023-07-14T19:09:52.462 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:45.100+0000 7f0d9959a700 1 -- 172.21.15.40:0/3903598740 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f0d8c0042c0 con 0x7f0d9c105f00 2023-07-14T19:09:52.462 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:45.100+0000 7f0d9959a700 1 -- 172.21.15.40:0/3903598740 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0d8c0119b0 con 0x7f0d9c105f00 2023-07-14T19:09:52.462 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:45.101+0000 7f0da0f97700 1 -- 172.21.15.40:0/3903598740 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0d9c105f00 msgr2=0x7f0d9c1082e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:09:52.463 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:45.101+0000 7f0da0f97700 1 --2- 172.21.15.40:0/3903598740 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0d9c105f00 0x7f0d9c1082e0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f0d8c0098a0 tx=0x7f0d8c009bb0 comp rx=0 tx=0).stop 2023-07-14T19:09:52.463 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:45.101+0000 7f0da0f97700 1 -- 172.21.15.40:0/3903598740 shutdown_connections 2023-07-14T19:09:52.463 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:45.101+0000 7f0da0f97700 1 --2- 172.21.15.40:0/3903598740 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0d9c105f00 0x7f0d9c1082e0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:09:52.463 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:45.101+0000 7f0da0f97700 1 -- 172.21.15.40:0/3903598740 >> 172.21.15.40:0/3903598740 conn(0x7f0d9c0ff9a0 msgr2=0x7f0d9c101db0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:09:52.463 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:45.101+0000 7f0da0f97700 1 -- 172.21.15.40:0/3903598740 shutdown_connections 2023-07-14T19:09:52.463 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:45.101+0000 7f0da0f97700 1 -- 172.21.15.40:0/3903598740 wait complete. 2023-07-14T19:09:52.463 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:45.102+0000 7f0da0f97700 1 Processor -- start 2023-07-14T19:09:52.464 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:45.102+0000 7f0da0f97700 1 -- start start 2023-07-14T19:09:52.464 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:45.102+0000 7f0da0f97700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0d9c105f00 0x7f0d9c199120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:09:52.464 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:45.102+0000 7f0da0f97700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d9c199660 con 0x7f0d9c105f00 2023-07-14T19:09:52.464 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:45.103+0000 7f0d9a59c700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0d9c105f00 0x7f0d9c199120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:09:52.464 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:45.103+0000 7f0d9a59c700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0d9c105f00 0x7f0d9c199120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:60004/0 (socket says 172.21.15.40:60004) 2023-07-14T19:09:52.464 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:45.103+0000 7f0d9a59c700 1 -- 172.21.15.40:0/1364520678 learned_addr learned my addr 172.21.15.40:0/1364520678 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-07-14T19:09:52.464 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:45.103+0000 7f0d9a59c700 1 -- 172.21.15.40:0/1364520678 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d8c009550 con 0x7f0d9c105f00 2023-07-14T19:09:52.465 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:45.103+0000 7f0d9a59c700 1 --2- 172.21.15.40:0/1364520678 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0d9c105f00 0x7f0d9c199120 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f0d8c003e60 tx=0x7f0d8c011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:09:52.465 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:45.103+0000 7f0d8b7fe700 1 -- 172.21.15.40:0/1364520678 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0d8c003970 con 0x7f0d9c105f00 2023-07-14T19:09:52.465 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:45.103+0000 7f0d8b7fe700 1 -- 172.21.15.40:0/1364520678 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f0d8c003ad0 con 0x7f0d9c105f00 2023-07-14T19:09:52.465 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:45.103+0000 7f0d8b7fe700 1 -- 172.21.15.40:0/1364520678 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0d8c0195e0 con 0x7f0d9c105f00 2023-07-14T19:09:52.465 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:45.103+0000 7f0da0f97700 1 -- 172.21.15.40:0/1364520678 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0d9c199860 con 0x7f0d9c105f00 2023-07-14T19:09:52.465 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:45.103+0000 7f0da0f97700 1 -- 172.21.15.40:0/1364520678 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0d9c199c80 con 0x7f0d9c105f00 2023-07-14T19:09:52.465 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:45.104+0000 7f0d8b7fe700 1 -- 172.21.15.40:0/1364520678 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 9) v1 ==== 45496+0+0 (secure 0 0 0) 0x7f0d8c019850 con 0x7f0d9c105f00 2023-07-14T19:09:52.466 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:45.105+0000 7f0d8b7fe700 1 --2- 172.21.15.40:0/1364520678 >> [v2:172.21.15.40:6800/3698757264,v1:172.21.15.40:6801/3698757264] conn(0x7f0d84038760 0x7f0d8403ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:09:52.466 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:45.105+0000 7f0d8b7fe700 1 -- 172.21.15.40:0/1364520678 --> [v2:172.21.15.40:6800/3698757264,v1:172.21.15.40:6801/3698757264] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f0d8403b320 con 0x7f0d84038760 2023-07-14T19:09:52.466 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:45.105+0000 7f0d8b7fe700 1 -- 172.21.15.40:0/1364520678 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f0d8c04c860 con 0x7f0d9c105f00 2023-07-14T19:09:52.466 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:45.105+0000 7f0d99d9b700 1 -- 172.21.15.40:0/1364520678 >> [v2:172.21.15.40:6800/3698757264,v1:172.21.15.40:6801/3698757264] conn(0x7f0d84038760 msgr2=0x7f0d8403ac10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.40:6800/3698757264 2023-07-14T19:09:52.466 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:45.105+0000 7f0d99d9b700 1 --2- 172.21.15.40:0/1364520678 >> [v2:172.21.15.40:6800/3698757264,v1:172.21.15.40:6801/3698757264] conn(0x7f0d84038760 0x7f0d8403ac10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-07-14T19:09:52.466 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:45.306+0000 7f0d99d9b700 1 -- 172.21.15.40:0/1364520678 >> [v2:172.21.15.40:6800/3698757264,v1:172.21.15.40:6801/3698757264] conn(0x7f0d84038760 msgr2=0x7f0d8403ac10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.40:6800/3698757264 2023-07-14T19:09:52.466 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:45.306+0000 7f0d99d9b700 1 --2- 172.21.15.40:0/1364520678 >> [v2:172.21.15.40:6800/3698757264,v1:172.21.15.40:6801/3698757264] conn(0x7f0d84038760 0x7f0d8403ac10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-07-14T19:09:52.467 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:45.707+0000 7f0d99d9b700 1 -- 172.21.15.40:0/1364520678 >> [v2:172.21.15.40:6800/3698757264,v1:172.21.15.40:6801/3698757264] conn(0x7f0d84038760 msgr2=0x7f0d8403ac10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.40:6800/3698757264 2023-07-14T19:09:52.467 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:45.707+0000 7f0d99d9b700 1 --2- 172.21.15.40:0/1364520678 >> [v2:172.21.15.40:6800/3698757264,v1:172.21.15.40:6801/3698757264] conn(0x7f0d84038760 0x7f0d8403ac10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-07-14T19:09:52.467 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:46.508+0000 7f0d99d9b700 1 -- 172.21.15.40:0/1364520678 >> [v2:172.21.15.40:6800/3698757264,v1:172.21.15.40:6801/3698757264] conn(0x7f0d84038760 msgr2=0x7f0d8403ac10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.40:6800/3698757264 2023-07-14T19:09:52.467 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:46.508+0000 7f0d99d9b700 1 --2- 172.21.15.40:0/1364520678 >> [v2:172.21.15.40:6800/3698757264,v1:172.21.15.40:6801/3698757264] conn(0x7f0d84038760 0x7f0d8403ac10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-07-14T19:09:52.467 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:48.109+0000 7f0d99d9b700 1 -- 172.21.15.40:0/1364520678 >> [v2:172.21.15.40:6800/3698757264,v1:172.21.15.40:6801/3698757264] conn(0x7f0d84038760 msgr2=0x7f0d8403ac10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.40:6800/3698757264 2023-07-14T19:09:52.467 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:48.109+0000 7f0d99d9b700 1 --2- 172.21.15.40:0/1364520678 >> [v2:172.21.15.40:6800/3698757264,v1:172.21.15.40:6801/3698757264] conn(0x7f0d84038760 0x7f0d8403ac10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2023-07-14T19:09:52.467 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:50.733+0000 7f0d8b7fe700 1 -- 172.21.15.40:0/1364520678 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mgrmap(e 10) v1 ==== 45263+0+0 (secure 0 0 0) 0x7f0d8c02d9b0 con 0x7f0d9c105f00 2023-07-14T19:09:52.467 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:50.733+0000 7f0d8b7fe700 1 -- 172.21.15.40:0/1364520678 >> [v2:172.21.15.40:6800/3698757264,v1:172.21.15.40:6801/3698757264] conn(0x7f0d84038760 msgr2=0x7f0d8403ac10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-07-14T19:09:52.468 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:50.733+0000 7f0d8b7fe700 1 --2- 172.21.15.40:0/1364520678 >> [v2:172.21.15.40:6800/3698757264,v1:172.21.15.40:6801/3698757264] conn(0x7f0d84038760 0x7f0d8403ac10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:09:52.468 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:51.735+0000 7f0d8b7fe700 1 -- 172.21.15.40:0/1364520678 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mgrmap(e 11) v1 ==== 45390+0+0 (secure 0 0 0) 0x7f0d8c019d80 con 0x7f0d9c105f00 2023-07-14T19:09:52.468 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:51.735+0000 7f0d8b7fe700 1 --2- 172.21.15.40:0/1364520678 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f0d8403bff0 0x7f0d8403e3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:09:52.468 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:51.735+0000 7f0d8b7fe700 1 -- 172.21.15.40:0/1364520678 --> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f0d8403b320 con 0x7f0d8403bff0 2023-07-14T19:09:52.468 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:51.735+0000 7f0d99d9b700 1 --2- 172.21.15.40:0/1364520678 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f0d8403bff0 0x7f0d8403e3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:09:52.468 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:51.736+0000 7f0d99d9b700 1 --2- 172.21.15.40:0/1364520678 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f0d8403bff0 0x7f0d8403e3d0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f0d90003a10 tx=0x7f0d90009270 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:09:52.468 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:51.737+0000 7f0d8b7fe700 1 -- 172.21.15.40:0/1364520678 <== mgr.14152 v2:172.21.15.40:6800/1693027118 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+6910 (secure 0 0 0) 0x7f0d8403b320 con 0x7f0d8403bff0 2023-07-14T19:09:52.469 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:51.747+0000 7f0da0f97700 1 -- 172.21.15.40:0/1364520678 --> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7f0d9c19a460 con 0x7f0d8403bff0 2023-07-14T19:09:52.469 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:51.748+0000 7f0d8b7fe700 1 -- 172.21.15.40:0/1364520678 <== mgr.14152 v2:172.21.15.40:6800/1693027118 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+52 (secure 0 0 0) 0x7f0d9c19a460 con 0x7f0d8403bff0 2023-07-14T19:09:52.469 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:51.748+0000 7f0da0f97700 1 -- 172.21.15.40:0/1364520678 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f0d8403bff0 msgr2=0x7f0d8403e3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:09:52.469 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:51.748+0000 7f0da0f97700 1 --2- 172.21.15.40:0/1364520678 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f0d8403bff0 0x7f0d8403e3d0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f0d90003a10 tx=0x7f0d90009270 comp rx=0 tx=0).stop 2023-07-14T19:09:52.469 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:51.748+0000 7f0da0f97700 1 -- 172.21.15.40:0/1364520678 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0d9c105f00 msgr2=0x7f0d9c199120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:09:52.469 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:51.748+0000 7f0da0f97700 1 --2- 172.21.15.40:0/1364520678 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0d9c105f00 0x7f0d9c199120 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f0d8c003e60 tx=0x7f0d8c011aa0 comp rx=0 tx=0).stop 2023-07-14T19:09:52.469 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:51.748+0000 7f0da0f97700 1 -- 172.21.15.40:0/1364520678 shutdown_connections 2023-07-14T19:09:52.470 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:51.748+0000 7f0da0f97700 1 --2- 172.21.15.40:0/1364520678 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f0d8403bff0 0x7f0d8403e3d0 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:09:52.470 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:51.748+0000 7f0da0f97700 1 --2- 172.21.15.40:0/1364520678 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0d9c105f00 0x7f0d9c199120 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:09:52.470 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:51.748+0000 7f0da0f97700 1 --2- 172.21.15.40:0/1364520678 >> [v2:172.21.15.40:6800/3698757264,v1:172.21.15.40:6801/3698757264] conn(0x7f0d84038760 0x7f0d8403ac10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:09:52.470 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:51.748+0000 7f0da0f97700 1 -- 172.21.15.40:0/1364520678 >> 172.21.15.40:0/1364520678 conn(0x7f0d9c0ff9a0 msgr2=0x7f0d9c100780 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:09:52.470 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:51.748+0000 7f0da0f97700 1 -- 172.21.15.40:0/1364520678 shutdown_connections 2023-07-14T19:09:52.470 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:51.748+0000 7f0da0f97700 1 -- 172.21.15.40:0/1364520678 wait complete. 2023-07-14T19:09:52.470 INFO:teuthology.orchestra.run.smithi040.stdout:mgr epoch 9 is available 2023-07-14T19:09:52.470 INFO:teuthology.orchestra.run.smithi040.stdout:Generating a dashboard self-signed certificate... 2023-07-14T19:09:53.722 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:09:53 smithi040 ceph-mon[96518]: from='client.14156 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2023-07-14T19:09:53.722 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:09:53 smithi040 ceph-mon[96518]: [14/Jul/2023:19:09:51] ENGINE Serving on https://172.21.15.40:7150 2023-07-14T19:09:53.722 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:09:53 smithi040 ceph-mon[96518]: [14/Jul/2023:19:09:51] ENGINE Serving on http://172.21.15.40:8765 2023-07-14T19:09:53.722 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:09:53 smithi040 ceph-mon[96518]: [14/Jul/2023:19:09:51] ENGINE Bus STARTED 2023-07-14T19:09:53.722 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:09:53 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:09:53.723 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:09:53 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:09:53.723 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:09:53 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:09:53.723 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:09:53 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:09:53.807 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout Self-signed certificate created 2023-07-14T19:09:53.807 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:52.947+0000 7fd55d6e1700 1 Processor -- start 2023-07-14T19:09:53.807 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:52.947+0000 7fd55d6e1700 1 -- start start 2023-07-14T19:09:53.808 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:52.947+0000 7fd55d6e1700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd5581042f0 0x7fd558104700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:09:53.808 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:52.947+0000 7fd55d6e1700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd558104c40 con 0x7fd5581042f0 2023-07-14T19:09:53.808 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:52.947+0000 7fd556ffd700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd5581042f0 0x7fd558104700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:09:53.808 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:52.947+0000 7fd556ffd700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd5581042f0 0x7fd558104700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:60058/0 (socket says 172.21.15.40:60058) 2023-07-14T19:09:53.808 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:52.947+0000 7fd556ffd700 1 -- 172.21.15.40:0/1363217161 learned_addr learned my addr 172.21.15.40:0/1363217161 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-07-14T19:09:53.809 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:52.947+0000 7fd556ffd700 1 -- 172.21.15.40:0/1363217161 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd558104d80 con 0x7fd5581042f0 2023-07-14T19:09:53.809 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:52.948+0000 7fd556ffd700 1 --2- 172.21.15.40:0/1363217161 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd5581042f0 0x7fd558104700 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fd5480098a0 tx=0x7fd548009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a7dbd52e03a4e2ab server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:09:53.809 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:52.948+0000 7fd555ffb700 1 -- 172.21.15.40:0/1363217161 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd548004160 con 0x7fd5581042f0 2023-07-14T19:09:53.809 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:52.948+0000 7fd555ffb700 1 -- 172.21.15.40:0/1363217161 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd5480042c0 con 0x7fd5581042f0 2023-07-14T19:09:53.810 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:52.948+0000 7fd555ffb700 1 -- 172.21.15.40:0/1363217161 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd5480119b0 con 0x7fd5581042f0 2023-07-14T19:09:53.810 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:52.948+0000 7fd55d6e1700 1 -- 172.21.15.40:0/1363217161 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd5581042f0 msgr2=0x7fd558104700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:09:53.810 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:52.948+0000 7fd55d6e1700 1 --2- 172.21.15.40:0/1363217161 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd5581042f0 0x7fd558104700 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fd5480098a0 tx=0x7fd548009bb0 comp rx=0 tx=0).stop 2023-07-14T19:09:53.810 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:52.948+0000 7fd55d6e1700 1 -- 172.21.15.40:0/1363217161 shutdown_connections 2023-07-14T19:09:53.811 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:52.948+0000 7fd55d6e1700 1 --2- 172.21.15.40:0/1363217161 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd5581042f0 0x7fd558104700 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:09:53.811 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:52.948+0000 7fd55d6e1700 1 -- 172.21.15.40:0/1363217161 >> 172.21.15.40:0/1363217161 conn(0x7fd5580ff9a0 msgr2=0x7fd558101db0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:09:53.811 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:52.948+0000 7fd55d6e1700 1 -- 172.21.15.40:0/1363217161 shutdown_connections 2023-07-14T19:09:53.811 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:52.948+0000 7fd55d6e1700 1 -- 172.21.15.40:0/1363217161 wait complete. 2023-07-14T19:09:53.811 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:52.949+0000 7fd55d6e1700 1 Processor -- start 2023-07-14T19:09:53.812 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:52.949+0000 7fd55d6e1700 1 -- start start 2023-07-14T19:09:53.812 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:52.949+0000 7fd55d6e1700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd5581042f0 0x7fd55819b2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:09:53.812 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:52.949+0000 7fd55d6e1700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd55819b800 con 0x7fd5581042f0 2023-07-14T19:09:53.812 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:52.950+0000 7fd556ffd700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd5581042f0 0x7fd55819b2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:09:53.813 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:52.950+0000 7fd556ffd700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd5581042f0 0x7fd55819b2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:60060/0 (socket says 172.21.15.40:60060) 2023-07-14T19:09:53.813 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:52.950+0000 7fd556ffd700 1 -- 172.21.15.40:0/1343404681 learned_addr learned my addr 172.21.15.40:0/1343404681 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-07-14T19:09:53.813 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:52.950+0000 7fd556ffd700 1 -- 172.21.15.40:0/1343404681 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd548009550 con 0x7fd5581042f0 2023-07-14T19:09:53.814 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:52.950+0000 7fd556ffd700 1 --2- 172.21.15.40:0/1343404681 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd5581042f0 0x7fd55819b2c0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fd548011980 tx=0x7fd548011a60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:09:53.814 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:52.950+0000 7fd53ffff700 1 -- 172.21.15.40:0/1343404681 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd548003a60 con 0x7fd5581042f0 2023-07-14T19:09:53.814 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:52.950+0000 7fd55d6e1700 1 -- 172.21.15.40:0/1343404681 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd55819ba00 con 0x7fd5581042f0 2023-07-14T19:09:53.814 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:52.950+0000 7fd53ffff700 1 -- 172.21.15.40:0/1343404681 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd548003bc0 con 0x7fd5581042f0 2023-07-14T19:09:53.815 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:52.950+0000 7fd53ffff700 1 -- 172.21.15.40:0/1343404681 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd54801a670 con 0x7fd5581042f0 2023-07-14T19:09:53.815 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:52.950+0000 7fd55d6e1700 1 -- 172.21.15.40:0/1343404681 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd55819be20 con 0x7fd5581042f0 2023-07-14T19:09:53.815 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:52.951+0000 7fd53ffff700 1 -- 172.21.15.40:0/1343404681 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 11) v1 ==== 45390+0+0 (secure 0 0 0) 0x7fd54801a930 con 0x7fd5581042f0 2023-07-14T19:09:53.815 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:52.951+0000 7fd53ffff700 1 --2- 172.21.15.40:0/1343404681 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7fd540038630 0x7fd54003aae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:09:53.816 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:52.951+0000 7fd53ffff700 1 -- 172.21.15.40:0/1343404681 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fd54804c930 con 0x7fd5581042f0 2023-07-14T19:09:53.816 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:52.952+0000 7fd5567fc700 1 --2- 172.21.15.40:0/1343404681 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7fd540038630 0x7fd54003aae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:09:53.816 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:52.952+0000 7fd55d6e1700 1 -- 172.21.15.40:0/1343404681 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd558190c20 con 0x7fd5581042f0 2023-07-14T19:09:53.817 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:52.952+0000 7fd5567fc700 1 --2- 172.21.15.40:0/1343404681 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7fd540038630 0x7fd54003aae0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fd54c006fd0 tx=0x7fd54c006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:09:53.817 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:52.956+0000 7fd53ffff700 1 -- 172.21.15.40:0/1343404681 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fd54801ac30 con 0x7fd5581042f0 2023-07-14T19:09:53.817 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:53.327+0000 7fd55d6e1700 1 -- 172.21.15.40:0/1343404681 --> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] -- mgr_command(tid 0: {"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}) v1 -- 0x7fd558106330 con 0x7fd540038630 2023-07-14T19:09:53.817 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:53.381+0000 7fd53ffff700 1 -- 172.21.15.40:0/1343404681 <== mgr.14152 v2:172.21.15.40:6800/1693027118 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+32 (secure 0 0 0) 0x7fd558106330 con 0x7fd540038630 2023-07-14T19:09:53.818 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:53.383+0000 7fd55d6e1700 1 -- 172.21.15.40:0/1343404681 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7fd540038630 msgr2=0x7fd54003aae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:09:53.818 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:53.383+0000 7fd55d6e1700 1 --2- 172.21.15.40:0/1343404681 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7fd540038630 0x7fd54003aae0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fd54c006fd0 tx=0x7fd54c006e40 comp rx=0 tx=0).stop 2023-07-14T19:09:53.818 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:53.383+0000 7fd55d6e1700 1 -- 172.21.15.40:0/1343404681 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd5581042f0 msgr2=0x7fd55819b2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:09:53.818 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:53.383+0000 7fd55d6e1700 1 --2- 172.21.15.40:0/1343404681 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd5581042f0 0x7fd55819b2c0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fd548011980 tx=0x7fd548011a60 comp rx=0 tx=0).stop 2023-07-14T19:09:53.819 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:53.383+0000 7fd55d6e1700 1 -- 172.21.15.40:0/1343404681 shutdown_connections 2023-07-14T19:09:53.819 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:53.383+0000 7fd55d6e1700 1 --2- 172.21.15.40:0/1343404681 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7fd540038630 0x7fd54003aae0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:09:53.819 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:53.383+0000 7fd55d6e1700 1 --2- 172.21.15.40:0/1343404681 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd5581042f0 0x7fd55819b2c0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:09:53.819 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:53.383+0000 7fd55d6e1700 1 -- 172.21.15.40:0/1343404681 >> 172.21.15.40:0/1343404681 conn(0x7fd5580ff9a0 msgr2=0x7fd55818dd60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:09:53.820 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:53.384+0000 7fd55d6e1700 1 -- 172.21.15.40:0/1343404681 shutdown_connections 2023-07-14T19:09:53.820 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:53.384+0000 7fd55d6e1700 1 -- 172.21.15.40:0/1343404681 wait complete. 2023-07-14T19:09:53.820 INFO:teuthology.orchestra.run.smithi040.stdout:Creating initial admin user... 2023-07-14T19:09:54.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:09:54 smithi040 ceph-mon[96518]: from='client.14164 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T19:09:54.873 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:09:54 smithi040 ceph-mon[96518]: mgrmap e12: x(active, since 2s) 2023-07-14T19:09:55.555 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout {"username": "admin", "password": "$2b$12$k8MddWgwpgXzb8rPeWU90OnFg0axXygUDxTJJ4A.HdeY8oOG1dMg.", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1689361794, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2023-07-14T19:09:55.555 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:54.338+0000 7f9e13e12700 1 Processor -- start 2023-07-14T19:09:55.556 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:54.339+0000 7f9e13e12700 1 -- start start 2023-07-14T19:09:55.556 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:54.339+0000 7f9e13e12700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9e0c1042f0 0x7f9e0c104700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:09:55.556 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:54.339+0000 7f9e13e12700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e0c104c40 con 0x7f9e0c1042f0 2023-07-14T19:09:55.556 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:54.339+0000 7f9e11bae700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9e0c1042f0 0x7f9e0c104700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:09:55.556 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:54.339+0000 7f9e11bae700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9e0c1042f0 0x7f9e0c104700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:60064/0 (socket says 172.21.15.40:60064) 2023-07-14T19:09:55.556 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:54.339+0000 7f9e11bae700 1 -- 172.21.15.40:0/1798767276 learned_addr learned my addr 172.21.15.40:0/1798767276 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-07-14T19:09:55.556 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:54.339+0000 7f9e11bae700 1 -- 172.21.15.40:0/1798767276 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e0c104d80 con 0x7f9e0c1042f0 2023-07-14T19:09:55.557 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:54.340+0000 7f9e11bae700 1 --2- 172.21.15.40:0/1798767276 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9e0c1042f0 0x7f9e0c104700 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f9dfc0098a0 tx=0x7f9dfc009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=faebf93ec8c51cbb server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:09:55.557 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:54.340+0000 7f9e10bac700 1 -- 172.21.15.40:0/1798767276 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f9dfc004160 con 0x7f9e0c1042f0 2023-07-14T19:09:55.557 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:54.340+0000 7f9e10bac700 1 -- 172.21.15.40:0/1798767276 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f9dfc0042c0 con 0x7f9e0c1042f0 2023-07-14T19:09:55.557 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:54.340+0000 7f9e10bac700 1 -- 172.21.15.40:0/1798767276 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f9dfc0119b0 con 0x7f9e0c1042f0 2023-07-14T19:09:55.557 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:54.340+0000 7f9e13e12700 1 -- 172.21.15.40:0/1798767276 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9e0c1042f0 msgr2=0x7f9e0c104700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:09:55.557 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:54.340+0000 7f9e13e12700 1 --2- 172.21.15.40:0/1798767276 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9e0c1042f0 0x7f9e0c104700 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f9dfc0098a0 tx=0x7f9dfc009bb0 comp rx=0 tx=0).stop 2023-07-14T19:09:55.557 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:54.340+0000 7f9e13e12700 1 -- 172.21.15.40:0/1798767276 shutdown_connections 2023-07-14T19:09:55.557 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:54.340+0000 7f9e13e12700 1 --2- 172.21.15.40:0/1798767276 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9e0c1042f0 0x7f9e0c104700 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:09:55.558 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:54.340+0000 7f9e13e12700 1 -- 172.21.15.40:0/1798767276 >> 172.21.15.40:0/1798767276 conn(0x7f9e0c0ff9a0 msgr2=0x7f9e0c101db0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:09:55.558 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:54.340+0000 7f9e13e12700 1 -- 172.21.15.40:0/1798767276 shutdown_connections 2023-07-14T19:09:55.558 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:54.341+0000 7f9e13e12700 1 -- 172.21.15.40:0/1798767276 wait complete. 2023-07-14T19:09:55.558 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:54.341+0000 7f9e13e12700 1 Processor -- start 2023-07-14T19:09:55.558 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:54.341+0000 7f9e13e12700 1 -- start start 2023-07-14T19:09:55.558 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:54.341+0000 7f9e13e12700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9e0c1042f0 0x7f9e0c197fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:09:55.558 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:54.341+0000 7f9e13e12700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e0c198520 con 0x7f9e0c1042f0 2023-07-14T19:09:55.559 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:54.341+0000 7f9e11bae700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9e0c1042f0 0x7f9e0c197fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:09:55.559 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:54.341+0000 7f9e11bae700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9e0c1042f0 0x7f9e0c197fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:60066/0 (socket says 172.21.15.40:60066) 2023-07-14T19:09:55.559 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:54.341+0000 7f9e11bae700 1 -- 172.21.15.40:0/3917447650 learned_addr learned my addr 172.21.15.40:0/3917447650 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-07-14T19:09:55.559 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:54.342+0000 7f9e11bae700 1 -- 172.21.15.40:0/3917447650 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9dfc009550 con 0x7f9e0c1042f0 2023-07-14T19:09:55.559 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:54.342+0000 7f9e11bae700 1 --2- 172.21.15.40:0/3917447650 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9e0c1042f0 0x7f9e0c197fe0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f9dfc0038d0 tx=0x7f9dfc0039b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:09:55.559 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:54.342+0000 7f9e02ffd700 1 -- 172.21.15.40:0/3917447650 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f9dfc003c00 con 0x7f9e0c1042f0 2023-07-14T19:09:55.559 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:54.342+0000 7f9e02ffd700 1 -- 172.21.15.40:0/3917447650 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f9dfc020070 con 0x7f9e0c1042f0 2023-07-14T19:09:55.560 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:54.342+0000 7f9e13e12700 1 -- 172.21.15.40:0/3917447650 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9e0c198720 con 0x7f9e0c1042f0 2023-07-14T19:09:55.560 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:54.342+0000 7f9e02ffd700 1 -- 172.21.15.40:0/3917447650 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f9dfc01c650 con 0x7f9e0c1042f0 2023-07-14T19:09:55.560 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:54.342+0000 7f9e13e12700 1 -- 172.21.15.40:0/3917447650 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e0c198b40 con 0x7f9e0c1042f0 2023-07-14T19:09:55.560 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:54.343+0000 7f9e02ffd700 1 -- 172.21.15.40:0/3917447650 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 12) v1 ==== 45496+0+0 (secure 0 0 0) 0x7f9dfc022030 con 0x7f9e0c1042f0 2023-07-14T19:09:55.560 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:54.343+0000 7f9e02ffd700 1 --2- 172.21.15.40:0/3917447650 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f9df8038790 0x7f9df803ac40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:09:55.560 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:54.343+0000 7f9e02ffd700 1 -- 172.21.15.40:0/3917447650 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f9dfc04f160 con 0x7f9e0c1042f0 2023-07-14T19:09:55.561 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:54.343+0000 7f9e13e12700 1 -- 172.21.15.40:0/3917447650 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9e0c061c70 con 0x7f9e0c1042f0 2023-07-14T19:09:55.562 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:54.343+0000 7f9e113ad700 1 --2- 172.21.15.40:0/3917447650 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f9df8038790 0x7f9df803ac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:09:55.562 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:54.344+0000 7f9e113ad700 1 --2- 172.21.15.40:0/3917447650 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f9df8038790 0x7f9df803ac40 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f9e08006fd0 tx=0x7f9e08006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:09:55.562 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:54.348+0000 7f9e02ffd700 1 -- 172.21.15.40:0/3917447650 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f9dfc024bc0 con 0x7f9e0c1042f0 2023-07-14T19:09:55.562 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:54.744+0000 7f9e13e12700 1 -- 172.21.15.40:0/3917447650 --> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] -- mgr_command(tid 0: {"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}) v1 -- 0x7f9e0c1087c0 con 0x7f9df8038790 2023-07-14T19:09:55.563 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:54.994+0000 7f9e02ffd700 1 -- 172.21.15.40:0/3917447650 <== mgr.14152 v2:172.21.15.40:6800/1693027118 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+252 (secure 0 0 0) 0x7f9e08009e20 con 0x7f9df8038790 2023-07-14T19:09:55.563 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:54.997+0000 7f9e13e12700 1 -- 172.21.15.40:0/3917447650 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f9df8038790 msgr2=0x7f9df803ac40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:09:55.563 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:54.997+0000 7f9e13e12700 1 --2- 172.21.15.40:0/3917447650 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f9df8038790 0x7f9df803ac40 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f9e08006fd0 tx=0x7f9e08006e40 comp rx=0 tx=0).stop 2023-07-14T19:09:55.563 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:54.997+0000 7f9e13e12700 1 -- 172.21.15.40:0/3917447650 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9e0c1042f0 msgr2=0x7f9e0c197fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:09:55.563 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:54.997+0000 7f9e13e12700 1 --2- 172.21.15.40:0/3917447650 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9e0c1042f0 0x7f9e0c197fe0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f9dfc0038d0 tx=0x7f9dfc0039b0 comp rx=0 tx=0).stop 2023-07-14T19:09:55.564 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:54.997+0000 7f9e13e12700 1 -- 172.21.15.40:0/3917447650 shutdown_connections 2023-07-14T19:09:55.564 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:54.997+0000 7f9e13e12700 1 --2- 172.21.15.40:0/3917447650 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f9df8038790 0x7f9df803ac40 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:09:55.564 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:54.997+0000 7f9e13e12700 1 --2- 172.21.15.40:0/3917447650 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9e0c1042f0 0x7f9e0c197fe0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:09:55.564 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:54.997+0000 7f9e13e12700 1 -- 172.21.15.40:0/3917447650 >> 172.21.15.40:0/3917447650 conn(0x7f9e0c0ff9a0 msgr2=0x7f9e0c100780 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:09:55.564 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:54.997+0000 7f9e13e12700 1 -- 172.21.15.40:0/3917447650 shutdown_connections 2023-07-14T19:09:55.565 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:54.997+0000 7f9e13e12700 1 -- 172.21.15.40:0/3917447650 wait complete. 2023-07-14T19:09:55.565 INFO:teuthology.orchestra.run.smithi040.stdout:Fetching dashboard port number... 2023-07-14T19:09:56.221 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:09:55 smithi040 ceph-mon[96518]: from='client.14166 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T19:09:56.222 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:09:55 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:09:56.951 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout 8443 2023-07-14T19:09:56.951 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:56.128+0000 7fb90c5ee700 1 Processor -- start 2023-07-14T19:09:56.951 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:56.128+0000 7fb90c5ee700 1 -- start start 2023-07-14T19:09:56.951 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:56.129+0000 7fb90c5ee700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb904079ce0 0x7fb90410a390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:09:56.952 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:56.129+0000 7fb90c5ee700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb904105d80 con 0x7fb904079ce0 2023-07-14T19:09:56.952 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:56.129+0000 7fb90a38a700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb904079ce0 0x7fb90410a390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:09:56.952 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:56.129+0000 7fb90a38a700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb904079ce0 0x7fb90410a390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:60072/0 (socket says 172.21.15.40:60072) 2023-07-14T19:09:56.952 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:56.129+0000 7fb90a38a700 1 -- 172.21.15.40:0/433604307 learned_addr learned my addr 172.21.15.40:0/433604307 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-07-14T19:09:56.952 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:56.129+0000 7fb90a38a700 1 -- 172.21.15.40:0/433604307 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb90410a8d0 con 0x7fb904079ce0 2023-07-14T19:09:56.952 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:56.129+0000 7fb90a38a700 1 --2- 172.21.15.40:0/433604307 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb904079ce0 0x7fb90410a390 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fb8f4009e80 tx=0x7fb8f400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d55a7f77fb66ac06 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:09:56.952 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:56.129+0000 7fb909388700 1 -- 172.21.15.40:0/433604307 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb8f400e070 con 0x7fb904079ce0 2023-07-14T19:09:56.953 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:56.129+0000 7fb909388700 1 -- 172.21.15.40:0/433604307 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb8f400b8c0 con 0x7fb904079ce0 2023-07-14T19:09:56.953 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:56.129+0000 7fb909388700 1 -- 172.21.15.40:0/433604307 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb8f40043e0 con 0x7fb904079ce0 2023-07-14T19:09:56.953 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:56.130+0000 7fb90c5ee700 1 -- 172.21.15.40:0/433604307 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb904079ce0 msgr2=0x7fb90410a390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:09:56.953 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:56.130+0000 7fb90c5ee700 1 --2- 172.21.15.40:0/433604307 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb904079ce0 0x7fb90410a390 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fb8f4009e80 tx=0x7fb8f400b330 comp rx=0 tx=0).stop 2023-07-14T19:09:56.953 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:56.130+0000 7fb90c5ee700 1 -- 172.21.15.40:0/433604307 shutdown_connections 2023-07-14T19:09:56.953 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:56.130+0000 7fb90c5ee700 1 --2- 172.21.15.40:0/433604307 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb904079ce0 0x7fb90410a390 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:09:56.953 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:56.130+0000 7fb90c5ee700 1 -- 172.21.15.40:0/433604307 >> 172.21.15.40:0/433604307 conn(0x7fb9040ff9a0 msgr2=0x7fb904101db0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:09:56.953 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:56.130+0000 7fb90c5ee700 1 -- 172.21.15.40:0/433604307 shutdown_connections 2023-07-14T19:09:56.954 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:56.130+0000 7fb90c5ee700 1 -- 172.21.15.40:0/433604307 wait complete. 2023-07-14T19:09:56.954 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:56.131+0000 7fb90c5ee700 1 Processor -- start 2023-07-14T19:09:56.954 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:56.131+0000 7fb90c5ee700 1 -- start start 2023-07-14T19:09:56.954 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:56.131+0000 7fb90c5ee700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb90419b290 0x7fb90419b6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:09:56.954 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:56.131+0000 7fb90c5ee700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb90419bbe0 con 0x7fb90419b290 2023-07-14T19:09:56.954 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:56.131+0000 7fb90a38a700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb90419b290 0x7fb90419b6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:09:56.954 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:56.131+0000 7fb90a38a700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb90419b290 0x7fb90419b6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:60074/0 (socket says 172.21.15.40:60074) 2023-07-14T19:09:56.955 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:56.131+0000 7fb90a38a700 1 -- 172.21.15.40:0/730753861 learned_addr learned my addr 172.21.15.40:0/730753861 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-07-14T19:09:56.955 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:56.131+0000 7fb90a38a700 1 -- 172.21.15.40:0/730753861 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8f4009b30 con 0x7fb90419b290 2023-07-14T19:09:56.955 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:56.132+0000 7fb90a38a700 1 --2- 172.21.15.40:0/730753861 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb90419b290 0x7fb90419b6a0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fb8f4009530 tx=0x7fb8f4003d00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:09:56.955 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:56.132+0000 7fb9037fe700 1 -- 172.21.15.40:0/730753861 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb8f400e040 con 0x7fb90419b290 2023-07-14T19:09:56.955 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:56.132+0000 7fb9037fe700 1 -- 172.21.15.40:0/730753861 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb8f400bd70 con 0x7fb90419b290 2023-07-14T19:09:56.955 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:56.132+0000 7fb90c5ee700 1 -- 172.21.15.40:0/730753861 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb90419bde0 con 0x7fb90419b290 2023-07-14T19:09:56.955 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:56.132+0000 7fb9037fe700 1 -- 172.21.15.40:0/730753861 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb8f401c700 con 0x7fb90419b290 2023-07-14T19:09:56.955 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:56.132+0000 7fb90c5ee700 1 -- 172.21.15.40:0/730753861 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb90419ea40 con 0x7fb90419b290 2023-07-14T19:09:56.956 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:56.133+0000 7fb9037fe700 1 -- 172.21.15.40:0/730753861 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 12) v1 ==== 45496+0+0 (secure 0 0 0) 0x7fb8f401c8f0 con 0x7fb90419b290 2023-07-14T19:09:56.956 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:56.133+0000 7fb9037fe700 1 --2- 172.21.15.40:0/730753861 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7fb8ec0387b0 0x7fb8ec03ac60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:09:56.956 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:56.133+0000 7fb9037fe700 1 -- 172.21.15.40:0/730753861 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fb8f404c6c0 con 0x7fb90419b290 2023-07-14T19:09:56.956 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:56.133+0000 7fb90c5ee700 1 -- 172.21.15.40:0/730753861 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb904061c70 con 0x7fb90419b290 2023-07-14T19:09:56.956 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:56.133+0000 7fb909b89700 1 --2- 172.21.15.40:0/730753861 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7fb8ec0387b0 0x7fb8ec03ac60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:09:56.957 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:56.133+0000 7fb909b89700 1 --2- 172.21.15.40:0/730753861 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7fb8ec0387b0 0x7fb8ec03ac60 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fb8f8006fd0 tx=0x7fb8f8006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:09:56.957 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:56.138+0000 7fb9037fe700 1 -- 172.21.15.40:0/730753861 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fb8f4017070 con 0x7fb90419b290 2023-07-14T19:09:56.957 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:56.495+0000 7fb90c5ee700 1 -- 172.21.15.40:0/730753861 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"} v 0) v1 -- 0x7fb904079ce0 con 0x7fb90419b290 2023-07-14T19:09:56.957 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:56.495+0000 7fb9037fe700 1 -- 172.21.15.40:0/730753861 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]=0 v8) v1 ==== 112+0+5 (secure 0 0 0) 0x7fb8f4020070 con 0x7fb90419b290 2023-07-14T19:09:56.957 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:56.498+0000 7fb90c5ee700 1 -- 172.21.15.40:0/730753861 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7fb8ec0387b0 msgr2=0x7fb8ec03ac60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:09:56.957 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:56.498+0000 7fb90c5ee700 1 --2- 172.21.15.40:0/730753861 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7fb8ec0387b0 0x7fb8ec03ac60 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fb8f8006fd0 tx=0x7fb8f8006e40 comp rx=0 tx=0).stop 2023-07-14T19:09:56.957 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:56.498+0000 7fb90c5ee700 1 -- 172.21.15.40:0/730753861 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb90419b290 msgr2=0x7fb90419b6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:09:56.957 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:56.498+0000 7fb90c5ee700 1 --2- 172.21.15.40:0/730753861 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb90419b290 0x7fb90419b6a0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fb8f4009530 tx=0x7fb8f4003d00 comp rx=0 tx=0).stop 2023-07-14T19:09:56.958 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:56.498+0000 7fb90c5ee700 1 -- 172.21.15.40:0/730753861 shutdown_connections 2023-07-14T19:09:56.958 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:56.498+0000 7fb90c5ee700 1 --2- 172.21.15.40:0/730753861 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7fb8ec0387b0 0x7fb8ec03ac60 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:09:56.958 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:56.498+0000 7fb90c5ee700 1 --2- 172.21.15.40:0/730753861 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb90419b290 0x7fb90419b6a0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:09:56.958 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:56.498+0000 7fb90c5ee700 1 -- 172.21.15.40:0/730753861 >> 172.21.15.40:0/730753861 conn(0x7fb9040ff9a0 msgr2=0x7fb904100760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:09:56.958 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:56.498+0000 7fb90c5ee700 1 -- 172.21.15.40:0/730753861 shutdown_connections 2023-07-14T19:09:56.958 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:56.498+0000 7fb90c5ee700 1 -- 172.21.15.40:0/730753861 wait complete. 2023-07-14T19:09:56.961 INFO:teuthology.orchestra.run.smithi040.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2023-07-14T19:09:56.961 INFO:teuthology.orchestra.run.smithi040.stdout:systemctl: stdout disabled 2023-07-14T19:09:56.969 INFO:teuthology.orchestra.run.smithi040.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2023-07-14T19:09:56.969 INFO:teuthology.orchestra.run.smithi040.stdout:systemctl: stdout inactive 2023-07-14T19:09:56.969 INFO:teuthology.orchestra.run.smithi040.stdout:firewalld.service is not enabled 2023-07-14T19:09:56.969 INFO:teuthology.orchestra.run.smithi040.stdout:Not possible to open ports <[8443]>. firewalld.service is not available 2023-07-14T19:09:56.971 INFO:teuthology.orchestra.run.smithi040.stdout:Ceph Dashboard is now available at: 2023-07-14T19:09:56.971 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-07-14T19:09:56.971 INFO:teuthology.orchestra.run.smithi040.stdout: URL: https://smithi040:8443/ 2023-07-14T19:09:56.971 INFO:teuthology.orchestra.run.smithi040.stdout: User: admin 2023-07-14T19:09:56.971 INFO:teuthology.orchestra.run.smithi040.stdout: Password: ogiuvx132g 2023-07-14T19:09:56.972 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-07-14T19:09:56.972 INFO:teuthology.orchestra.run.smithi040.stdout:Saving cluster configuration to /var/lib/ceph/adc52f04-2279-11ee-9b33-001a4aab830c/config directory 2023-07-14T19:09:56.972 INFO:teuthology.orchestra.run.smithi040.stdout:Enabling autotune for osd_memory_target 2023-07-14T19:09:57.876 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:09:57 smithi040 ceph-mon[96518]: from='client.? 172.21.15.40:0/730753861' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2023-07-14T19:09:58.320 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:57.489+0000 7f4079298700 1 Processor -- start 2023-07-14T19:09:58.320 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:57.489+0000 7f4079298700 1 -- start start 2023-07-14T19:09:58.321 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:57.489+0000 7f4079298700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f40741042f0 0x7f4074104700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:09:58.321 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:57.489+0000 7f4079298700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4074104c40 con 0x7f40741042f0 2023-07-14T19:09:58.321 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:57.489+0000 7f4072ffd700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f40741042f0 0x7f4074104700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:09:58.321 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:57.490+0000 7f4072ffd700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f40741042f0 0x7f4074104700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:60080/0 (socket says 172.21.15.40:60080) 2023-07-14T19:09:58.321 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:57.490+0000 7f4072ffd700 1 -- 172.21.15.40:0/3932630475 learned_addr learned my addr 172.21.15.40:0/3932630475 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-07-14T19:09:58.321 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:57.490+0000 7f4072ffd700 1 -- 172.21.15.40:0/3932630475 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4074104d80 con 0x7f40741042f0 2023-07-14T19:09:58.322 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:57.490+0000 7f4072ffd700 1 --2- 172.21.15.40:0/3932630475 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f40741042f0 0x7f4074104700 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f40640098a0 tx=0x7f4064009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=658c6d14366a7430 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:09:58.322 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:57.490+0000 7f40727fc700 1 -- 172.21.15.40:0/3932630475 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4064004160 con 0x7f40741042f0 2023-07-14T19:09:58.322 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:57.490+0000 7f40727fc700 1 -- 172.21.15.40:0/3932630475 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f40640042c0 con 0x7f40741042f0 2023-07-14T19:09:58.322 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:57.490+0000 7f40727fc700 1 -- 172.21.15.40:0/3932630475 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f40640119b0 con 0x7f40741042f0 2023-07-14T19:09:58.322 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:57.491+0000 7f4079298700 1 -- 172.21.15.40:0/3932630475 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f40741042f0 msgr2=0x7f4074104700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:09:58.322 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:57.491+0000 7f4079298700 1 --2- 172.21.15.40:0/3932630475 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f40741042f0 0x7f4074104700 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f40640098a0 tx=0x7f4064009bb0 comp rx=0 tx=0).stop 2023-07-14T19:09:58.322 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:57.491+0000 7f4079298700 1 -- 172.21.15.40:0/3932630475 shutdown_connections 2023-07-14T19:09:58.322 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:57.491+0000 7f4079298700 1 --2- 172.21.15.40:0/3932630475 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f40741042f0 0x7f4074104700 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:09:58.323 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:57.491+0000 7f4079298700 1 -- 172.21.15.40:0/3932630475 >> 172.21.15.40:0/3932630475 conn(0x7f40740ff9a0 msgr2=0x7f4074101db0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:09:58.323 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:57.491+0000 7f4079298700 1 -- 172.21.15.40:0/3932630475 shutdown_connections 2023-07-14T19:09:58.323 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:57.491+0000 7f4079298700 1 -- 172.21.15.40:0/3932630475 wait complete. 2023-07-14T19:09:58.323 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:57.492+0000 7f4079298700 1 Processor -- start 2023-07-14T19:09:58.323 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:57.492+0000 7f4079298700 1 -- start start 2023-07-14T19:09:58.323 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:57.492+0000 7f4079298700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f40741042f0 0x7f4074196ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:09:58.324 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:57.492+0000 7f4079298700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4074197420 con 0x7f40741042f0 2023-07-14T19:09:58.324 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:57.492+0000 7f4072ffd700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f40741042f0 0x7f4074196ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:09:58.324 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:57.492+0000 7f4072ffd700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f40741042f0 0x7f4074196ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:60082/0 (socket says 172.21.15.40:60082) 2023-07-14T19:09:58.324 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:57.492+0000 7f4072ffd700 1 -- 172.21.15.40:0/196578480 learned_addr learned my addr 172.21.15.40:0/196578480 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-07-14T19:09:58.324 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:57.492+0000 7f4072ffd700 1 -- 172.21.15.40:0/196578480 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4064009550 con 0x7f40741042f0 2023-07-14T19:09:58.324 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:57.492+0000 7f4072ffd700 1 --2- 172.21.15.40:0/196578480 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f40741042f0 0x7f4074196ee0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f4064011980 tx=0x7f4064011a60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:09:58.324 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:57.493+0000 7f4070ff9700 1 -- 172.21.15.40:0/196578480 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4064003a60 con 0x7f40741042f0 2023-07-14T19:09:58.325 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:57.493+0000 7f4070ff9700 1 -- 172.21.15.40:0/196578480 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f4064003bc0 con 0x7f40741042f0 2023-07-14T19:09:58.325 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:57.493+0000 7f4079298700 1 -- 172.21.15.40:0/196578480 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4074197620 con 0x7f40741042f0 2023-07-14T19:09:58.325 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:57.493+0000 7f4070ff9700 1 -- 172.21.15.40:0/196578480 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f406401a670 con 0x7f40741042f0 2023-07-14T19:09:58.325 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:57.493+0000 7f4079298700 1 -- 172.21.15.40:0/196578480 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4074197a40 con 0x7f40741042f0 2023-07-14T19:09:58.325 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:57.494+0000 7f4070ff9700 1 -- 172.21.15.40:0/196578480 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 13) v1 ==== 45539+0+0 (secure 0 0 0) 0x7f406401a8f0 con 0x7f40741042f0 2023-07-14T19:09:58.325 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:57.494+0000 7f4070ff9700 1 --2- 172.21.15.40:0/196578480 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f406003aa40 0x7f406003cef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:09:58.325 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:57.494+0000 7f4070ff9700 1 -- 172.21.15.40:0/196578480 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f406404c9d0 con 0x7f40741042f0 2023-07-14T19:09:58.325 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:57.494+0000 7f406bfff700 1 --2- 172.21.15.40:0/196578480 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f406003aa40 0x7f406003cef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:09:58.326 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:57.494+0000 7f4079298700 1 -- 172.21.15.40:0/196578480 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4074190c20 con 0x7f40741042f0 2023-07-14T19:09:58.326 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:57.494+0000 7f406bfff700 1 --2- 172.21.15.40:0/196578480 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f406003aa40 0x7f406003cef0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f405c006fd0 tx=0x7f405c006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:09:58.326 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:57.499+0000 7f4070ff9700 1 -- 172.21.15.40:0/196578480 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f4064018d90 con 0x7f40741042f0 2023-07-14T19:09:58.326 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:57.851+0000 7f4079298700 1 -- 172.21.15.40:0/196578480 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command([{prefix=config set, name=osd_memory_target_autotune}] v 0) v1 -- 0x7f40740008d0 con 0x7f40741042f0 2023-07-14T19:09:58.326 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:57.852+0000 7f4070ff9700 1 -- 172.21.15.40:0/196578480 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{prefix=config set, name=osd_memory_target_autotune}]=0 v8)=0 v8) v1 ==== 127+0+0 (secure 0 0 0) 0x7f406404a030 con 0x7f40741042f0 2023-07-14T19:09:58.326 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:57.854+0000 7f4079298700 1 -- 172.21.15.40:0/196578480 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f406003aa40 msgr2=0x7f406003cef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:09:58.326 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:57.854+0000 7f4079298700 1 --2- 172.21.15.40:0/196578480 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f406003aa40 0x7f406003cef0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f405c006fd0 tx=0x7f405c006e40 comp rx=0 tx=0).stop 2023-07-14T19:09:58.327 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:57.854+0000 7f4079298700 1 -- 172.21.15.40:0/196578480 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f40741042f0 msgr2=0x7f4074196ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:09:58.327 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:57.854+0000 7f4079298700 1 --2- 172.21.15.40:0/196578480 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f40741042f0 0x7f4074196ee0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f4064011980 tx=0x7f4064011a60 comp rx=0 tx=0).stop 2023-07-14T19:09:58.327 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:57.854+0000 7f4079298700 1 -- 172.21.15.40:0/196578480 shutdown_connections 2023-07-14T19:09:58.327 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:57.854+0000 7f4079298700 1 --2- 172.21.15.40:0/196578480 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f406003aa40 0x7f406003cef0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:09:58.327 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:57.854+0000 7f4079298700 1 --2- 172.21.15.40:0/196578480 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f40741042f0 0x7f4074196ee0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:09:58.327 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:57.854+0000 7f4079298700 1 -- 172.21.15.40:0/196578480 >> 172.21.15.40:0/196578480 conn(0x7f40740ff9a0 msgr2=0x7f407418dd60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:09:58.327 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:57.854+0000 7f4079298700 1 -- 172.21.15.40:0/196578480 shutdown_connections 2023-07-14T19:09:58.327 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:57.854+0000 7f4079298700 1 -- 172.21.15.40:0/196578480 wait complete. 2023-07-14T19:09:58.537 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:09:58 smithi040 ceph-mon[96518]: mgrmap e13: x(active, since 6s) 2023-07-14T19:09:59.867 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:58.819+0000 7f5e062ea700 1 Processor -- start 2023-07-14T19:09:59.867 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:58.819+0000 7f5e062ea700 1 -- start start 2023-07-14T19:09:59.868 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:58.820+0000 7f5e062ea700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5e00105430 0x7f5e00105840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:09:59.868 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:58.820+0000 7f5e062ea700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e00105d80 con 0x7f5e00105430 2023-07-14T19:09:59.868 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:58.820+0000 7f5dfffff700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5e00105430 0x7f5e00105840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:09:59.868 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:58.820+0000 7f5dfffff700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5e00105430 0x7f5e00105840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:60086/0 (socket says 172.21.15.40:60086) 2023-07-14T19:09:59.868 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:58.820+0000 7f5dfffff700 1 -- 172.21.15.40:0/2616401335 learned_addr learned my addr 172.21.15.40:0/2616401335 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-07-14T19:09:59.868 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:58.820+0000 7f5dfffff700 1 -- 172.21.15.40:0/2616401335 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e00105ec0 con 0x7f5e00105430 2023-07-14T19:09:59.869 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:58.820+0000 7f5dfffff700 1 --2- 172.21.15.40:0/2616401335 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5e00105430 0x7f5e00105840 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f5de80098a0 tx=0x7f5de8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d447a1f9049aa736 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:09:59.869 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:58.820+0000 7f5dfeffd700 1 -- 172.21.15.40:0/2616401335 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5de8004160 con 0x7f5e00105430 2023-07-14T19:09:59.869 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:58.820+0000 7f5dfeffd700 1 -- 172.21.15.40:0/2616401335 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f5de80042c0 con 0x7f5e00105430 2023-07-14T19:09:59.869 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:58.820+0000 7f5dfeffd700 1 -- 172.21.15.40:0/2616401335 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5de80119b0 con 0x7f5e00105430 2023-07-14T19:09:59.869 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:58.821+0000 7f5e062ea700 1 -- 172.21.15.40:0/2616401335 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5e00105430 msgr2=0x7f5e00105840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:09:59.869 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:58.821+0000 7f5e062ea700 1 --2- 172.21.15.40:0/2616401335 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5e00105430 0x7f5e00105840 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f5de80098a0 tx=0x7f5de8009bb0 comp rx=0 tx=0).stop 2023-07-14T19:09:59.869 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:58.821+0000 7f5e062ea700 1 -- 172.21.15.40:0/2616401335 shutdown_connections 2023-07-14T19:09:59.870 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:58.821+0000 7f5e062ea700 1 --2- 172.21.15.40:0/2616401335 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5e00105430 0x7f5e00105840 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:09:59.870 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:58.821+0000 7f5e062ea700 1 -- 172.21.15.40:0/2616401335 >> 172.21.15.40:0/2616401335 conn(0x7f5e00100a80 msgr2=0x7f5e00102eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:09:59.870 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:58.821+0000 7f5e062ea700 1 -- 172.21.15.40:0/2616401335 shutdown_connections 2023-07-14T19:09:59.870 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:58.821+0000 7f5e062ea700 1 -- 172.21.15.40:0/2616401335 wait complete. 2023-07-14T19:09:59.870 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:58.822+0000 7f5e062ea700 1 Processor -- start 2023-07-14T19:09:59.870 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:58.822+0000 7f5e062ea700 1 -- start start 2023-07-14T19:09:59.870 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:58.822+0000 7f5e062ea700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5e00105430 0x7f5e00199180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:09:59.871 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:58.822+0000 7f5e062ea700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e001996c0 con 0x7f5e00105430 2023-07-14T19:09:59.871 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:58.822+0000 7f5dfffff700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5e00105430 0x7f5e00199180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:09:59.871 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:58.822+0000 7f5dfffff700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5e00105430 0x7f5e00199180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:60088/0 (socket says 172.21.15.40:60088) 2023-07-14T19:09:59.871 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:58.822+0000 7f5dfffff700 1 -- 172.21.15.40:0/934054051 learned_addr learned my addr 172.21.15.40:0/934054051 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-07-14T19:09:59.871 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:58.822+0000 7f5dfffff700 1 -- 172.21.15.40:0/934054051 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5de8009550 con 0x7f5e00105430 2023-07-14T19:09:59.871 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:58.823+0000 7f5dfffff700 1 --2- 172.21.15.40:0/934054051 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5e00105430 0x7f5e00199180 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f5de8003f00 tx=0x7f5de8011a30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:09:59.871 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:58.823+0000 7f5dfd7fa700 1 -- 172.21.15.40:0/934054051 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5de8003b20 con 0x7f5e00105430 2023-07-14T19:09:59.872 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:58.823+0000 7f5dfd7fa700 1 -- 172.21.15.40:0/934054051 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f5de8003c80 con 0x7f5e00105430 2023-07-14T19:09:59.872 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:58.823+0000 7f5dfd7fa700 1 -- 172.21.15.40:0/934054051 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5de8019770 con 0x7f5e00105430 2023-07-14T19:09:59.872 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:58.823+0000 7f5e062ea700 1 -- 172.21.15.40:0/934054051 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5e001998c0 con 0x7f5e00105430 2023-07-14T19:09:59.872 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:58.823+0000 7f5e062ea700 1 -- 172.21.15.40:0/934054051 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e00199ce0 con 0x7f5e00105430 2023-07-14T19:09:59.872 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:58.824+0000 7f5dfd7fa700 1 -- 172.21.15.40:0/934054051 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 13) v1 ==== 45539+0+0 (secure 0 0 0) 0x7f5de80199e0 con 0x7f5e00105430 2023-07-14T19:09:59.872 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:58.824+0000 7f5dfd7fa700 1 --2- 172.21.15.40:0/934054051 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f5dec038850 0x7f5dec03ad00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:09:59.872 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:58.824+0000 7f5e062ea700 1 -- 172.21.15.40:0/934054051 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5e00061c70 con 0x7f5e00105430 2023-07-14T19:09:59.873 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:58.824+0000 7f5dfd7fa700 1 -- 172.21.15.40:0/934054051 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f5de804cc40 con 0x7f5e00105430 2023-07-14T19:09:59.873 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:58.824+0000 7f5dff7fe700 1 --2- 172.21.15.40:0/934054051 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f5dec038850 0x7f5dec03ad00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:09:59.873 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:58.824+0000 7f5dff7fe700 1 --2- 172.21.15.40:0/934054051 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f5dec038850 0x7f5dec03ad00 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f5df0006fd0 tx=0x7f5df0006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:09:59.873 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:58.829+0000 7f5dfd7fa700 1 -- 172.21.15.40:0/934054051 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f5de80188f0 con 0x7f5e00105430 2023-07-14T19:09:59.873 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:59.402+0000 7f5e062ea700 1 -- 172.21.15.40:0/934054051 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command([{prefix=config-key set, key=mgr/dashboard/cluster/status}] v 0) v1 -- 0x7f5e00107330 con 0x7f5e00105430 2023-07-14T19:09:59.873 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:59.405+0000 7f5dfd7fa700 1 -- 172.21.15.40:0/934054051 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{prefix=config-key set, key=mgr/dashboard/cluster/status}]=0 set mgr/dashboard/cluster/status v28)=0 set mgr/dashboard/cluster/status v28) v1 ==== 153+0+0 (secure 0 0 0) 0x7f5de8018350 con 0x7f5e00105430 2023-07-14T19:09:59.873 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:59.407+0000 7f5e062ea700 1 -- 172.21.15.40:0/934054051 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f5dec038850 msgr2=0x7f5dec03ad00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:09:59.874 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:59.407+0000 7f5e062ea700 1 --2- 172.21.15.40:0/934054051 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f5dec038850 0x7f5dec03ad00 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f5df0006fd0 tx=0x7f5df0006e40 comp rx=0 tx=0).stop 2023-07-14T19:09:59.874 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:59.407+0000 7f5e062ea700 1 -- 172.21.15.40:0/934054051 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5e00105430 msgr2=0x7f5e00199180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:09:59.874 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:59.407+0000 7f5e062ea700 1 --2- 172.21.15.40:0/934054051 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5e00105430 0x7f5e00199180 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f5de8003f00 tx=0x7f5de8011a30 comp rx=0 tx=0).stop 2023-07-14T19:09:59.874 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:59.407+0000 7f5e062ea700 1 -- 172.21.15.40:0/934054051 shutdown_connections 2023-07-14T19:09:59.875 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:59.407+0000 7f5e062ea700 1 --2- 172.21.15.40:0/934054051 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f5dec038850 0x7f5dec03ad00 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:09:59.875 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:59.407+0000 7f5e062ea700 1 --2- 172.21.15.40:0/934054051 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5e00105430 0x7f5e00199180 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:09:59.875 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:59.407+0000 7f5e062ea700 1 -- 172.21.15.40:0/934054051 >> 172.21.15.40:0/934054051 conn(0x7f5e00100a80 msgr2=0x7f5e00101830 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:09:59.875 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:59.407+0000 7f5e062ea700 1 -- 172.21.15.40:0/934054051 shutdown_connections 2023-07-14T19:09:59.876 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2023-07-14T19:09:59.407+0000 7f5e062ea700 1 -- 172.21.15.40:0/934054051 wait complete. 2023-07-14T19:09:59.876 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr set mgr/dashboard/cluster/status 2023-07-14T19:09:59.876 INFO:teuthology.orchestra.run.smithi040.stdout:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2023-07-14T19:09:59.876 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-07-14T19:09:59.877 INFO:teuthology.orchestra.run.smithi040.stdout: sudo /home/ubuntu/cephtest/cephadm shell --fsid adc52f04-2279-11ee-9b33-001a4aab830c -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2023-07-14T19:09:59.877 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-07-14T19:09:59.877 INFO:teuthology.orchestra.run.smithi040.stdout:Or, if you are only running a single cluster on this host: 2023-07-14T19:09:59.878 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-07-14T19:09:59.878 INFO:teuthology.orchestra.run.smithi040.stdout: sudo /home/ubuntu/cephtest/cephadm shell 2023-07-14T19:09:59.878 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-07-14T19:09:59.878 INFO:teuthology.orchestra.run.smithi040.stdout:Please consider enabling telemetry to help improve Ceph: 2023-07-14T19:09:59.879 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-07-14T19:09:59.879 INFO:teuthology.orchestra.run.smithi040.stdout: ceph telemetry on 2023-07-14T19:09:59.879 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-07-14T19:09:59.879 INFO:teuthology.orchestra.run.smithi040.stdout:For more information see: 2023-07-14T19:09:59.880 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-07-14T19:09:59.880 INFO:teuthology.orchestra.run.smithi040.stdout: https://docs.ceph.com/en/latest/mgr/telemetry/ 2023-07-14T19:09:59.881 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-07-14T19:09:59.881 INFO:teuthology.orchestra.run.smithi040.stdout:Bootstrap complete. 2023-07-14T19:09:59.903 INFO:tasks.cephadm:Fetching config... 2023-07-14T19:09:59.903 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2023-07-14T19:09:59.904 DEBUG:teuthology.orchestra.run.smithi040:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2023-07-14T19:09:59.921 INFO:tasks.cephadm:Fetching client.admin keyring... 2023-07-14T19:09:59.922 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2023-07-14T19:09:59.922 DEBUG:teuthology.orchestra.run.smithi040:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2023-07-14T19:09:59.981 INFO:tasks.cephadm:Fetching mon keyring... 2023-07-14T19:09:59.982 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2023-07-14T19:09:59.982 DEBUG:teuthology.orchestra.run.smithi040:> sudo dd if=/var/lib/ceph/adc52f04-2279-11ee-9b33-001a4aab830c/mon.a/keyring of=/dev/stdout 2023-07-14T19:10:00.050 INFO:tasks.cephadm:Fetching pub ssh key... 2023-07-14T19:10:00.051 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2023-07-14T19:10:00.051 DEBUG:teuthology.orchestra.run.smithi040:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2023-07-14T19:10:00.110 INFO:tasks.cephadm:Installing pub ssh key for root users... 2023-07-14T19:10:00.110 DEBUG:teuthology.orchestra.run.smithi040:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCx4XHNpeOaW8LCJ1HelEIxHYdFfJK2jKnOC8RvOyvlpn71Ync460tNVCQJXb3RCoqlxh61OhMdqTjl2siDjQyHPkGNKKl/tOSQMsrLi+lIJ7HgISudmy7A3mm87JxOXHs0imEP2Muq5gGZFE9TfDnfezULGMtW97ztYiegJb3//fVTtfezL0JXdpFDaWE34GYoEb7CAnUIlNce+UZtxXCp28xJwyiJHIoBpHFmxL4QJ9BueRBKrvWMdFWAg1WMyK9XH1hXCK74PLLFa3uMpAa/Qq2R/h4Fr5gkPGUr5JoJEu4+Wy20hOHHOpC3nFCrRzPi+SKSWtbPWiS+eHRuTwY3sQGyU57tdtgclKRjj4J49Ckiw0nWggstdKE4EFqCskeOzT4kJuAu5yKqMNcQ4zhPw58/zZD9Ipag3i7TphF1j4QlvcQ4V57ovBNwkEHnyD0yI/C0JwU8B+Ko5GQLGMderXrKxFM78mrHdCwh9ok+kqWfLZME+IzqNkO/40g0HxE= ceph-adc52f04-2279-11ee-9b33-001a4aab830c' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2023-07-14T19:10:00.189 INFO:teuthology.orchestra.run.smithi040.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCx4XHNpeOaW8LCJ1HelEIxHYdFfJK2jKnOC8RvOyvlpn71Ync460tNVCQJXb3RCoqlxh61OhMdqTjl2siDjQyHPkGNKKl/tOSQMsrLi+lIJ7HgISudmy7A3mm87JxOXHs0imEP2Muq5gGZFE9TfDnfezULGMtW97ztYiegJb3//fVTtfezL0JXdpFDaWE34GYoEb7CAnUIlNce+UZtxXCp28xJwyiJHIoBpHFmxL4QJ9BueRBKrvWMdFWAg1WMyK9XH1hXCK74PLLFa3uMpAa/Qq2R/h4Fr5gkPGUr5JoJEu4+Wy20hOHHOpC3nFCrRzPi+SKSWtbPWiS+eHRuTwY3sQGyU57tdtgclKRjj4J49Ckiw0nWggstdKE4EFqCskeOzT4kJuAu5yKqMNcQ4zhPw58/zZD9Ipag3i7TphF1j4QlvcQ4V57ovBNwkEHnyD0yI/C0JwU8B+Ko5GQLGMderXrKxFM78mrHdCwh9ok+kqWfLZME+IzqNkO/40g0HxE= ceph-adc52f04-2279-11ee-9b33-001a4aab830c 2023-07-14T19:10:00.205 DEBUG:teuthology.orchestra.run.smithi130:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCx4XHNpeOaW8LCJ1HelEIxHYdFfJK2jKnOC8RvOyvlpn71Ync460tNVCQJXb3RCoqlxh61OhMdqTjl2siDjQyHPkGNKKl/tOSQMsrLi+lIJ7HgISudmy7A3mm87JxOXHs0imEP2Muq5gGZFE9TfDnfezULGMtW97ztYiegJb3//fVTtfezL0JXdpFDaWE34GYoEb7CAnUIlNce+UZtxXCp28xJwyiJHIoBpHFmxL4QJ9BueRBKrvWMdFWAg1WMyK9XH1hXCK74PLLFa3uMpAa/Qq2R/h4Fr5gkPGUr5JoJEu4+Wy20hOHHOpC3nFCrRzPi+SKSWtbPWiS+eHRuTwY3sQGyU57tdtgclKRjj4J49Ckiw0nWggstdKE4EFqCskeOzT4kJuAu5yKqMNcQ4zhPw58/zZD9Ipag3i7TphF1j4QlvcQ4V57ovBNwkEHnyD0yI/C0JwU8B+Ko5GQLGMderXrKxFM78mrHdCwh9ok+kqWfLZME+IzqNkO/40g0HxE= ceph-adc52f04-2279-11ee-9b33-001a4aab830c' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2023-07-14T19:10:00.251 INFO:teuthology.orchestra.run.smithi130.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCx4XHNpeOaW8LCJ1HelEIxHYdFfJK2jKnOC8RvOyvlpn71Ync460tNVCQJXb3RCoqlxh61OhMdqTjl2siDjQyHPkGNKKl/tOSQMsrLi+lIJ7HgISudmy7A3mm87JxOXHs0imEP2Muq5gGZFE9TfDnfezULGMtW97ztYiegJb3//fVTtfezL0JXdpFDaWE34GYoEb7CAnUIlNce+UZtxXCp28xJwyiJHIoBpHFmxL4QJ9BueRBKrvWMdFWAg1WMyK9XH1hXCK74PLLFa3uMpAa/Qq2R/h4Fr5gkPGUr5JoJEu4+Wy20hOHHOpC3nFCrRzPi+SKSWtbPWiS+eHRuTwY3sQGyU57tdtgclKRjj4J49Ckiw0nWggstdKE4EFqCskeOzT4kJuAu5yKqMNcQ4zhPw58/zZD9Ipag3i7TphF1j4QlvcQ4V57ovBNwkEHnyD0yI/C0JwU8B+Ko5GQLGMderXrKxFM78mrHdCwh9ok+kqWfLZME+IzqNkO/40g0HxE= ceph-adc52f04-2279-11ee-9b33-001a4aab830c 2023-07-14T19:10:00.263 DEBUG:teuthology.orchestra.run.smithi191:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCx4XHNpeOaW8LCJ1HelEIxHYdFfJK2jKnOC8RvOyvlpn71Ync460tNVCQJXb3RCoqlxh61OhMdqTjl2siDjQyHPkGNKKl/tOSQMsrLi+lIJ7HgISudmy7A3mm87JxOXHs0imEP2Muq5gGZFE9TfDnfezULGMtW97ztYiegJb3//fVTtfezL0JXdpFDaWE34GYoEb7CAnUIlNce+UZtxXCp28xJwyiJHIoBpHFmxL4QJ9BueRBKrvWMdFWAg1WMyK9XH1hXCK74PLLFa3uMpAa/Qq2R/h4Fr5gkPGUr5JoJEu4+Wy20hOHHOpC3nFCrRzPi+SKSWtbPWiS+eHRuTwY3sQGyU57tdtgclKRjj4J49Ckiw0nWggstdKE4EFqCskeOzT4kJuAu5yKqMNcQ4zhPw58/zZD9Ipag3i7TphF1j4QlvcQ4V57ovBNwkEHnyD0yI/C0JwU8B+Ko5GQLGMderXrKxFM78mrHdCwh9ok+kqWfLZME+IzqNkO/40g0HxE= ceph-adc52f04-2279-11ee-9b33-001a4aab830c' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2023-07-14T19:10:00.310 INFO:teuthology.orchestra.run.smithi191.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCx4XHNpeOaW8LCJ1HelEIxHYdFfJK2jKnOC8RvOyvlpn71Ync460tNVCQJXb3RCoqlxh61OhMdqTjl2siDjQyHPkGNKKl/tOSQMsrLi+lIJ7HgISudmy7A3mm87JxOXHs0imEP2Muq5gGZFE9TfDnfezULGMtW97ztYiegJb3//fVTtfezL0JXdpFDaWE34GYoEb7CAnUIlNce+UZtxXCp28xJwyiJHIoBpHFmxL4QJ9BueRBKrvWMdFWAg1WMyK9XH1hXCK74PLLFa3uMpAa/Qq2R/h4Fr5gkPGUr5JoJEu4+Wy20hOHHOpC3nFCrRzPi+SKSWtbPWiS+eHRuTwY3sQGyU57tdtgclKRjj4J49Ckiw0nWggstdKE4EFqCskeOzT4kJuAu5yKqMNcQ4zhPw58/zZD9Ipag3i7TphF1j4QlvcQ4V57ovBNwkEHnyD0yI/C0JwU8B+Ko5GQLGMderXrKxFM78mrHdCwh9ok+kqWfLZME+IzqNkO/40g0HxE= ceph-adc52f04-2279-11ee-9b33-001a4aab830c 2023-07-14T19:10:00.323 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid adc52f04-2279-11ee-9b33-001a4aab830c -- ceph config set mgr mgr/cephadm/allow_ptrace true 2023-07-14T19:10:00.722 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:10:00 smithi040 ceph-mon[96518]: from='client.? 172.21.15.40:0/934054051' entity='client.admin' 2023-07-14T19:10:00.888 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/adc52f04-2279-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T19:10:02.535 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:10:02.531+0000 7fc44a932700 1 -- 172.21.15.40:0/1733550560 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc444100270 msgr2=0x7fc444102650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:10:02.535 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:10:02.531+0000 7fc44a932700 1 --2- 172.21.15.40:0/1733550560 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc444100270 0x7fc444102650 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fc434009910 tx=0x7fc434009c20 comp rx=0 tx=0).stop 2023-07-14T19:10:02.535 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:10:02.532+0000 7fc44a932700 1 -- 172.21.15.40:0/1733550560 shutdown_connections 2023-07-14T19:10:02.536 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:10:02.532+0000 7fc44a932700 1 --2- 172.21.15.40:0/1733550560 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc444100270 0x7fc444102650 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:10:02.536 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:10:02.532+0000 7fc44a932700 1 -- 172.21.15.40:0/1733550560 >> 172.21.15.40:0/1733550560 conn(0x7fc4440f9bb0 msgr2=0x7fc4440fc000 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:10:02.536 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:10:02.532+0000 7fc44a932700 1 -- 172.21.15.40:0/1733550560 shutdown_connections 2023-07-14T19:10:02.537 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:10:02.532+0000 7fc44a932700 1 -- 172.21.15.40:0/1733550560 wait complete. 2023-07-14T19:10:02.537 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:10:02.532+0000 7fc44a932700 1 Processor -- start 2023-07-14T19:10:02.537 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:10:02.532+0000 7fc44a932700 1 -- start start 2023-07-14T19:10:02.537 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:10:02.533+0000 7fc44a932700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc444100270 0x7fc444194860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:10:02.538 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:10:02.533+0000 7fc44a932700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc444194da0 con 0x7fc444100270 2023-07-14T19:10:02.538 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:10:02.533+0000 7fc443fff700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc444100270 0x7fc444194860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:10:02.538 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:10:02.533+0000 7fc443fff700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc444100270 0x7fc444194860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:60094/0 (socket says 172.21.15.40:60094) 2023-07-14T19:10:02.538 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:10:02.533+0000 7fc443fff700 1 -- 172.21.15.40:0/1829024227 learned_addr learned my addr 172.21.15.40:0/1829024227 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-07-14T19:10:02.539 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:10:02.533+0000 7fc443fff700 1 -- 172.21.15.40:0/1829024227 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4340095f0 con 0x7fc444100270 2023-07-14T19:10:02.540 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:10:02.533+0000 7fc443fff700 1 --2- 172.21.15.40:0/1829024227 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc444100270 0x7fc444194860 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fc434004f40 tx=0x7fc4340057c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:10:02.541 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:10:02.533+0000 7fc4417fa700 1 -- 172.21.15.40:0/1829024227 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fc43401d070 con 0x7fc444100270 2023-07-14T19:10:02.541 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:10:02.533+0000 7fc44a932700 1 -- 172.21.15.40:0/1829024227 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc444194fa0 con 0x7fc444100270 2023-07-14T19:10:02.541 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:10:02.533+0000 7fc4417fa700 1 -- 172.21.15.40:0/1829024227 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc4340044e0 con 0x7fc444100270 2023-07-14T19:10:02.542 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:10:02.533+0000 7fc4417fa700 1 -- 172.21.15.40:0/1829024227 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fc434003a20 con 0x7fc444100270 2023-07-14T19:10:02.542 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:10:02.533+0000 7fc44a932700 1 -- 172.21.15.40:0/1829024227 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc4441953e0 con 0x7fc444100270 2023-07-14T19:10:02.542 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:10:02.534+0000 7fc4417fa700 1 -- 172.21.15.40:0/1829024227 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 13) v1 ==== 45539+0+0 (secure 0 0 0) 0x7fc434003ca0 con 0x7fc444100270 2023-07-14T19:10:02.542 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:10:02.534+0000 7fc44a932700 1 -- 172.21.15.40:0/1829024227 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc44418e300 con 0x7fc444100270 2023-07-14T19:10:02.543 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:10:02.535+0000 7fc4417fa700 1 --2- 172.21.15.40:0/1829024227 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7fc42c0387b0 0x7fc42c03ac60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:10:02.543 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:10:02.535+0000 7fc4417fa700 1 -- 172.21.15.40:0/1829024227 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fc43404c850 con 0x7fc444100270 2023-07-14T19:10:02.543 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:10:02.535+0000 7fc4437fe700 1 --2- 172.21.15.40:0/1829024227 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7fc42c0387b0 0x7fc42c03ac60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:10:02.543 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:10:02.535+0000 7fc4437fe700 1 --2- 172.21.15.40:0/1829024227 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7fc42c0387b0 0x7fc42c03ac60 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fc438006fd0 tx=0x7fc438006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:10:02.544 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:10:02.540+0000 7fc4417fa700 1 -- 172.21.15.40:0/1829024227 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fc43402a950 con 0x7fc444100270 2023-07-14T19:10:02.744 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:10:02.743+0000 7fc44a932700 1 -- 172.21.15.40:0/1829024227 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/allow_ptrace}] v 0) v1 -- 0x7fc44402c8e0 con 0x7fc444100270 2023-07-14T19:10:02.747 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:10:02.746+0000 7fc4417fa700 1 -- 172.21.15.40:0/1829024227 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/allow_ptrace}]=0 v9)=0 v9) v1 ==== 125+0+0 (secure 0 0 0) 0x7fc43402a360 con 0x7fc444100270 2023-07-14T19:10:02.750 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:10:02.748+0000 7fc44a932700 1 -- 172.21.15.40:0/1829024227 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7fc42c0387b0 msgr2=0x7fc42c03ac60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:10:02.751 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:10:02.748+0000 7fc44a932700 1 --2- 172.21.15.40:0/1829024227 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7fc42c0387b0 0x7fc42c03ac60 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fc438006fd0 tx=0x7fc438006e40 comp rx=0 tx=0).stop 2023-07-14T19:10:02.751 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:10:02.749+0000 7fc44a932700 1 -- 172.21.15.40:0/1829024227 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc444100270 msgr2=0x7fc444194860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:10:02.751 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:10:02.749+0000 7fc44a932700 1 --2- 172.21.15.40:0/1829024227 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc444100270 0x7fc444194860 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fc434004f40 tx=0x7fc4340057c0 comp rx=0 tx=0).stop 2023-07-14T19:10:02.751 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:10:02.749+0000 7fc44a932700 1 -- 172.21.15.40:0/1829024227 shutdown_connections 2023-07-14T19:10:02.752 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:10:02.749+0000 7fc44a932700 1 --2- 172.21.15.40:0/1829024227 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7fc42c0387b0 0x7fc42c03ac60 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:10:02.752 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:10:02.749+0000 7fc44a932700 1 --2- 172.21.15.40:0/1829024227 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc444100270 0x7fc444194860 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:10:02.752 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:10:02.749+0000 7fc44a932700 1 -- 172.21.15.40:0/1829024227 >> 172.21.15.40:0/1829024227 conn(0x7fc4440f9bb0 msgr2=0x7fc4440fa8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:10:02.752 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:10:02.749+0000 7fc44a932700 1 -- 172.21.15.40:0/1829024227 shutdown_connections 2023-07-14T19:10:02.752 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:10:02.749+0000 7fc44a932700 1 -- 172.21.15.40:0/1829024227 wait complete. 2023-07-14T19:10:03.253 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2023-07-14T19:10:03.254 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid adc52f04-2279-11ee-9b33-001a4aab830c -- ceph orch client-keyring set client.admin '*' --mode 0755 2023-07-14T19:10:03.813 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/adc52f04-2279-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T19:10:04.036 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:10:03 smithi040 ceph-mon[96518]: from='client.? 172.21.15.40:0/1829024227' entity='client.admin' 2023-07-14T19:10:04.036 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:10:03 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:10:04.037 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:10:03 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:10:04.037 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:10:03 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi040", "name": "osd_memory_target"}]: dispatch 2023-07-14T19:10:04.037 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:10:03 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:10:04.037 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:10:03 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T19:10:04.037 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:10:03 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:10:05.158 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:10:05.156+0000 7f5703490700 1 -- 172.21.15.40:0/2942382639 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f56fc0fd9d0 msgr2=0x7f56fc0fdde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:10:05.159 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:10:05.156+0000 7f5703490700 1 --2- 172.21.15.40:0/2942382639 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f56fc0fd9d0 0x7f56fc0fdde0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f56ec009910 tx=0x7f56ec009c20 comp rx=0 tx=0).stop 2023-07-14T19:10:05.159 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:10:05.156+0000 7f5703490700 1 -- 172.21.15.40:0/2942382639 shutdown_connections 2023-07-14T19:10:05.159 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:10:05.156+0000 7f5703490700 1 --2- 172.21.15.40:0/2942382639 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f56fc0fd9d0 0x7f56fc0fdde0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:10:05.159 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:10:05.156+0000 7f5703490700 1 -- 172.21.15.40:0/2942382639 >> 172.21.15.40:0/2942382639 conn(0x7f56fc0f9340 msgr2=0x7f56fc0fb770 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:10:05.159 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:10:05.156+0000 7f5703490700 1 -- 172.21.15.40:0/2942382639 shutdown_connections 2023-07-14T19:10:05.159 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:10:05.156+0000 7f5703490700 1 -- 172.21.15.40:0/2942382639 wait complete. 2023-07-14T19:10:05.160 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:10:05.157+0000 7f5703490700 1 Processor -- start 2023-07-14T19:10:05.160 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:10:05.157+0000 7f5703490700 1 -- start start 2023-07-14T19:10:05.160 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:10:05.157+0000 7f5703490700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f56fc0fd9d0 0x7f56fc194880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:10:05.160 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:10:05.157+0000 7f5703490700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56fc194dc0 con 0x7f56fc0fd9d0 2023-07-14T19:10:05.160 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:10:05.157+0000 7f570122c700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f56fc0fd9d0 0x7f56fc194880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:10:05.160 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:10:05.157+0000 7f570122c700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f56fc0fd9d0 0x7f56fc194880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:60100/0 (socket says 172.21.15.40:60100) 2023-07-14T19:10:05.161 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:10:05.157+0000 7f570122c700 1 -- 172.21.15.40:0/840216490 learned_addr learned my addr 172.21.15.40:0/840216490 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-07-14T19:10:05.161 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:10:05.157+0000 7f570122c700 1 -- 172.21.15.40:0/840216490 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56ec0095f0 con 0x7f56fc0fd9d0 2023-07-14T19:10:05.162 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:10:05.157+0000 7f570122c700 1 --2- 172.21.15.40:0/840216490 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f56fc0fd9d0 0x7f56fc194880 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f56ec006010 tx=0x7f56ec0057c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:10:05.162 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:10:05.157+0000 7f56f27fc700 1 -- 172.21.15.40:0/840216490 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f56ec01d070 con 0x7f56fc0fd9d0 2023-07-14T19:10:05.162 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:10:05.158+0000 7f56f27fc700 1 -- 172.21.15.40:0/840216490 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f56ec0044e0 con 0x7f56fc0fd9d0 2023-07-14T19:10:05.163 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:10:05.158+0000 7f5703490700 1 -- 172.21.15.40:0/840216490 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f56fc194fc0 con 0x7f56fc0fd9d0 2023-07-14T19:10:05.163 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:10:05.158+0000 7f56f27fc700 1 -- 172.21.15.40:0/840216490 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f56ec0039b0 con 0x7f56fc0fd9d0 2023-07-14T19:10:05.163 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:10:05.158+0000 7f5703490700 1 -- 172.21.15.40:0/840216490 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f56fc195400 con 0x7f56fc0fd9d0 2023-07-14T19:10:05.163 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:10:05.158+0000 7f56f27fc700 1 -- 172.21.15.40:0/840216490 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 13) v1 ==== 45539+0+0 (secure 0 0 0) 0x7f56ec003c20 con 0x7f56fc0fd9d0 2023-07-14T19:10:05.163 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:10:05.159+0000 7f56f27fc700 1 --2- 172.21.15.40:0/840216490 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f56e8038800 0x7f56e803acb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:10:05.163 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:10:05.159+0000 7f5703490700 1 -- 172.21.15.40:0/840216490 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f56fc18e2e0 con 0x7f56fc0fd9d0 2023-07-14T19:10:05.163 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:10:05.159+0000 7f56f27fc700 1 -- 172.21.15.40:0/840216490 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f56ec04c810 con 0x7f56fc0fd9d0 2023-07-14T19:10:05.164 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:10:05.159+0000 7f5700a2b700 1 --2- 172.21.15.40:0/840216490 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f56e8038800 0x7f56e803acb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:10:05.164 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:10:05.159+0000 7f5700a2b700 1 --2- 172.21.15.40:0/840216490 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f56e8038800 0x7f56e803acb0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f56f8006fd0 tx=0x7f56f8006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:10:05.165 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:10:05.163+0000 7f56f27fc700 1 -- 172.21.15.40:0/840216490 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f56ec02a360 con 0x7f56fc0fd9d0 2023-07-14T19:10:05.366 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:10:05.364+0000 7f5703490700 1 -- 172.21.15.40:0/840216490 --> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] -- mgr_command(tid 0: {"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}) v1 -- 0x7f56fc060b10 con 0x7f56e8038800 2023-07-14T19:10:05.369 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:10:05.368+0000 7f56f27fc700 1 -- 172.21.15.40:0/840216490 <== mgr.14152 v2:172.21.15.40:6800/1693027118 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f56fc060b10 con 0x7f56e8038800 2023-07-14T19:10:05.372 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:10:05.370+0000 7f5703490700 1 -- 172.21.15.40:0/840216490 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f56e8038800 msgr2=0x7f56e803acb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:10:05.372 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:10:05.370+0000 7f5703490700 1 --2- 172.21.15.40:0/840216490 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f56e8038800 0x7f56e803acb0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f56f8006fd0 tx=0x7f56f8006e40 comp rx=0 tx=0).stop 2023-07-14T19:10:05.372 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:10:05.370+0000 7f5703490700 1 -- 172.21.15.40:0/840216490 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f56fc0fd9d0 msgr2=0x7f56fc194880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:10:05.372 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:10:05.370+0000 7f5703490700 1 --2- 172.21.15.40:0/840216490 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f56fc0fd9d0 0x7f56fc194880 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f56ec006010 tx=0x7f56ec0057c0 comp rx=0 tx=0).stop 2023-07-14T19:10:05.372 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:10:05.370+0000 7f5703490700 1 -- 172.21.15.40:0/840216490 shutdown_connections 2023-07-14T19:10:05.372 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:10:05.370+0000 7f5703490700 1 --2- 172.21.15.40:0/840216490 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f56e8038800 0x7f56e803acb0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:10:05.373 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:10:05.370+0000 7f5703490700 1 --2- 172.21.15.40:0/840216490 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f56fc0fd9d0 0x7f56fc194880 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:10:05.373 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:10:05.370+0000 7f5703490700 1 -- 172.21.15.40:0/840216490 >> 172.21.15.40:0/840216490 conn(0x7f56fc0f9340 msgr2=0x7f56fc0fa070 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:10:05.373 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:10:05.370+0000 7f5703490700 1 -- 172.21.15.40:0/840216490 shutdown_connections 2023-07-14T19:10:05.373 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:10:05.370+0000 7f5703490700 1 -- 172.21.15.40:0/840216490 wait complete. 2023-07-14T19:10:05.880 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi130 2023-07-14T19:10:05.881 DEBUG:teuthology.orchestra.run.smithi130:> set -ex 2023-07-14T19:10:05.881 DEBUG:teuthology.orchestra.run.smithi130:> dd of=/etc/ceph/ceph.conf 2023-07-14T19:10:05.902 DEBUG:teuthology.orchestra.run.smithi130:> set -ex 2023-07-14T19:10:05.902 DEBUG:teuthology.orchestra.run.smithi130:> dd of=/etc/ceph/ceph.client.admin.keyring 2023-07-14T19:10:05.961 INFO:tasks.cephadm:Adding host smithi130 to orchestrator... 2023-07-14T19:10:05.962 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid adc52f04-2279-11ee-9b33-001a4aab830c -- ceph orch host add smithi130 2023-07-14T19:10:06.532 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/adc52f04-2279-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T19:10:06.721 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:10:06 smithi040 ceph-mon[96518]: from='client.14176 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T19:10:06.722 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:10:06 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:10:06.722 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:10:06 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T19:10:06.722 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:10:06 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:10:06.722 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:10:06 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T19:10:06.722 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:10:06 smithi040 ceph-mon[96518]: Updating smithi040:/etc/ceph/ceph.conf 2023-07-14T19:10:06.722 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:10:06 smithi040 ceph-mon[96518]: Updating smithi040:/var/lib/ceph/adc52f04-2279-11ee-9b33-001a4aab830c/config/ceph.conf 2023-07-14T19:10:07.722 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:10:07 smithi040 ceph-mon[96518]: Updating smithi040:/etc/ceph/ceph.client.admin.keyring 2023-07-14T19:10:07.722 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:10:07 smithi040 ceph-mon[96518]: Updating smithi040:/var/lib/ceph/adc52f04-2279-11ee-9b33-001a4aab830c/config/ceph.client.admin.keyring 2023-07-14T19:10:07.723 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:10:07 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:10:07.723 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:10:07 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:10:07.723 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:10:07 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:10:07.865 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:10:07.863+0000 7fc0d6fc6700 1 -- 172.21.15.40:0/2949987474 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc0d0104660 msgr2=0x7fc0d0104a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:10:07.865 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:10:07.863+0000 7fc0d6fc6700 1 --2- 172.21.15.40:0/2949987474 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc0d0104660 0x7fc0d0104a30 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fc0c4009910 tx=0x7fc0c4009c20 comp rx=0 tx=0).stop 2023-07-14T19:10:07.866 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:10:07.863+0000 7fc0d6fc6700 1 -- 172.21.15.40:0/2949987474 shutdown_connections 2023-07-14T19:10:07.866 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:10:07.863+0000 7fc0d6fc6700 1 --2- 172.21.15.40:0/2949987474 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc0d0104660 0x7fc0d0104a30 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:10:07.866 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:10:07.863+0000 7fc0d6fc6700 1 -- 172.21.15.40:0/2949987474 >> 172.21.15.40:0/2949987474 conn(0x7fc0d00fa140 msgr2=0x7fc0d00fc550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:10:07.866 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:10:07.863+0000 7fc0d6fc6700 1 -- 172.21.15.40:0/2949987474 shutdown_connections 2023-07-14T19:10:07.866 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:10:07.863+0000 7fc0d6fc6700 1 -- 172.21.15.40:0/2949987474 wait complete. 2023-07-14T19:10:07.868 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:10:07.864+0000 7fc0d6fc6700 1 Processor -- start 2023-07-14T19:10:07.869 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:10:07.864+0000 7fc0d6fc6700 1 -- start start 2023-07-14T19:10:07.869 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:10:07.864+0000 7fc0d6fc6700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc0d0104660 0x7fc0d00fa5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:10:07.869 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:10:07.864+0000 7fc0d6fc6700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0d00fc4c0 con 0x7fc0d0104660 2023-07-14T19:10:07.869 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:10:07.864+0000 7fc0d4d62700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc0d0104660 0x7fc0d00fa5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:10:07.869 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:10:07.864+0000 7fc0d4d62700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc0d0104660 0x7fc0d00fa5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:60106/0 (socket says 172.21.15.40:60106) 2023-07-14T19:10:07.869 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:10:07.864+0000 7fc0d4d62700 1 -- 172.21.15.40:0/2592075015 learned_addr learned my addr 172.21.15.40:0/2592075015 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-07-14T19:10:07.869 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:10:07.865+0000 7fc0d4d62700 1 -- 172.21.15.40:0/2592075015 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0c40095f0 con 0x7fc0d0104660 2023-07-14T19:10:07.870 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:10:07.865+0000 7fc0d4d62700 1 --2- 172.21.15.40:0/2592075015 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc0d0104660 0x7fc0d00fa5e0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fc0c4006010 tx=0x7fc0c4004ff0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:10:07.870 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:10:07.865+0000 7fc0cdffb700 1 -- 172.21.15.40:0/2592075015 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fc0c401d070 con 0x7fc0d0104660 2023-07-14T19:10:07.870 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:10:07.865+0000 7fc0cdffb700 1 -- 172.21.15.40:0/2592075015 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc0c400bc10 con 0x7fc0d0104660 2023-07-14T19:10:07.870 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:10:07.865+0000 7fc0d6fc6700 1 -- 172.21.15.40:0/2592075015 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc0d00fab20 con 0x7fc0d0104660 2023-07-14T19:10:07.870 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:10:07.865+0000 7fc0cdffb700 1 -- 172.21.15.40:0/2592075015 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fc0c4003910 con 0x7fc0d0104660 2023-07-14T19:10:07.871 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:10:07.865+0000 7fc0d6fc6700 1 -- 172.21.15.40:0/2592075015 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc0d00fb020 con 0x7fc0d0104660 2023-07-14T19:10:07.871 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:10:07.866+0000 7fc0cdffb700 1 -- 172.21.15.40:0/2592075015 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 13) v1 ==== 45539+0+0 (secure 0 0 0) 0x7fc0c4003b80 con 0x7fc0d0104660 2023-07-14T19:10:07.871 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:10:07.866+0000 7fc0d6fc6700 1 -- 172.21.15.40:0/2592075015 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc0d0061c70 con 0x7fc0d0104660 2023-07-14T19:10:07.871 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:10:07.866+0000 7fc0cdffb700 1 --2- 172.21.15.40:0/2592075015 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7fc0c0038800 0x7fc0c003acb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:10:07.871 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:10:07.866+0000 7fc0cdffb700 1 -- 172.21.15.40:0/2592075015 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fc0c404c480 con 0x7fc0d0104660 2023-07-14T19:10:07.871 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:10:07.866+0000 7fc0cffff700 1 --2- 172.21.15.40:0/2592075015 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7fc0c0038800 0x7fc0c003acb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:10:07.872 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:10:07.867+0000 7fc0cffff700 1 --2- 172.21.15.40:0/2592075015 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7fc0c0038800 0x7fc0c003acb0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fc0bc006fd0 tx=0x7fc0bc006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:10:07.872 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:10:07.871+0000 7fc0cdffb700 1 -- 172.21.15.40:0/2592075015 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fc0c400ba70 con 0x7fc0d0104660 2023-07-14T19:10:08.073 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:10:08.072+0000 7fc0d6fc6700 1 -- 172.21.15.40:0/2592075015 --> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi130", "target": ["mon-mgr", ""]}) v1 -- 0x7fc0d00fbbb0 con 0x7fc0c0038800 2023-07-14T19:10:09.221 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:10:08 smithi040 ceph-mon[96518]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi130", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T19:10:09.222 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:10:08 smithi040 ceph-mon[96518]: Deploying cephadm binary to smithi130 2023-07-14T19:10:10.830 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:10:10.828+0000 7fc0cdffb700 1 -- 172.21.15.40:0/2592075015 <== mgr.14152 v2:172.21.15.40:6800/1693027118 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+49 (secure 0 0 0) 0x7fc0d00fbbb0 con 0x7fc0c0038800 2023-07-14T19:10:10.830 INFO:teuthology.orchestra.run.smithi040.stdout:Added host 'smithi130' with addr '172.21.15.130' 2023-07-14T19:10:10.832 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:10:10.831+0000 7fc0d6fc6700 1 -- 172.21.15.40:0/2592075015 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7fc0c0038800 msgr2=0x7fc0c003acb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:10:10.833 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:10:10.831+0000 7fc0d6fc6700 1 --2- 172.21.15.40:0/2592075015 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7fc0c0038800 0x7fc0c003acb0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fc0bc006fd0 tx=0x7fc0bc006e40 comp rx=0 tx=0).stop 2023-07-14T19:10:10.833 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:10:10.831+0000 7fc0d6fc6700 1 -- 172.21.15.40:0/2592075015 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc0d0104660 msgr2=0x7fc0d00fa5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:10:10.833 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:10:10.831+0000 7fc0d6fc6700 1 --2- 172.21.15.40:0/2592075015 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc0d0104660 0x7fc0d00fa5e0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fc0c4006010 tx=0x7fc0c4004ff0 comp rx=0 tx=0).stop 2023-07-14T19:10:10.833 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:10:10.831+0000 7fc0d6fc6700 1 -- 172.21.15.40:0/2592075015 shutdown_connections 2023-07-14T19:10:10.833 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:10:10.831+0000 7fc0d6fc6700 1 --2- 172.21.15.40:0/2592075015 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7fc0c0038800 0x7fc0c003acb0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:10:10.833 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:10:10.831+0000 7fc0d6fc6700 1 --2- 172.21.15.40:0/2592075015 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc0d0104660 0x7fc0d00fa5e0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:10:10.833 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:10:10.831+0000 7fc0d6fc6700 1 -- 172.21.15.40:0/2592075015 >> 172.21.15.40:0/2592075015 conn(0x7fc0d00fa140 msgr2=0x7fc0d0102160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:10:10.834 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:10:10.831+0000 7fc0d6fc6700 1 -- 172.21.15.40:0/2592075015 shutdown_connections 2023-07-14T19:10:10.834 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:10:10.831+0000 7fc0d6fc6700 1 -- 172.21.15.40:0/2592075015 wait complete. 2023-07-14T19:10:11.300 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid adc52f04-2279-11ee-9b33-001a4aab830c -- ceph orch host ls --format=json 2023-07-14T19:10:11.863 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/adc52f04-2279-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T19:10:12.102 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:10:11 smithi040 ceph-mon[96518]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T19:10:12.103 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:10:11 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:10:12.103 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:10:11 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T19:10:12.104 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:10:11 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:10:12.972 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:10:12 smithi040 ceph-mon[96518]: Added host smithi130 2023-07-14T19:10:12.972 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:10:12 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:10:13.214 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:10:13.212+0000 7f6c89f97700 1 -- 172.21.15.40:0/535417397 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6c84104660 msgr2=0x7f6c84104a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:10:13.215 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:10:13.212+0000 7f6c89f97700 1 --2- 172.21.15.40:0/535417397 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6c84104660 0x7f6c84104a30 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f6c74009910 tx=0x7f6c74009c20 comp rx=0 tx=0).stop 2023-07-14T19:10:13.215 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:10:13.212+0000 7f6c89f97700 1 -- 172.21.15.40:0/535417397 shutdown_connections 2023-07-14T19:10:13.215 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:10:13.212+0000 7f6c89f97700 1 --2- 172.21.15.40:0/535417397 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6c84104660 0x7f6c84104a30 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:10:13.215 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:10:13.212+0000 7f6c89f97700 1 -- 172.21.15.40:0/535417397 >> 172.21.15.40:0/535417397 conn(0x7f6c840743e0 msgr2=0x7f6c840747e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:10:13.215 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:10:13.212+0000 7f6c89f97700 1 -- 172.21.15.40:0/535417397 shutdown_connections 2023-07-14T19:10:13.216 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:10:13.212+0000 7f6c89f97700 1 -- 172.21.15.40:0/535417397 wait complete. 2023-07-14T19:10:13.216 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:10:13.213+0000 7f6c89f97700 1 Processor -- start 2023-07-14T19:10:13.216 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:10:13.213+0000 7f6c89f97700 1 -- start start 2023-07-14T19:10:13.216 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:10:13.213+0000 7f6c89f97700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6c84104660 0x7f6c841986e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:10:13.216 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:10:13.213+0000 7f6c89f97700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c84198c20 con 0x7f6c84104660 2023-07-14T19:10:13.217 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:10:13.213+0000 7f6c837fe700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6c84104660 0x7f6c841986e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:10:13.217 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:10:13.213+0000 7f6c837fe700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6c84104660 0x7f6c841986e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:60132/0 (socket says 172.21.15.40:60132) 2023-07-14T19:10:13.217 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:10:13.213+0000 7f6c837fe700 1 -- 172.21.15.40:0/1432800076 learned_addr learned my addr 172.21.15.40:0/1432800076 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-07-14T19:10:13.218 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:10:13.213+0000 7f6c837fe700 1 -- 172.21.15.40:0/1432800076 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c740095f0 con 0x7f6c84104660 2023-07-14T19:10:13.218 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:10:13.214+0000 7f6c837fe700 1 --2- 172.21.15.40:0/1432800076 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6c84104660 0x7f6c841986e0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f6c74006010 tx=0x7f6c74004ff0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:10:13.218 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:10:13.214+0000 7f6c80ff9700 1 -- 172.21.15.40:0/1432800076 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6c7401d070 con 0x7f6c84104660 2023-07-14T19:10:13.219 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:10:13.214+0000 7f6c89f97700 1 -- 172.21.15.40:0/1432800076 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6c84198e80 con 0x7f6c84104660 2023-07-14T19:10:13.219 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:10:13.214+0000 7f6c80ff9700 1 -- 172.21.15.40:0/1432800076 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f6c7400bc10 con 0x7f6c84104660 2023-07-14T19:10:13.219 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:10:13.214+0000 7f6c80ff9700 1 -- 172.21.15.40:0/1432800076 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6c74003910 con 0x7f6c84104660 2023-07-14T19:10:13.219 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:10:13.214+0000 7f6c89f97700 1 -- 172.21.15.40:0/1432800076 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6c84199300 con 0x7f6c84104660 2023-07-14T19:10:13.219 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:10:13.215+0000 7f6c89f97700 1 -- 172.21.15.40:0/1432800076 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6c84061c70 con 0x7f6c84104660 2023-07-14T19:10:13.219 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:10:13.215+0000 7f6c80ff9700 1 -- 172.21.15.40:0/1432800076 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 13) v1 ==== 45539+0+0 (secure 0 0 0) 0x7f6c7400bd80 con 0x7f6c84104660 2023-07-14T19:10:13.219 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:10:13.215+0000 7f6c80ff9700 1 --2- 172.21.15.40:0/1432800076 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f6c7003a9f0 0x7f6c7003cea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:10:13.219 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:10:13.215+0000 7f6c80ff9700 1 -- 172.21.15.40:0/1432800076 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f6c7404c4c0 con 0x7f6c84104660 2023-07-14T19:10:13.220 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:10:13.215+0000 7f6c82ffd700 1 --2- 172.21.15.40:0/1432800076 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f6c7003a9f0 0x7f6c7003cea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:10:13.220 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:10:13.216+0000 7f6c82ffd700 1 --2- 172.21.15.40:0/1432800076 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f6c7003a9f0 0x7f6c7003cea0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f6c6c006fd0 tx=0x7f6c6c006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:10:13.221 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:10:13.220+0000 7f6c80ff9700 1 -- 172.21.15.40:0/1432800076 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f6c74029360 con 0x7f6c84104660 2023-07-14T19:10:13.425 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:10:13.423+0000 7f6c89f97700 1 -- 172.21.15.40:0/1432800076 --> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f6c84193130 con 0x7f6c7003a9f0 2023-07-14T19:10:13.426 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:10:13.425+0000 7f6c80ff9700 1 -- 172.21.15.40:0/1432800076 <== mgr.14152 v2:172.21.15.40:6800/1693027118 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+160 (secure 0 0 0) 0x7f6c84193130 con 0x7f6c7003a9f0 2023-07-14T19:10:13.426 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-07-14T19:10:13.427 INFO:teuthology.orchestra.run.smithi040.stdout:[{"addr": "172.21.15.40", "hostname": "smithi040", "labels": [], "status": ""}, {"addr": "172.21.15.130", "hostname": "smithi130", "labels": [], "status": ""}] 2023-07-14T19:10:13.428 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:10:13.427+0000 7f6c89f97700 1 -- 172.21.15.40:0/1432800076 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f6c7003a9f0 msgr2=0x7f6c7003cea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:10:13.428 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:10:13.427+0000 7f6c89f97700 1 --2- 172.21.15.40:0/1432800076 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f6c7003a9f0 0x7f6c7003cea0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f6c6c006fd0 tx=0x7f6c6c006e40 comp rx=0 tx=0).stop 2023-07-14T19:10:13.428 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:10:13.427+0000 7f6c89f97700 1 -- 172.21.15.40:0/1432800076 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6c84104660 msgr2=0x7f6c841986e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:10:13.429 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:10:13.427+0000 7f6c89f97700 1 --2- 172.21.15.40:0/1432800076 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6c84104660 0x7f6c841986e0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f6c74006010 tx=0x7f6c74004ff0 comp rx=0 tx=0).stop 2023-07-14T19:10:13.429 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:10:13.427+0000 7f6c89f97700 1 -- 172.21.15.40:0/1432800076 shutdown_connections 2023-07-14T19:10:13.429 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:10:13.427+0000 7f6c89f97700 1 --2- 172.21.15.40:0/1432800076 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f6c7003a9f0 0x7f6c7003cea0 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:10:13.429 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:10:13.427+0000 7f6c89f97700 1 --2- 172.21.15.40:0/1432800076 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6c84104660 0x7f6c841986e0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:10:13.429 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:10:13.427+0000 7f6c89f97700 1 -- 172.21.15.40:0/1432800076 >> 172.21.15.40:0/1432800076 conn(0x7f6c840743e0 msgr2=0x7f6c84105c00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:10:13.429 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:10:13.427+0000 7f6c89f97700 1 -- 172.21.15.40:0/1432800076 shutdown_connections 2023-07-14T19:10:13.429 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:10:13.427+0000 7f6c89f97700 1 -- 172.21.15.40:0/1432800076 wait complete. 2023-07-14T19:10:13.936 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi191 2023-07-14T19:10:13.936 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2023-07-14T19:10:13.937 DEBUG:teuthology.orchestra.run.smithi191:> dd of=/etc/ceph/ceph.conf 2023-07-14T19:10:13.957 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2023-07-14T19:10:13.957 DEBUG:teuthology.orchestra.run.smithi191:> dd of=/etc/ceph/ceph.client.admin.keyring 2023-07-14T19:10:14.017 INFO:tasks.cephadm:Adding host smithi191 to orchestrator... 2023-07-14T19:10:14.017 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid adc52f04-2279-11ee-9b33-001a4aab830c -- ceph orch host add smithi191 2023-07-14T19:10:14.045 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:10:13 smithi040 ceph-mon[96518]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T19:10:14.045 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:10:13 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:10:14.578 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/adc52f04-2279-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T19:10:15.095 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:10:14 smithi040 ceph-mon[96518]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-07-14T19:10:15.772 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:10:15.768+0000 7f2709dc4700 1 -- 172.21.15.40:0/172529360 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2704104670 msgr2=0x7f2704104a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:10:15.772 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:10:15.768+0000 7f2709dc4700 1 --2- 172.21.15.40:0/172529360 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2704104670 0x7f2704104a40 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f26f4009910 tx=0x7f26f4009c20 comp rx=0 tx=0).stop 2023-07-14T19:10:15.773 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:10:15.768+0000 7f2709dc4700 1 -- 172.21.15.40:0/172529360 shutdown_connections 2023-07-14T19:10:15.773 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:10:15.768+0000 7f2709dc4700 1 --2- 172.21.15.40:0/172529360 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2704104670 0x7f2704104a40 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:10:15.773 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:10:15.768+0000 7f2709dc4700 1 -- 172.21.15.40:0/172529360 >> 172.21.15.40:0/172529360 conn(0x7f27040fa150 msgr2=0x7f27040fc560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:10:15.773 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:10:15.768+0000 7f2709dc4700 1 -- 172.21.15.40:0/172529360 shutdown_connections 2023-07-14T19:10:15.773 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:10:15.768+0000 7f2709dc4700 1 -- 172.21.15.40:0/172529360 wait complete. 2023-07-14T19:10:15.774 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:10:15.769+0000 7f2709dc4700 1 Processor -- start 2023-07-14T19:10:15.774 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:10:15.769+0000 7f2709dc4700 1 -- start start 2023-07-14T19:10:15.774 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:10:15.769+0000 7f2709dc4700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2704104670 0x7f27041958f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:10:15.774 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:10:15.769+0000 7f2709dc4700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2704195e30 con 0x7f2704104670 2023-07-14T19:10:15.774 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:10:15.769+0000 7f27037fe700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2704104670 0x7f27041958f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:10:15.774 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:10:15.770+0000 7f27037fe700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2704104670 0x7f27041958f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:60138/0 (socket says 172.21.15.40:60138) 2023-07-14T19:10:15.775 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:10:15.770+0000 7f27037fe700 1 -- 172.21.15.40:0/2447470449 learned_addr learned my addr 172.21.15.40:0/2447470449 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-07-14T19:10:15.775 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:10:15.770+0000 7f27037fe700 1 -- 172.21.15.40:0/2447470449 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26f40095f0 con 0x7f2704104670 2023-07-14T19:10:15.775 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:10:15.770+0000 7f27037fe700 1 --2- 172.21.15.40:0/2447470449 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2704104670 0x7f27041958f0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f26f4004f40 tx=0x7f26f4005020 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:10:15.775 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:10:15.770+0000 7f2700ff9700 1 -- 172.21.15.40:0/2447470449 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f26f401d070 con 0x7f2704104670 2023-07-14T19:10:15.775 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:10:15.770+0000 7f2700ff9700 1 -- 172.21.15.40:0/2447470449 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f26f4004370 con 0x7f2704104670 2023-07-14T19:10:15.776 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:10:15.770+0000 7f2709dc4700 1 -- 172.21.15.40:0/2447470449 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2704196090 con 0x7f2704104670 2023-07-14T19:10:15.776 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:10:15.770+0000 7f2700ff9700 1 -- 172.21.15.40:0/2447470449 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f26f40039b0 con 0x7f2704104670 2023-07-14T19:10:15.776 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:10:15.770+0000 7f2709dc4700 1 -- 172.21.15.40:0/2447470449 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2704196590 con 0x7f2704104670 2023-07-14T19:10:15.777 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:10:15.771+0000 7f2700ff9700 1 -- 172.21.15.40:0/2447470449 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 13) v1 ==== 45539+0+0 (secure 0 0 0) 0x7f26f4003c20 con 0x7f2704104670 2023-07-14T19:10:15.777 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:10:15.771+0000 7f2709dc4700 1 -- 172.21.15.40:0/2447470449 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2704103910 con 0x7f2704104670 2023-07-14T19:10:15.777 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:10:15.771+0000 7f2700ff9700 1 --2- 172.21.15.40:0/2447470449 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f26ec038800 0x7f26ec03acb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:10:15.777 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:10:15.771+0000 7f2700ff9700 1 -- 172.21.15.40:0/2447470449 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f26f404c670 con 0x7f2704104670 2023-07-14T19:10:15.778 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:10:15.772+0000 7f2702ffd700 1 --2- 172.21.15.40:0/2447470449 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f26ec038800 0x7f26ec03acb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:10:15.778 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:10:15.772+0000 7f2702ffd700 1 --2- 172.21.15.40:0/2447470449 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f26ec038800 0x7f26ec03acb0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f26f8006fd0 tx=0x7f26f8006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:10:15.778 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:10:15.776+0000 7f2700ff9700 1 -- 172.21.15.40:0/2447470449 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f26f4029770 con 0x7f2704104670 2023-07-14T19:10:15.971 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:10:15 smithi040 ceph-mon[96518]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T19:10:15.983 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:10:15.982+0000 7f2709dc4700 1 -- 172.21.15.40:0/2447470449 --> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi191", "target": ["mon-mgr", ""]}) v1 -- 0x7f2704068870 con 0x7f26ec038800 2023-07-14T19:10:17.222 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:10:16 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:10:17.222 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:10:16 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:10:17.222 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:10:16 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:10:17.223 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:10:16 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:10:17.223 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:10:16 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi130", "name": "osd_memory_target"}]: dispatch 2023-07-14T19:10:17.223 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:10:16 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:10:17.223 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:10:16 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T19:10:17.223 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:10:16 smithi040 ceph-mon[96518]: Updating smithi130:/etc/ceph/ceph.conf 2023-07-14T19:10:17.224 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:10:16 smithi040 ceph-mon[96518]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi191", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T19:10:17.224 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:10:16 smithi040 ceph-mon[96518]: Updating smithi130:/var/lib/ceph/adc52f04-2279-11ee-9b33-001a4aab830c/config/ceph.conf 2023-07-14T19:10:17.224 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:10:16 smithi040 ceph-mon[96518]: Updating smithi130:/etc/ceph/ceph.client.admin.keyring 2023-07-14T19:10:17.224 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:10:16 smithi040 ceph-mon[96518]: Deploying cephadm binary to smithi191 2023-07-14T19:10:17.224 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:10:16 smithi040 ceph-mon[96518]: Updating smithi130:/var/lib/ceph/adc52f04-2279-11ee-9b33-001a4aab830c/config/ceph.client.admin.keyring 2023-07-14T19:10:17.224 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:10:16 smithi040 ceph-mon[96518]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T19:10:18.221 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:10:17 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:10:18.222 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:10:17 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:10:18.222 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:10:17 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:10:18.776 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:10:18.774+0000 7f2700ff9700 1 -- 172.21.15.40:0/2447470449 <== mgr.14152 v2:172.21.15.40:6800/1693027118 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+49 (secure 0 0 0) 0x7f2704068870 con 0x7f26ec038800 2023-07-14T19:10:18.776 INFO:teuthology.orchestra.run.smithi040.stdout:Added host 'smithi191' with addr '172.21.15.191' 2023-07-14T19:10:18.778 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:10:18.777+0000 7f2709dc4700 1 -- 172.21.15.40:0/2447470449 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f26ec038800 msgr2=0x7f26ec03acb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:10:18.778 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:10:18.777+0000 7f2709dc4700 1 --2- 172.21.15.40:0/2447470449 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f26ec038800 0x7f26ec03acb0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f26f8006fd0 tx=0x7f26f8006e40 comp rx=0 tx=0).stop 2023-07-14T19:10:18.778 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:10:18.777+0000 7f2709dc4700 1 -- 172.21.15.40:0/2447470449 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2704104670 msgr2=0x7f27041958f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:10:18.779 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:10:18.777+0000 7f2709dc4700 1 --2- 172.21.15.40:0/2447470449 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2704104670 0x7f27041958f0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f26f4004f40 tx=0x7f26f4005020 comp rx=0 tx=0).stop 2023-07-14T19:10:18.779 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:10:18.777+0000 7f2709dc4700 1 -- 172.21.15.40:0/2447470449 shutdown_connections 2023-07-14T19:10:18.779 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:10:18.777+0000 7f2709dc4700 1 --2- 172.21.15.40:0/2447470449 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f26ec038800 0x7f26ec03acb0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:10:18.779 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:10:18.777+0000 7f2709dc4700 1 --2- 172.21.15.40:0/2447470449 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2704104670 0x7f27041958f0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:10:18.779 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:10:18.777+0000 7f2709dc4700 1 -- 172.21.15.40:0/2447470449 >> 172.21.15.40:0/2447470449 conn(0x7f27040fa150 msgr2=0x7f27040fadb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:10:18.779 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:10:18.777+0000 7f2709dc4700 1 -- 172.21.15.40:0/2447470449 shutdown_connections 2023-07-14T19:10:18.779 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:10:18.777+0000 7f2709dc4700 1 -- 172.21.15.40:0/2447470449 wait complete. 2023-07-14T19:10:19.181 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:10:18 smithi040 ceph-mon[96518]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T19:10:19.181 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:10:18 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:10:19.181 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:10:18 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T19:10:19.236 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid adc52f04-2279-11ee-9b33-001a4aab830c -- ceph orch host ls --format=json 2023-07-14T19:10:19.839 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/adc52f04-2279-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T19:10:20.391 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:10:20 smithi040 ceph-mon[96518]: Added host smithi191 2023-07-14T19:10:20.392 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:10:20 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:10:20.392 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:10:20 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:10:21.084 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:10:21.082+0000 7ff7c2330700 1 -- 172.21.15.40:0/2889556757 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff7bc104660 msgr2=0x7ff7bc104a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:10:21.085 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:10:21.082+0000 7ff7c2330700 1 --2- 172.21.15.40:0/2889556757 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff7bc104660 0x7ff7bc104a30 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7ff7ac009910 tx=0x7ff7ac009c20 comp rx=0 tx=0).stop 2023-07-14T19:10:21.085 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:10:21.082+0000 7ff7c2330700 1 -- 172.21.15.40:0/2889556757 shutdown_connections 2023-07-14T19:10:21.085 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:10:21.082+0000 7ff7c2330700 1 --2- 172.21.15.40:0/2889556757 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff7bc104660 0x7ff7bc104a30 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:10:21.085 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:10:21.082+0000 7ff7c2330700 1 -- 172.21.15.40:0/2889556757 >> 172.21.15.40:0/2889556757 conn(0x7ff7bc0fa140 msgr2=0x7ff7bc0fc550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:10:21.085 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:10:21.082+0000 7ff7c2330700 1 -- 172.21.15.40:0/2889556757 shutdown_connections 2023-07-14T19:10:21.086 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:10:21.082+0000 7ff7c2330700 1 -- 172.21.15.40:0/2889556757 wait complete. 2023-07-14T19:10:21.086 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:10:21.083+0000 7ff7c2330700 1 Processor -- start 2023-07-14T19:10:21.086 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:10:21.083+0000 7ff7c2330700 1 -- start start 2023-07-14T19:10:21.087 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:10:21.083+0000 7ff7c2330700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff7bc104660 0x7ff7bc199790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:10:21.087 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:10:21.083+0000 7ff7c2330700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7bc199cd0 con 0x7ff7bc104660 2023-07-14T19:10:21.089 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:10:21.083+0000 7ff7bbfff700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff7bc104660 0x7ff7bc199790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:10:21.089 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:10:21.083+0000 7ff7bbfff700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff7bc104660 0x7ff7bc199790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:60164/0 (socket says 172.21.15.40:60164) 2023-07-14T19:10:21.089 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:10:21.083+0000 7ff7bbfff700 1 -- 172.21.15.40:0/2902107356 learned_addr learned my addr 172.21.15.40:0/2902107356 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-07-14T19:10:21.089 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:10:21.084+0000 7ff7bbfff700 1 -- 172.21.15.40:0/2902107356 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7ac0095f0 con 0x7ff7bc104660 2023-07-14T19:10:21.089 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:10:21.084+0000 7ff7bbfff700 1 --2- 172.21.15.40:0/2902107356 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff7bc104660 0x7ff7bc199790 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7ff7ac006010 tx=0x7ff7ac004ff0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:10:21.090 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:10:21.084+0000 7ff7b97fa700 1 -- 172.21.15.40:0/2902107356 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff7ac01d070 con 0x7ff7bc104660 2023-07-14T19:10:21.090 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:10:21.084+0000 7ff7b97fa700 1 -- 172.21.15.40:0/2902107356 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff7ac00bc10 con 0x7ff7bc104660 2023-07-14T19:10:21.090 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:10:21.084+0000 7ff7c2330700 1 -- 172.21.15.40:0/2902107356 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff7bc199f30 con 0x7ff7bc104660 2023-07-14T19:10:21.090 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:10:21.084+0000 7ff7b97fa700 1 -- 172.21.15.40:0/2902107356 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff7ac003910 con 0x7ff7bc104660 2023-07-14T19:10:21.090 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:10:21.084+0000 7ff7c2330700 1 -- 172.21.15.40:0/2902107356 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff7bc193810 con 0x7ff7bc104660 2023-07-14T19:10:21.090 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:10:21.085+0000 7ff7b97fa700 1 -- 172.21.15.40:0/2902107356 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 13) v1 ==== 45539+0+0 (secure 0 0 0) 0x7ff7ac003b80 con 0x7ff7bc104660 2023-07-14T19:10:21.091 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:10:21.085+0000 7ff7c2330700 1 -- 172.21.15.40:0/2902107356 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff7bc061c70 con 0x7ff7bc104660 2023-07-14T19:10:21.091 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:10:21.085+0000 7ff7b97fa700 1 --2- 172.21.15.40:0/2902107356 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7ff7a4038800 0x7ff7a403acb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:10:21.091 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:10:21.085+0000 7ff7b97fa700 1 -- 172.21.15.40:0/2902107356 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7ff7ac04c470 con 0x7ff7bc104660 2023-07-14T19:10:21.091 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:10:21.085+0000 7ff7bb7fe700 1 --2- 172.21.15.40:0/2902107356 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7ff7a4038800 0x7ff7a403acb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:10:21.091 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:10:21.086+0000 7ff7bb7fe700 1 --2- 172.21.15.40:0/2902107356 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7ff7a4038800 0x7ff7a403acb0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7ff7b0006fd0 tx=0x7ff7b0006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:10:21.092 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:10:21.090+0000 7ff7b97fa700 1 -- 172.21.15.40:0/2902107356 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7ff7ac00fd00 con 0x7ff7bc104660 2023-07-14T19:10:21.296 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:10:21.295+0000 7ff7c2330700 1 -- 172.21.15.40:0/2902107356 --> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7ff7bc1942d0 con 0x7ff7a4038800 2023-07-14T19:10:21.297 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:10:21.296+0000 7ff7b97fa700 1 -- 172.21.15.40:0/2902107356 <== mgr.14152 v2:172.21.15.40:6800/1693027118 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+240 (secure 0 0 0) 0x7ff7bc1942d0 con 0x7ff7a4038800 2023-07-14T19:10:21.297 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-07-14T19:10:21.298 INFO:teuthology.orchestra.run.smithi040.stdout:[{"addr": "172.21.15.40", "hostname": "smithi040", "labels": [], "status": ""}, {"addr": "172.21.15.130", "hostname": "smithi130", "labels": [], "status": ""}, {"addr": "172.21.15.191", "hostname": "smithi191", "labels": [], "status": ""}] 2023-07-14T19:10:21.299 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:10:21.298+0000 7ff7c2330700 1 -- 172.21.15.40:0/2902107356 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7ff7a4038800 msgr2=0x7ff7a403acb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:10:21.300 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:10:21.298+0000 7ff7c2330700 1 --2- 172.21.15.40:0/2902107356 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7ff7a4038800 0x7ff7a403acb0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7ff7b0006fd0 tx=0x7ff7b0006e40 comp rx=0 tx=0).stop 2023-07-14T19:10:21.300 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:10:21.298+0000 7ff7c2330700 1 -- 172.21.15.40:0/2902107356 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff7bc104660 msgr2=0x7ff7bc199790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:10:21.300 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:10:21.298+0000 7ff7c2330700 1 --2- 172.21.15.40:0/2902107356 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff7bc104660 0x7ff7bc199790 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7ff7ac006010 tx=0x7ff7ac004ff0 comp rx=0 tx=0).stop 2023-07-14T19:10:21.300 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:10:21.298+0000 7ff7c2330700 1 -- 172.21.15.40:0/2902107356 shutdown_connections 2023-07-14T19:10:21.300 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:10:21.298+0000 7ff7c2330700 1 --2- 172.21.15.40:0/2902107356 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7ff7a4038800 0x7ff7a403acb0 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:10:21.301 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:10:21.298+0000 7ff7c2330700 1 --2- 172.21.15.40:0/2902107356 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff7bc104660 0x7ff7bc199790 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:10:21.301 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:10:21.298+0000 7ff7c2330700 1 -- 172.21.15.40:0/2902107356 >> 172.21.15.40:0/2902107356 conn(0x7ff7bc0fa140 msgr2=0x7ff7bc0fada0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:10:21.301 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:10:21.298+0000 7ff7c2330700 1 -- 172.21.15.40:0/2902107356 shutdown_connections 2023-07-14T19:10:21.301 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:10:21.298+0000 7ff7c2330700 1 -- 172.21.15.40:0/2902107356 wait complete. 2023-07-14T19:10:21.435 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:10:21 smithi040 ceph-mon[96518]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T19:10:21.696 INFO:tasks.cephadm:Setting crush tunables to default 2023-07-14T19:10:21.697 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid adc52f04-2279-11ee-9b33-001a4aab830c -- ceph osd crush tunables default 2023-07-14T19:10:22.256 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/adc52f04-2279-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T19:10:22.472 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:10:22 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:10:22.472 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:10:22 smithi040 ceph-mon[96518]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-07-14T19:10:23.341 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:10:23 smithi040 ceph-mon[96518]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T19:10:23.527 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:10:23.525+0000 7fd13d2d4700 1 -- 172.21.15.40:0/3170361484 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd1381057c0 msgr2=0x7fd138105b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:10:23.527 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:10:23.525+0000 7fd13d2d4700 1 --2- 172.21.15.40:0/3170361484 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd1381057c0 0x7fd138105b90 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fd12c009910 tx=0x7fd12c009c20 comp rx=0 tx=0).stop 2023-07-14T19:10:23.528 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:10:23.525+0000 7fd13d2d4700 1 -- 172.21.15.40:0/3170361484 shutdown_connections 2023-07-14T19:10:23.528 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:10:23.525+0000 7fd13d2d4700 1 --2- 172.21.15.40:0/3170361484 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd1381057c0 0x7fd138105b90 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:10:23.528 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:10:23.525+0000 7fd13d2d4700 1 -- 172.21.15.40:0/3170361484 >> 172.21.15.40:0/3170361484 conn(0x7fd1380743e0 msgr2=0x7fd1380747e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:10:23.528 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:10:23.525+0000 7fd13d2d4700 1 -- 172.21.15.40:0/3170361484 shutdown_connections 2023-07-14T19:10:23.528 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:10:23.525+0000 7fd13d2d4700 1 -- 172.21.15.40:0/3170361484 wait complete. 2023-07-14T19:10:23.528 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:10:23.526+0000 7fd13d2d4700 1 Processor -- start 2023-07-14T19:10:23.529 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:10:23.526+0000 7fd13d2d4700 1 -- start start 2023-07-14T19:10:23.530 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:10:23.526+0000 7fd13d2d4700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd1381057c0 0x7fd13819a970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:10:23.531 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:10:23.526+0000 7fd13d2d4700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd13819aeb0 con 0x7fd1381057c0 2023-07-14T19:10:23.531 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:10:23.526+0000 7fd136ffd700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd1381057c0 0x7fd13819a970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:10:23.531 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:10:23.526+0000 7fd136ffd700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd1381057c0 0x7fd13819a970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:60170/0 (socket says 172.21.15.40:60170) 2023-07-14T19:10:23.531 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:10:23.526+0000 7fd136ffd700 1 -- 172.21.15.40:0/1079880469 learned_addr learned my addr 172.21.15.40:0/1079880469 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-07-14T19:10:23.531 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:10:23.527+0000 7fd136ffd700 1 -- 172.21.15.40:0/1079880469 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd12c0095f0 con 0x7fd1381057c0 2023-07-14T19:10:23.531 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:10:23.527+0000 7fd136ffd700 1 --2- 172.21.15.40:0/1079880469 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd1381057c0 0x7fd13819a970 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fd12c006010 tx=0x7fd12c004ff0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:10:23.531 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:10:23.527+0000 7fd11ffff700 1 -- 172.21.15.40:0/1079880469 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd12c01d070 con 0x7fd1381057c0 2023-07-14T19:10:23.532 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:10:23.527+0000 7fd11ffff700 1 -- 172.21.15.40:0/1079880469 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd12c00bc10 con 0x7fd1381057c0 2023-07-14T19:10:23.532 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:10:23.527+0000 7fd13d2d4700 1 -- 172.21.15.40:0/1079880469 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd13819b0b0 con 0x7fd1381057c0 2023-07-14T19:10:23.532 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:10:23.527+0000 7fd11ffff700 1 -- 172.21.15.40:0/1079880469 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd12c003910 con 0x7fd1381057c0 2023-07-14T19:10:23.532 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:10:23.527+0000 7fd13d2d4700 1 -- 172.21.15.40:0/1079880469 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd1381949f0 con 0x7fd1381057c0 2023-07-14T19:10:23.532 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:10:23.528+0000 7fd13d2d4700 1 -- 172.21.15.40:0/1079880469 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd138061c70 con 0x7fd1381057c0 2023-07-14T19:10:23.532 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:10:23.528+0000 7fd11ffff700 1 -- 172.21.15.40:0/1079880469 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 13) v1 ==== 45539+0+0 (secure 0 0 0) 0x7fd12c00bd80 con 0x7fd1381057c0 2023-07-14T19:10:23.532 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:10:23.529+0000 7fd11ffff700 1 --2- 172.21.15.40:0/1079880469 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7fd1200387b0 0x7fd12003ac60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:10:23.532 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:10:23.529+0000 7fd11ffff700 1 -- 172.21.15.40:0/1079880469 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fd12c04c940 con 0x7fd1381057c0 2023-07-14T19:10:23.533 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:10:23.529+0000 7fd1367fc700 1 --2- 172.21.15.40:0/1079880469 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7fd1200387b0 0x7fd12003ac60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:10:23.533 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:10:23.529+0000 7fd1367fc700 1 --2- 172.21.15.40:0/1079880469 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7fd1200387b0 0x7fd12003ac60 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fd128006fd0 tx=0x7fd128006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:10:23.534 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:10:23.533+0000 7fd11ffff700 1 -- 172.21.15.40:0/1079880469 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fd12c00e3e0 con 0x7fd1381057c0 2023-07-14T19:10:23.739 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:10:23.737+0000 7fd13d2d4700 1 -- 172.21.15.40:0/1079880469 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7fd1381953b0 con 0x7fd1381057c0 2023-07-14T19:10:24.203 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:10:24.201+0000 7fd11ffff700 1 -- 172.21.15.40:0/1079880469 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v4) v1 ==== 124+0+0 (secure 0 0 0) 0x7fd12c026030 con 0x7fd1381057c0 2023-07-14T19:10:24.205 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:10:24.203+0000 7fd13d2d4700 1 -- 172.21.15.40:0/1079880469 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7fd1200387b0 msgr2=0x7fd12003ac60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:10:24.206 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:10:24.203+0000 7fd13d2d4700 1 --2- 172.21.15.40:0/1079880469 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7fd1200387b0 0x7fd12003ac60 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fd128006fd0 tx=0x7fd128006e40 comp rx=0 tx=0).stop 2023-07-14T19:10:24.206 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:10:24.203+0000 7fd13d2d4700 1 -- 172.21.15.40:0/1079880469 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd1381057c0 msgr2=0x7fd13819a970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:10:24.206 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:10:24.203+0000 7fd13d2d4700 1 --2- 172.21.15.40:0/1079880469 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd1381057c0 0x7fd13819a970 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fd12c006010 tx=0x7fd12c004ff0 comp rx=0 tx=0).stop 2023-07-14T19:10:24.206 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:10:24.203+0000 7fd13d2d4700 1 -- 172.21.15.40:0/1079880469 shutdown_connections 2023-07-14T19:10:24.206 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:10:24.203+0000 7fd13d2d4700 1 --2- 172.21.15.40:0/1079880469 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7fd1200387b0 0x7fd12003ac60 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:10:24.207 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:10:24.203+0000 7fd13d2d4700 1 --2- 172.21.15.40:0/1079880469 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd1381057c0 0x7fd13819a970 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:10:24.207 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:10:24.203+0000 7fd13d2d4700 1 -- 172.21.15.40:0/1079880469 >> 172.21.15.40:0/1079880469 conn(0x7fd1380743e0 msgr2=0x7fd138106d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:10:24.207 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:10:24.203+0000 7fd13d2d4700 1 -- 172.21.15.40:0/1079880469 shutdown_connections 2023-07-14T19:10:24.207 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:10:24.203+0000 7fd13d2d4700 1 -- 172.21.15.40:0/1079880469 wait complete. 2023-07-14T19:10:24.207 INFO:teuthology.orchestra.run.smithi040.stderr:adjusted tunables profile to default 2023-07-14T19:10:24.471 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:10:24 smithi040 ceph-mon[96518]: from='client.? 172.21.15.40:0/1079880469' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2023-07-14T19:10:24.472 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:10:24 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:10:24.472 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:10:24 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:10:24.472 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:10:24 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:10:24.472 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:10:24 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:10:24.472 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:10:24 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi191", "name": "osd_memory_target"}]: dispatch 2023-07-14T19:10:24.473 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:10:24 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:10:24.473 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:10:24 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T19:10:24.704 INFO:tasks.cephadm:Adding mon.a on smithi040 2023-07-14T19:10:24.705 INFO:tasks.cephadm:Adding mon.b on smithi130 2023-07-14T19:10:24.705 INFO:tasks.cephadm:Adding mon.c on smithi191 2023-07-14T19:10:24.705 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid adc52f04-2279-11ee-9b33-001a4aab830c -- ceph orch apply mon '3;smithi040:172.21.15.40=a;smithi130:172.21.15.130=b;smithi191:172.21.15.191=c' 2023-07-14T19:10:25.288 INFO:teuthology.orchestra.run.smithi191.stderr:Inferring config /var/lib/ceph/adc52f04-2279-11ee-9b33-001a4aab830c/config/ceph.conf 2023-07-14T19:10:25.472 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:10:25 smithi040 ceph-mon[96518]: Updating smithi191:/etc/ceph/ceph.conf 2023-07-14T19:10:25.472 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:10:25 smithi040 ceph-mon[96518]: Updating smithi191:/var/lib/ceph/adc52f04-2279-11ee-9b33-001a4aab830c/config/ceph.conf 2023-07-14T19:10:25.472 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:10:25 smithi040 ceph-mon[96518]: from='client.? 172.21.15.40:0/1079880469' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2023-07-14T19:10:25.473 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:10:25 smithi040 ceph-mon[96518]: osdmap e4: 0 total, 0 up, 0 in 2023-07-14T19:10:25.473 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:10:25 smithi040 ceph-mon[96518]: Updating smithi191:/etc/ceph/ceph.client.admin.keyring 2023-07-14T19:10:25.473 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:10:25 smithi040 ceph-mon[96518]: Updating smithi191:/var/lib/ceph/adc52f04-2279-11ee-9b33-001a4aab830c/config/ceph.client.admin.keyring 2023-07-14T19:10:25.473 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:10:25 smithi040 ceph-mon[96518]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T19:10:25.473 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:10:25 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:10:25.473 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:10:25 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:10:25.473 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:10:25 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:10:26.514 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:10:26.512+0000 7fbbcc5d4700 1 -- 172.21.15.191:0/3388523858 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbbc4104660 msgr2=0x7fbbc4104a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:10:26.515 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:10:26.512+0000 7fbbcc5d4700 1 --2- 172.21.15.191:0/3388523858 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbbc4104660 0x7fbbc4104a30 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fbbb4009910 tx=0x7fbbb4009c20 comp rx=0 tx=0).stop 2023-07-14T19:10:26.515 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:10:26.512+0000 7fbbcc5d4700 1 -- 172.21.15.191:0/3388523858 shutdown_connections 2023-07-14T19:10:26.515 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:10:26.512+0000 7fbbcc5d4700 1 --2- 172.21.15.191:0/3388523858 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbbc4104660 0x7fbbc4104a30 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:10:26.515 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:10:26.512+0000 7fbbcc5d4700 1 -- 172.21.15.191:0/3388523858 >> 172.21.15.191:0/3388523858 conn(0x7fbbc40743e0 msgr2=0x7fbbc40747e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:10:26.515 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:10:26.512+0000 7fbbcc5d4700 1 -- 172.21.15.191:0/3388523858 shutdown_connections 2023-07-14T19:10:26.515 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:10:26.512+0000 7fbbcc5d4700 1 -- 172.21.15.191:0/3388523858 wait complete. 2023-07-14T19:10:26.516 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:10:26.512+0000 7fbbcc5d4700 1 Processor -- start 2023-07-14T19:10:26.516 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:10:26.512+0000 7fbbcc5d4700 1 -- start start 2023-07-14T19:10:26.516 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:10:26.513+0000 7fbbcc5d4700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbbc4104660 0x7fbbc41998e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:10:26.516 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:10:26.513+0000 7fbbcc5d4700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbbc4199e20 con 0x7fbbc4104660 2023-07-14T19:10:26.516 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:10:26.513+0000 7fbbca370700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbbc4104660 0x7fbbc41998e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:10:26.516 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:10:26.513+0000 7fbbca370700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbbc4104660 0x7fbbc41998e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.191:38232/0 (socket says 172.21.15.191:38232) 2023-07-14T19:10:26.516 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:10:26.513+0000 7fbbca370700 1 -- 172.21.15.191:0/2874790698 learned_addr learned my addr 172.21.15.191:0/2874790698 (peer_addr_for_me v2:172.21.15.191:0/0) 2023-07-14T19:10:26.518 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:10:26.513+0000 7fbbca370700 1 -- 172.21.15.191:0/2874790698 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbbb40095f0 con 0x7fbbc4104660 2023-07-14T19:10:26.518 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:10:26.513+0000 7fbbca370700 1 --2- 172.21.15.191:0/2874790698 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbbc4104660 0x7fbbc41998e0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fbbb4004f40 tx=0x7fbbb4005020 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:10:26.518 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:10:26.513+0000 7fbbbb7fe700 1 -- 172.21.15.191:0/2874790698 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fbbb401d070 con 0x7fbbc4104660 2023-07-14T19:10:26.518 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:10:26.514+0000 7fbbcc5d4700 1 -- 172.21.15.191:0/2874790698 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbbc419a080 con 0x7fbbc4104660 2023-07-14T19:10:26.518 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:10:26.514+0000 7fbbbb7fe700 1 -- 172.21.15.191:0/2874790698 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fbbb4004370 con 0x7fbbc4104660 2023-07-14T19:10:26.518 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:10:26.514+0000 7fbbcc5d4700 1 -- 172.21.15.191:0/2874790698 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbbc41938d0 con 0x7fbbc4104660 2023-07-14T19:10:26.518 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:10:26.514+0000 7fbbbb7fe700 1 -- 172.21.15.191:0/2874790698 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fbbb40039a0 con 0x7fbbc4104660 2023-07-14T19:10:26.519 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:10:26.514+0000 7fbbbb7fe700 1 -- 172.21.15.191:0/2874790698 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 13) v1 ==== 45539+0+0 (secure 0 0 0) 0x7fbbb4003b00 con 0x7fbbc4104660 2023-07-14T19:10:26.519 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:10:26.514+0000 7fbbcc5d4700 1 -- 172.21.15.191:0/2874790698 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbbc4061c70 con 0x7fbbc4104660 2023-07-14T19:10:26.519 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:10:26.515+0000 7fbbbb7fe700 1 --2- 172.21.15.191:0/2874790698 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7fbbb003a9c0 0x7fbbb003ce70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:10:26.519 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:10:26.515+0000 7fbbbb7fe700 1 -- 172.21.15.191:0/2874790698 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fbbb404c440 con 0x7fbbc4104660 2023-07-14T19:10:26.519 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:10:26.515+0000 7fbbc9b6f700 1 --2- 172.21.15.191:0/2874790698 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7fbbb003a9c0 0x7fbbb003ce70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:10:26.519 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:10:26.515+0000 7fbbc9b6f700 1 --2- 172.21.15.191:0/2874790698 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7fbbb003a9c0 0x7fbbb003ce70 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fbbc0006fd0 tx=0x7fbbc0006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:10:26.521 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:10:26.520+0000 7fbbbb7fe700 1 -- 172.21.15.191:0/2874790698 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fbbb400f460 con 0x7fbbc4104660 2023-07-14T19:10:26.713 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:10:26.712+0000 7fbbcc5d4700 1 -- 172.21.15.191:0/2874790698 --> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi040:172.21.15.40=a;smithi130:172.21.15.130=b;smithi191:172.21.15.191=c", "target": ["mon-mgr", ""]}) v1 -- 0x7fbbc41944c0 con 0x7fbbb003a9c0 2023-07-14T19:10:26.723 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:10:26.721+0000 7fbbbb7fe700 1 -- 172.21.15.191:0/2874790698 <== mgr.14152 v2:172.21.15.40:6800/1693027118 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7fbbc41944c0 con 0x7fbbb003a9c0 2023-07-14T19:10:26.723 INFO:teuthology.orchestra.run.smithi191.stdout:Scheduled mon update... 2023-07-14T19:10:26.725 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:10:26.724+0000 7fbbcc5d4700 1 -- 172.21.15.191:0/2874790698 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7fbbb003a9c0 msgr2=0x7fbbb003ce70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:10:26.726 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:10:26.724+0000 7fbbcc5d4700 1 --2- 172.21.15.191:0/2874790698 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7fbbb003a9c0 0x7fbbb003ce70 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fbbc0006fd0 tx=0x7fbbc0006e40 comp rx=0 tx=0).stop 2023-07-14T19:10:26.726 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:10:26.724+0000 7fbbcc5d4700 1 -- 172.21.15.191:0/2874790698 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbbc4104660 msgr2=0x7fbbc41998e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:10:26.726 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:10:26.724+0000 7fbbcc5d4700 1 --2- 172.21.15.191:0/2874790698 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbbc4104660 0x7fbbc41998e0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fbbb4004f40 tx=0x7fbbb4005020 comp rx=0 tx=0).stop 2023-07-14T19:10:26.726 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:10:26.724+0000 7fbbcc5d4700 1 -- 172.21.15.191:0/2874790698 shutdown_connections 2023-07-14T19:10:26.726 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:10:26.724+0000 7fbbcc5d4700 1 --2- 172.21.15.191:0/2874790698 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7fbbb003a9c0 0x7fbbb003ce70 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:10:26.726 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:10:26.724+0000 7fbbcc5d4700 1 --2- 172.21.15.191:0/2874790698 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbbc4104660 0x7fbbc41998e0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:10:26.726 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:10:26.724+0000 7fbbcc5d4700 1 -- 172.21.15.191:0/2874790698 >> 172.21.15.191:0/2874790698 conn(0x7fbbc40743e0 msgr2=0x7fbbc4105c00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:10:26.727 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:10:26.724+0000 7fbbcc5d4700 1 -- 172.21.15.191:0/2874790698 shutdown_connections 2023-07-14T19:10:26.727 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:10:26.724+0000 7fbbcc5d4700 1 -- 172.21.15.191:0/2874790698 wait complete. 2023-07-14T19:10:27.154 DEBUG:teuthology.orchestra.run.smithi130:mon.b> sudo journalctl -f -n 0 -u ceph-adc52f04-2279-11ee-9b33-001a4aab830c@mon.b.service 2023-07-14T19:10:27.157 DEBUG:teuthology.orchestra.run.smithi191:mon.c> sudo journalctl -f -n 0 -u ceph-adc52f04-2279-11ee-9b33-001a4aab830c@mon.c.service 2023-07-14T19:10:27.161 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2023-07-14T19:10:27.161 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid adc52f04-2279-11ee-9b33-001a4aab830c -- ceph mon dump -f json 2023-07-14T19:10:27.190 INFO:journalctl@ceph.mon.c.smithi191.stdout:-- Logs begin at Fri 2023-07-14 18:50:54 UTC. -- 2023-07-14T19:10:27.193 INFO:journalctl@ceph.mon.b.smithi130.stdout:-- Logs begin at Fri 2023-07-14 18:50:32 UTC. -- 2023-07-14T19:10:27.972 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:10:27 smithi040 ceph-mon[96518]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi040:172.21.15.40=a;smithi130:172.21.15.130=b;smithi191:172.21.15.191=c", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T19:10:27.972 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:10:27 smithi040 ceph-mon[96518]: Saving service mon spec with placement smithi040:172.21.15.40=a;smithi130:172.21.15.130=b;smithi191:172.21.15.191=c;count:3 2023-07-14T19:10:27.972 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:10:27 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:10:27.972 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:10:27 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T19:10:27.972 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:10:27 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:10:27.973 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:10:27 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T19:10:27.973 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:10:27 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:10:27.973 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:10:27 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-07-14T19:10:27.973 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:10:27 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:10:27.973 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:10:27 smithi040 ceph-mon[96518]: Deploying daemon mon.c on smithi191 2023-07-14T19:10:27.973 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:10:27 smithi040 ceph-mon[96518]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T19:10:28.093 INFO:teuthology.orchestra.run.smithi191.stderr:Inferring config /var/lib/ceph/adc52f04-2279-11ee-9b33-001a4aab830c/config/ceph.conf 2023-07-14T19:10:29.221 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:10:28 smithi040 ceph-mon[96518]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T19:10:30.052 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:10:30.049+0000 7fa4ebfc5700 1 -- 172.21.15.191:0/2238441480 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa4e41057c0 msgr2=0x7fa4e4105b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:10:30.052 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:10:30.049+0000 7fa4ebfc5700 1 --2- 172.21.15.191:0/2238441480 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa4e41057c0 0x7fa4e4105b90 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fa4e0009910 tx=0x7fa4e0009c20 comp rx=0 tx=0).stop 2023-07-14T19:10:30.052 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:10:30.050+0000 7fa4ebfc5700 1 -- 172.21.15.191:0/2238441480 shutdown_connections 2023-07-14T19:10:30.052 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:10:30.050+0000 7fa4ebfc5700 1 --2- 172.21.15.191:0/2238441480 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa4e41057c0 0x7fa4e4105b90 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:10:30.053 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:10:30.050+0000 7fa4ebfc5700 1 -- 172.21.15.191:0/2238441480 >> 172.21.15.191:0/2238441480 conn(0x7fa4e40743e0 msgr2=0x7fa4e40747e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:10:30.053 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:10:30.050+0000 7fa4ebfc5700 1 -- 172.21.15.191:0/2238441480 shutdown_connections 2023-07-14T19:10:30.053 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:10:30.050+0000 7fa4ebfc5700 1 -- 172.21.15.191:0/2238441480 wait complete. 2023-07-14T19:10:30.053 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:10:30.050+0000 7fa4ebfc5700 1 Processor -- start 2023-07-14T19:10:30.053 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:10:30.050+0000 7fa4ebfc5700 1 -- start start 2023-07-14T19:10:30.054 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:10:30.051+0000 7fa4ebfc5700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa4e41057c0 0x7fa4e419bea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:10:30.055 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:10:30.051+0000 7fa4ebfc5700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4e419c3e0 con 0x7fa4e41057c0 2023-07-14T19:10:30.055 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:10:30.051+0000 7fa4e9d61700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa4e41057c0 0x7fa4e419bea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:10:30.059 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:10:30.052+0000 7fa4e9d61700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa4e41057c0 0x7fa4e419bea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.191:38238/0 (socket says 172.21.15.191:38238) 2023-07-14T19:10:30.059 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:10:30.052+0000 7fa4e9d61700 1 -- 172.21.15.191:0/855704549 learned_addr learned my addr 172.21.15.191:0/855704549 (peer_addr_for_me v2:172.21.15.191:0/0) 2023-07-14T19:10:30.059 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:10:30.058+0000 7fa4e9d61700 1 -- 172.21.15.191:0/855704549 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4e00095f0 con 0x7fa4e41057c0 2023-07-14T19:10:30.061 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:10:30.060+0000 7fa4e9d61700 1 --2- 172.21.15.191:0/855704549 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa4e41057c0 0x7fa4e419bea0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fa4e0004f40 tx=0x7fa4e0005020 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:10:30.062 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:10:30.060+0000 7fa4daffd700 1 -- 172.21.15.191:0/855704549 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa4e001d070 con 0x7fa4e41057c0 2023-07-14T19:10:30.062 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:10:30.060+0000 7fa4daffd700 1 -- 172.21.15.191:0/855704549 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa4e0004370 con 0x7fa4e41057c0 2023-07-14T19:10:30.063 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:10:30.060+0000 7fa4ebfc5700 1 -- 172.21.15.191:0/855704549 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa4e419c5e0 con 0x7fa4e41057c0 2023-07-14T19:10:30.063 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:10:30.060+0000 7fa4daffd700 1 -- 172.21.15.191:0/855704549 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa4e00039b0 con 0x7fa4e41057c0 2023-07-14T19:10:30.064 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:10:30.060+0000 7fa4ebfc5700 1 -- 172.21.15.191:0/855704549 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa4e419ca80 con 0x7fa4e41057c0 2023-07-14T19:10:30.065 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:10:30.061+0000 7fa4ebfc5700 1 -- 172.21.15.191:0/855704549 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa4e4061c70 con 0x7fa4e41057c0 2023-07-14T19:10:30.065 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:10:30.062+0000 7fa4daffd700 1 -- 172.21.15.191:0/855704549 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 13) v1 ==== 45539+0+0 (secure 0 0 0) 0x7fa4e0003b10 con 0x7fa4e41057c0 2023-07-14T19:10:30.065 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:10:30.062+0000 7fa4daffd700 1 --2- 172.21.15.191:0/855704549 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7fa4d0038800 0x7fa4d003acb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:10:30.066 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:10:30.062+0000 7fa4daffd700 1 -- 172.21.15.191:0/855704549 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fa4e004c730 con 0x7fa4e41057c0 2023-07-14T19:10:30.066 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:10:30.062+0000 7fa4e9560700 1 --2- 172.21.15.191:0/855704549 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7fa4d0038800 0x7fa4d003acb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:10:30.066 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:10:30.063+0000 7fa4e9560700 1 --2- 172.21.15.191:0/855704549 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7fa4d0038800 0x7fa4d003acb0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fa4d4006fd0 tx=0x7fa4d4006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:10:30.068 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:10:30.067+0000 7fa4daffd700 1 -- 172.21.15.191:0/855704549 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fa4e0029950 con 0x7fa4e41057c0 2023-07-14T19:10:30.376 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:10:30.374+0000 7fa4ebfc5700 1 -- 172.21.15.191:0/855704549 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa4e4102500 con 0x7fa4e41057c0 2023-07-14T19:10:30.377 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:10:30.375+0000 7fa4daffd700 1 -- 172.21.15.191:0/855704549 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+736 (secure 0 0 0) 0x7fa4e0026070 con 0x7fa4e41057c0 2023-07-14T19:10:30.377 INFO:teuthology.orchestra.run.smithi191.stdout: 2023-07-14T19:10:30.377 INFO:teuthology.orchestra.run.smithi191.stdout:{"epoch":1,"fsid":"adc52f04-2279-11ee-9b33-001a4aab830c","modified":"2023-07-14T19:08:46.682427Z","created":"2023-07-14T19:08:46.682427Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:3300","nonce":0},{"type":"v1","addr":"172.21.15.40:6789","nonce":0}]},"addr":"172.21.15.40:6789/0","public_addr":"172.21.15.40:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-07-14T19:10:30.379 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:10:30.377+0000 7fa4ebfc5700 1 -- 172.21.15.191:0/855704549 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7fa4d0038800 msgr2=0x7fa4d003acb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:10:30.379 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:10:30.377+0000 7fa4ebfc5700 1 --2- 172.21.15.191:0/855704549 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7fa4d0038800 0x7fa4d003acb0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fa4d4006fd0 tx=0x7fa4d4006e40 comp rx=0 tx=0).stop 2023-07-14T19:10:30.379 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:10:30.377+0000 7fa4ebfc5700 1 -- 172.21.15.191:0/855704549 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa4e41057c0 msgr2=0x7fa4e419bea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:10:30.379 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:10:30.377+0000 7fa4ebfc5700 1 --2- 172.21.15.191:0/855704549 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa4e41057c0 0x7fa4e419bea0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fa4e0004f40 tx=0x7fa4e0005020 comp rx=0 tx=0).stop 2023-07-14T19:10:30.379 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:10:30.377+0000 7fa4ebfc5700 1 -- 172.21.15.191:0/855704549 shutdown_connections 2023-07-14T19:10:30.380 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:10:30.377+0000 7fa4ebfc5700 1 --2- 172.21.15.191:0/855704549 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7fa4d0038800 0x7fa4d003acb0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:10:30.380 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:10:30.377+0000 7fa4ebfc5700 1 --2- 172.21.15.191:0/855704549 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa4e41057c0 0x7fa4e419bea0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:10:30.380 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:10:30.377+0000 7fa4ebfc5700 1 -- 172.21.15.191:0/855704549 >> 172.21.15.191:0/855704549 conn(0x7fa4e40743e0 msgr2=0x7fa4e4106d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:10:30.380 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:10:30.377+0000 7fa4ebfc5700 1 -- 172.21.15.191:0/855704549 shutdown_connections 2023-07-14T19:10:30.380 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:10:30.377+0000 7fa4ebfc5700 1 -- 172.21.15.191:0/855704549 wait complete. 2023-07-14T19:10:30.380 INFO:teuthology.orchestra.run.smithi191.stderr:dumped monmap epoch 1 2023-07-14T19:10:30.398 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:30 smithi191 systemd[1]: Starting Ceph mon.c for adc52f04-2279-11ee-9b33-001a4aab830c... 2023-07-14T19:10:30.721 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:10:30 smithi040 ceph-mon[96518]: from='client.? 172.21.15.191:0/855704549' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-07-14T19:10:30.722 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:30 smithi191 podman[98247]: 2023-07-14T19:10:31.222 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:30 smithi191 ceph-mon[98311]: set uid:gid to 167:167 (ceph:ceph) 2023-07-14T19:10:31.223 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:30 smithi191 ceph-mon[98311]: ceph version 18.0.0-4940-g81c9387d (81c9387d2e7576da065747ec9c16d1c7d69016ec) reef (dev), process ceph-mon, pid 7 2023-07-14T19:10:31.223 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:30 smithi191 ceph-mon[98311]: pidfile_write: ignore empty --pid-file 2023-07-14T19:10:31.223 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:30 smithi191 bash[98247]: a35dab69edf677b718433fffafe80a06a58f31759d20e28fd111d8620975b4c8 2023-07-14T19:10:31.223 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:30 smithi191 ceph-mon[98311]: load: jerasure load: lrc 2023-07-14T19:10:31.223 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:30 smithi191 ceph-mon[98311]: rocksdb: RocksDB version: 7.9.2 2023-07-14T19:10:31.223 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:30 smithi191 ceph-mon[98311]: rocksdb: Git sha 0 2023-07-14T19:10:31.224 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:30 smithi191 ceph-mon[98311]: rocksdb: Compile date 2023-07-14 14:45:42 2023-07-14T19:10:31.224 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:30 smithi191 ceph-mon[98311]: rocksdb: DB SUMMARY 2023-07-14T19:10:31.224 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:30 smithi191 ceph-mon[98311]: rocksdb: DB Session ID: S0M6V714I3O5UA97VU2H 2023-07-14T19:10:31.224 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:30 smithi191 ceph-mon[98311]: rocksdb: CURRENT file: CURRENT 2023-07-14T19:10:31.224 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:30 smithi191 ceph-mon[98311]: rocksdb: IDENTITY file: IDENTITY 2023-07-14T19:10:31.224 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:30 smithi191 ceph-mon[98311]: rocksdb: MANIFEST file: MANIFEST-000005 size: 59 Bytes 2023-07-14T19:10:31.224 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:30 smithi191 ceph-mon[98311]: rocksdb: SST files in /var/lib/ceph/mon/ceph-c/store.db dir, Total Num: 0, files: 2023-07-14T19:10:31.225 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:30 smithi191 ceph-mon[98311]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-c/store.db: 000004.log size: 511 ; 2023-07-14T19:10:31.225 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:30 smithi191 ceph-mon[98311]: rocksdb: Options.error_if_exists: 0 2023-07-14T19:10:31.225 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:30 smithi191 ceph-mon[98311]: rocksdb: Options.create_if_missing: 0 2023-07-14T19:10:31.225 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:30 smithi191 ceph-mon[98311]: rocksdb: Options.paranoid_checks: 1 2023-07-14T19:10:31.225 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:30 smithi191 ceph-mon[98311]: rocksdb: Options.flush_verify_memtable_count: 1 2023-07-14T19:10:31.225 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:30 smithi191 ceph-mon[98311]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2023-07-14T19:10:31.225 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:30 smithi191 ceph-mon[98311]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2023-07-14T19:10:31.226 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:30 smithi191 ceph-mon[98311]: rocksdb: Options.env: 0x56164f228c60 2023-07-14T19:10:31.226 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:30 smithi191 ceph-mon[98311]: rocksdb: Options.fs: PosixFileSystem 2023-07-14T19:10:31.226 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:30 smithi191 ceph-mon[98311]: rocksdb: Options.info_log: 0x5616505152a0 2023-07-14T19:10:31.226 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:30 smithi191 ceph-mon[98311]: rocksdb: Options.max_file_opening_threads: 16 2023-07-14T19:10:31.226 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:30 smithi191 ceph-mon[98311]: rocksdb: Options.statistics: (nil) 2023-07-14T19:10:31.226 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:30 smithi191 ceph-mon[98311]: rocksdb: Options.use_fsync: 0 2023-07-14T19:10:31.226 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:30 smithi191 ceph-mon[98311]: rocksdb: Options.max_log_file_size: 0 2023-07-14T19:10:31.226 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:30 smithi191 ceph-mon[98311]: rocksdb: Options.max_manifest_file_size: 1073741824 2023-07-14T19:10:31.227 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:30 smithi191 ceph-mon[98311]: rocksdb: Options.log_file_time_to_roll: 0 2023-07-14T19:10:31.227 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:30 smithi191 ceph-mon[98311]: rocksdb: Options.keep_log_file_num: 1000 2023-07-14T19:10:31.227 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:30 smithi191 ceph-mon[98311]: rocksdb: Options.recycle_log_file_num: 0 2023-07-14T19:10:31.227 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:30 smithi191 ceph-mon[98311]: rocksdb: Options.allow_fallocate: 1 2023-07-14T19:10:31.227 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:30 smithi191 ceph-mon[98311]: rocksdb: Options.allow_mmap_reads: 0 2023-07-14T19:10:31.227 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:30 smithi191 ceph-mon[98311]: rocksdb: Options.allow_mmap_writes: 0 2023-07-14T19:10:31.227 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:30 smithi191 ceph-mon[98311]: rocksdb: Options.use_direct_reads: 0 2023-07-14T19:10:31.228 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:30 smithi191 ceph-mon[98311]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2023-07-14T19:10:31.228 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:30 smithi191 ceph-mon[98311]: rocksdb: Options.create_missing_column_families: 0 2023-07-14T19:10:31.228 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:30 smithi191 ceph-mon[98311]: rocksdb: Options.db_log_dir: 2023-07-14T19:10:31.228 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:30 smithi191 ceph-mon[98311]: rocksdb: Options.wal_dir: 2023-07-14T19:10:31.228 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:30 smithi191 ceph-mon[98311]: rocksdb: Options.table_cache_numshardbits: 6 2023-07-14T19:10:31.228 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:30 smithi191 ceph-mon[98311]: rocksdb: Options.WAL_ttl_seconds: 0 2023-07-14T19:10:31.228 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:30 smithi191 ceph-mon[98311]: rocksdb: Options.WAL_size_limit_MB: 0 2023-07-14T19:10:31.228 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:30 smithi191 ceph-mon[98311]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2023-07-14T19:10:31.229 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:30 smithi191 ceph-mon[98311]: rocksdb: Options.manifest_preallocation_size: 4194304 2023-07-14T19:10:31.229 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:30 smithi191 ceph-mon[98311]: rocksdb: Options.is_fd_close_on_exec: 1 2023-07-14T19:10:31.229 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:30 smithi191 ceph-mon[98311]: rocksdb: Options.advise_random_on_open: 1 2023-07-14T19:10:31.229 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:30 smithi191 ceph-mon[98311]: rocksdb: Options.db_write_buffer_size: 0 2023-07-14T19:10:31.229 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:30 smithi191 ceph-mon[98311]: rocksdb: Options.write_buffer_manager: 0x5616504363c0 2023-07-14T19:10:31.229 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:30 smithi191 ceph-mon[98311]: rocksdb: Options.access_hint_on_compaction_start: 1 2023-07-14T19:10:31.229 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:30 smithi191 ceph-mon[98311]: rocksdb: Options.random_access_max_buffer_size: 1048576 2023-07-14T19:10:31.229 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:30 smithi191 ceph-mon[98311]: rocksdb: Options.use_adaptive_mutex: 0 2023-07-14T19:10:31.230 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:30 smithi191 ceph-mon[98311]: rocksdb: Options.rate_limiter: (nil) 2023-07-14T19:10:31.230 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:30 smithi191 ceph-mon[98311]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2023-07-14T19:10:31.230 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:30 smithi191 ceph-mon[98311]: rocksdb: Options.wal_recovery_mode: 2 2023-07-14T19:10:31.230 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:30 smithi191 ceph-mon[98311]: rocksdb: Options.enable_thread_tracking: 0 2023-07-14T19:10:31.230 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:30 smithi191 ceph-mon[98311]: rocksdb: Options.enable_pipelined_write: 0 2023-07-14T19:10:31.230 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:30 smithi191 ceph-mon[98311]: rocksdb: Options.unordered_write: 0 2023-07-14T19:10:31.231 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:30 smithi191 ceph-mon[98311]: rocksdb: Options.allow_concurrent_memtable_write: 1 2023-07-14T19:10:31.231 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:30 smithi191 ceph-mon[98311]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2023-07-14T19:10:31.231 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:30 smithi191 ceph-mon[98311]: rocksdb: Options.write_thread_max_yield_usec: 100 2023-07-14T19:10:31.231 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:30 smithi191 ceph-mon[98311]: rocksdb: Options.write_thread_slow_yield_usec: 3 2023-07-14T19:10:31.231 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:30 smithi191 ceph-mon[98311]: rocksdb: Options.row_cache: None 2023-07-14T19:10:31.231 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:30 smithi191 ceph-mon[98311]: rocksdb: Options.wal_filter: None 2023-07-14T19:10:31.231 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:30 smithi191 ceph-mon[98311]: rocksdb: Options.avoid_flush_during_recovery: 0 2023-07-14T19:10:31.232 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:30 smithi191 ceph-mon[98311]: rocksdb: Options.allow_ingest_behind: 0 2023-07-14T19:10:31.232 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:30 smithi191 ceph-mon[98311]: rocksdb: Options.two_write_queues: 0 2023-07-14T19:10:31.232 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:30 smithi191 ceph-mon[98311]: rocksdb: Options.manual_wal_flush: 0 2023-07-14T19:10:31.232 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:30 smithi191 ceph-mon[98311]: rocksdb: Options.wal_compression: 0 2023-07-14T19:10:31.232 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:30 smithi191 ceph-mon[98311]: rocksdb: Options.atomic_flush: 0 2023-07-14T19:10:31.232 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:30 smithi191 ceph-mon[98311]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2023-07-14T19:10:31.232 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:30 smithi191 ceph-mon[98311]: rocksdb: Options.persist_stats_to_disk: 0 2023-07-14T19:10:31.233 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:30 smithi191 ceph-mon[98311]: rocksdb: Options.write_dbid_to_manifest: 0 2023-07-14T19:10:31.233 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:30 smithi191 ceph-mon[98311]: rocksdb: Options.log_readahead_size: 0 2023-07-14T19:10:31.233 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:30 smithi191 ceph-mon[98311]: rocksdb: Options.file_checksum_gen_factory: Unknown 2023-07-14T19:10:31.233 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:30 smithi191 ceph-mon[98311]: rocksdb: Options.best_efforts_recovery: 0 2023-07-14T19:10:31.233 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:30 smithi191 ceph-mon[98311]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2023-07-14T19:10:31.233 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:30 smithi191 ceph-mon[98311]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2023-07-14T19:10:31.233 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:30 smithi191 ceph-mon[98311]: rocksdb: Options.allow_data_in_errors: 0 2023-07-14T19:10:31.233 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:30 smithi191 ceph-mon[98311]: rocksdb: Options.db_host_id: __hostname__ 2023-07-14T19:10:31.235 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:30 smithi191 ceph-mon[98311]: rocksdb: Options.enforce_single_del_contracts: true 2023-07-14T19:10:31.235 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:30 smithi191 ceph-mon[98311]: rocksdb: Options.max_background_jobs: 2 2023-07-14T19:10:31.235 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:30 smithi191 ceph-mon[98311]: rocksdb: Options.max_background_compactions: -1 2023-07-14T19:10:31.235 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:30 smithi191 ceph-mon[98311]: rocksdb: Options.max_subcompactions: 1 2023-07-14T19:10:31.235 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:30 smithi191 ceph-mon[98311]: rocksdb: Options.avoid_flush_during_shutdown: 0 2023-07-14T19:10:31.236 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:30 smithi191 ceph-mon[98311]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2023-07-14T19:10:31.236 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:30 smithi191 ceph-mon[98311]: rocksdb: Options.delayed_write_rate : 16777216 2023-07-14T19:10:31.236 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:30 smithi191 ceph-mon[98311]: rocksdb: Options.max_total_wal_size: 0 2023-07-14T19:10:31.236 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:30 smithi191 ceph-mon[98311]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2023-07-14T19:10:31.236 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:30 smithi191 ceph-mon[98311]: rocksdb: Options.stats_dump_period_sec: 600 2023-07-14T19:10:31.236 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:30 smithi191 ceph-mon[98311]: rocksdb: Options.stats_persist_period_sec: 600 2023-07-14T19:10:31.236 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:30 smithi191 ceph-mon[98311]: rocksdb: Options.stats_history_buffer_size: 1048576 2023-07-14T19:10:31.236 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:30 smithi191 ceph-mon[98311]: rocksdb: Options.max_open_files: -1 2023-07-14T19:10:31.237 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:30 smithi191 ceph-mon[98311]: rocksdb: Options.bytes_per_sync: 0 2023-07-14T19:10:31.237 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:30 smithi191 ceph-mon[98311]: rocksdb: Options.wal_bytes_per_sync: 0 2023-07-14T19:10:31.237 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:30 smithi191 ceph-mon[98311]: rocksdb: Options.strict_bytes_per_sync: 0 2023-07-14T19:10:31.237 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:30 smithi191 ceph-mon[98311]: rocksdb: Options.compaction_readahead_size: 0 2023-07-14T19:10:31.237 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:30 smithi191 ceph-mon[98311]: rocksdb: Options.max_background_flushes: -1 2023-07-14T19:10:31.237 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:30 smithi191 ceph-mon[98311]: rocksdb: Compression algorithms supported: 2023-07-14T19:10:31.237 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:30 smithi191 ceph-mon[98311]: rocksdb: kZSTDNotFinalCompression supported: 0 2023-07-14T19:10:31.238 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:30 smithi191 ceph-mon[98311]: rocksdb: kZSTD supported: 0 2023-07-14T19:10:31.238 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:30 smithi191 ceph-mon[98311]: rocksdb: kXpressCompression supported: 0 2023-07-14T19:10:31.238 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:30 smithi191 ceph-mon[98311]: rocksdb: kLZ4HCCompression supported: 1 2023-07-14T19:10:31.238 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:30 smithi191 ceph-mon[98311]: rocksdb: kZlibCompression supported: 1 2023-07-14T19:10:31.238 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:30 smithi191 ceph-mon[98311]: rocksdb: kSnappyCompression supported: 1 2023-07-14T19:10:31.238 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:30 smithi191 ceph-mon[98311]: rocksdb: kLZ4Compression supported: 1 2023-07-14T19:10:31.238 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:30 smithi191 ceph-mon[98311]: rocksdb: kBZip2Compression supported: 0 2023-07-14T19:10:31.238 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:30 smithi191 ceph-mon[98311]: rocksdb: Fast CRC32 supported: Supported on x86 2023-07-14T19:10:31.239 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:30 smithi191 ceph-mon[98311]: rocksdb: DMutex implementation: pthread_mutex_t 2023-07-14T19:10:31.239 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:30 smithi191 ceph-mon[98311]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000005 2023-07-14T19:10:31.239 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:30 smithi191 ceph-mon[98311]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2023-07-14T19:10:31.239 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:30 smithi191 ceph-mon[98311]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2023-07-14T19:10:31.239 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:30 smithi191 ceph-mon[98311]: rocksdb: Options.merge_operator: 2023-07-14T19:10:31.239 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:30 smithi191 ceph-mon[98311]: rocksdb: Options.compaction_filter: None 2023-07-14T19:10:31.239 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:30 smithi191 ceph-mon[98311]: rocksdb: Options.compaction_filter_factory: None 2023-07-14T19:10:31.239 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:30 smithi191 ceph-mon[98311]: rocksdb: Options.sst_partitioner_factory: None 2023-07-14T19:10:31.240 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:30 smithi191 ceph-mon[98311]: rocksdb: Options.memtable_factory: SkipListFactory 2023-07-14T19:10:31.240 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:30 smithi191 ceph-mon[98311]: rocksdb: Options.table_factory: BlockBasedTable 2023-07-14T19:10:31.240 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:30 smithi191 ceph-mon[98311]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x5616505153a0) 2023-07-14T19:10:31.240 INFO:journalctl@ceph.mon.c.smithi191.stdout: cache_index_and_filter_blocks: 1 2023-07-14T19:10:31.240 INFO:journalctl@ceph.mon.c.smithi191.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2023-07-14T19:10:31.240 INFO:journalctl@ceph.mon.c.smithi191.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2023-07-14T19:10:31.240 INFO:journalctl@ceph.mon.c.smithi191.stdout: pin_top_level_index_and_filter: 1 2023-07-14T19:10:31.241 INFO:journalctl@ceph.mon.c.smithi191.stdout: index_type: 0 2023-07-14T19:10:31.241 INFO:journalctl@ceph.mon.c.smithi191.stdout: data_block_index_type: 0 2023-07-14T19:10:31.241 INFO:journalctl@ceph.mon.c.smithi191.stdout: index_shortening: 1 2023-07-14T19:10:31.241 INFO:journalctl@ceph.mon.c.smithi191.stdout: data_block_hash_table_util_ratio: 0.750000 2023-07-14T19:10:31.241 INFO:journalctl@ceph.mon.c.smithi191.stdout: checksum: 4 2023-07-14T19:10:31.241 INFO:journalctl@ceph.mon.c.smithi191.stdout: no_block_cache: 0 2023-07-14T19:10:31.241 INFO:journalctl@ceph.mon.c.smithi191.stdout: block_cache: 0x5616504d51f0 2023-07-14T19:10:31.241 INFO:journalctl@ceph.mon.c.smithi191.stdout: block_cache_name: BinnedLRUCache 2023-07-14T19:10:31.242 INFO:journalctl@ceph.mon.c.smithi191.stdout: block_cache_options: 2023-07-14T19:10:31.242 INFO:journalctl@ceph.mon.c.smithi191.stdout: capacity : 536870912 2023-07-14T19:10:31.242 INFO:journalctl@ceph.mon.c.smithi191.stdout: num_shard_bits : 4 2023-07-14T19:10:31.242 INFO:journalctl@ceph.mon.c.smithi191.stdout: strict_capacity_limit : 0 2023-07-14T19:10:31.242 INFO:journalctl@ceph.mon.c.smithi191.stdout: high_pri_pool_ratio: 0.000 2023-07-14T19:10:31.242 INFO:journalctl@ceph.mon.c.smithi191.stdout: block_cache_compressed: (nil) 2023-07-14T19:10:31.242 INFO:journalctl@ceph.mon.c.smithi191.stdout: persistent_cache: (nil) 2023-07-14T19:10:31.242 INFO:journalctl@ceph.mon.c.smithi191.stdout: block_size: 4096 2023-07-14T19:10:31.243 INFO:journalctl@ceph.mon.c.smithi191.stdout: block_size_deviation: 10 2023-07-14T19:10:31.243 INFO:journalctl@ceph.mon.c.smithi191.stdout: block_restart_interval: 16 2023-07-14T19:10:31.243 INFO:journalctl@ceph.mon.c.smithi191.stdout: index_block_restart_interval: 1 2023-07-14T19:10:31.243 INFO:journalctl@ceph.mon.c.smithi191.stdout: metadata_block_size: 4096 2023-07-14T19:10:31.243 INFO:journalctl@ceph.mon.c.smithi191.stdout: partition_filters: 0 2023-07-14T19:10:31.243 INFO:journalctl@ceph.mon.c.smithi191.stdout: use_delta_encoding: 1 2023-07-14T19:10:31.243 INFO:journalctl@ceph.mon.c.smithi191.stdout: filter_policy: bloomfilter 2023-07-14T19:10:31.243 INFO:journalctl@ceph.mon.c.smithi191.stdout: whole_key_filtering: 1 2023-07-14T19:10:31.244 INFO:journalctl@ceph.mon.c.smithi191.stdout: verify_compression: 0 2023-07-14T19:10:31.244 INFO:journalctl@ceph.mon.c.smithi191.stdout: read_amp_bytes_per_bit: 0 2023-07-14T19:10:31.244 INFO:journalctl@ceph.mon.c.smithi191.stdout: format_version: 5 2023-07-14T19:10:31.244 INFO:journalctl@ceph.mon.c.smithi191.stdout: enable_index_compression: 1 2023-07-14T19:10:31.244 INFO:journalctl@ceph.mon.c.smithi191.stdout: block_align: 0 2023-07-14T19:10:31.244 INFO:journalctl@ceph.mon.c.smithi191.stdout: max_auto_readahead_size: 262144 2023-07-14T19:10:31.244 INFO:journalctl@ceph.mon.c.smithi191.stdout: prepopulate_block_cache: 0 2023-07-14T19:10:31.244 INFO:journalctl@ceph.mon.c.smithi191.stdout: initial_auto_readahead_size: 8192 2023-07-14T19:10:31.245 INFO:journalctl@ceph.mon.c.smithi191.stdout: num_file_reads_for_auto_readahead: 2 2023-07-14T19:10:31.245 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:30 smithi191 ceph-mon[98311]: rocksdb: Options.write_buffer_size: 33554432 2023-07-14T19:10:31.245 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:30 smithi191 ceph-mon[98311]: rocksdb: Options.max_write_buffer_number: 2 2023-07-14T19:10:31.245 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:30 smithi191 ceph-mon[98311]: rocksdb: Options.compression: NoCompression 2023-07-14T19:10:31.245 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:30 smithi191 ceph-mon[98311]: rocksdb: Options.bottommost_compression: Disabled 2023-07-14T19:10:31.245 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:30 smithi191 ceph-mon[98311]: rocksdb: Options.prefix_extractor: nullptr 2023-07-14T19:10:31.245 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:30 smithi191 ceph-mon[98311]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2023-07-14T19:10:31.245 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:30 smithi191 ceph-mon[98311]: rocksdb: Options.num_levels: 7 2023-07-14T19:10:31.246 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:30 smithi191 ceph-mon[98311]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2023-07-14T19:10:31.246 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:30 smithi191 ceph-mon[98311]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2023-07-14T19:10:31.246 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:30 smithi191 ceph-mon[98311]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2023-07-14T19:10:31.246 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:30 smithi191 ceph-mon[98311]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2023-07-14T19:10:31.246 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:30 smithi191 ceph-mon[98311]: rocksdb: Options.bottommost_compression_opts.level: 32767 2023-07-14T19:10:31.246 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:30 smithi191 ceph-mon[98311]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2023-07-14T19:10:31.246 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:30 smithi191 ceph-mon[98311]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2023-07-14T19:10:31.247 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:30 smithi191 ceph-mon[98311]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2023-07-14T19:10:31.247 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:30 smithi191 ceph-mon[98311]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2023-07-14T19:10:31.247 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:30 smithi191 ceph-mon[98311]: rocksdb: Options.bottommost_compression_opts.enabled: false 2023-07-14T19:10:31.247 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:30 smithi191 ceph-mon[98311]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2023-07-14T19:10:31.247 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:30 smithi191 ceph-mon[98311]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2023-07-14T19:10:31.247 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:30 smithi191 ceph-mon[98311]: rocksdb: Options.compression_opts.window_bits: -14 2023-07-14T19:10:31.247 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:30 smithi191 ceph-mon[98311]: rocksdb: Options.compression_opts.level: 32767 2023-07-14T19:10:31.247 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:30 smithi191 ceph-mon[98311]: rocksdb: Options.compression_opts.strategy: 0 2023-07-14T19:10:31.248 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:30 smithi191 ceph-mon[98311]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2023-07-14T19:10:31.248 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:30 smithi191 ceph-mon[98311]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2023-07-14T19:10:31.248 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:30 smithi191 ceph-mon[98311]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2023-07-14T19:10:31.248 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:30 smithi191 ceph-mon[98311]: rocksdb: Options.compression_opts.parallel_threads: 1 2023-07-14T19:10:31.248 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:30 smithi191 ceph-mon[98311]: rocksdb: Options.compression_opts.enabled: false 2023-07-14T19:10:31.248 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:30 smithi191 ceph-mon[98311]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2023-07-14T19:10:31.248 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:30 smithi191 ceph-mon[98311]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2023-07-14T19:10:31.248 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:30 smithi191 ceph-mon[98311]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2023-07-14T19:10:31.249 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:30 smithi191 ceph-mon[98311]: rocksdb: Options.level0_stop_writes_trigger: 36 2023-07-14T19:10:31.249 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:30 smithi191 ceph-mon[98311]: rocksdb: Options.target_file_size_base: 67108864 2023-07-14T19:10:31.249 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:30 smithi191 ceph-mon[98311]: rocksdb: Options.target_file_size_multiplier: 1 2023-07-14T19:10:31.249 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:30 smithi191 ceph-mon[98311]: rocksdb: Options.max_bytes_for_level_base: 268435456 2023-07-14T19:10:31.249 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:30 smithi191 ceph-mon[98311]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2023-07-14T19:10:31.249 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:30 smithi191 ceph-mon[98311]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2023-07-14T19:10:31.249 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:30 smithi191 ceph-mon[98311]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2023-07-14T19:10:31.250 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:30 smithi191 ceph-mon[98311]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2023-07-14T19:10:31.250 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:30 smithi191 ceph-mon[98311]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2023-07-14T19:10:31.250 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:30 smithi191 ceph-mon[98311]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2023-07-14T19:10:31.250 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:30 smithi191 ceph-mon[98311]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2023-07-14T19:10:31.250 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:30 smithi191 ceph-mon[98311]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2023-07-14T19:10:31.250 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:30 smithi191 ceph-mon[98311]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2023-07-14T19:10:31.250 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:30 smithi191 ceph-mon[98311]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2023-07-14T19:10:31.250 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:30 smithi191 ceph-mon[98311]: rocksdb: Options.max_compaction_bytes: 1677721600 2023-07-14T19:10:31.251 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:30 smithi191 ceph-mon[98311]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2023-07-14T19:10:31.251 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:30 smithi191 ceph-mon[98311]: rocksdb: Options.arena_block_size: 1048576 2023-07-14T19:10:31.251 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:30 smithi191 ceph-mon[98311]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2023-07-14T19:10:31.251 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:30 smithi191 ceph-mon[98311]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2023-07-14T19:10:31.251 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:30 smithi191 ceph-mon[98311]: rocksdb: Options.disable_auto_compactions: 0 2023-07-14T19:10:31.251 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:30 smithi191 ceph-mon[98311]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2023-07-14T19:10:31.251 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:30 smithi191 ceph-mon[98311]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2023-07-14T19:10:31.251 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:30 smithi191 ceph-mon[98311]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2023-07-14T19:10:31.252 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:30 smithi191 ceph-mon[98311]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2023-07-14T19:10:31.252 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:30 smithi191 ceph-mon[98311]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2023-07-14T19:10:31.252 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:30 smithi191 ceph-mon[98311]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2023-07-14T19:10:31.252 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:30 smithi191 ceph-mon[98311]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2023-07-14T19:10:31.252 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:30 smithi191 ceph-mon[98311]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2023-07-14T19:10:31.252 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:30 smithi191 ceph-mon[98311]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2023-07-14T19:10:31.252 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:30 smithi191 ceph-mon[98311]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2023-07-14T19:10:31.253 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:30 smithi191 ceph-mon[98311]: rocksdb: Options.table_properties_collectors: 2023-07-14T19:10:31.253 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:30 smithi191 ceph-mon[98311]: rocksdb: Options.inplace_update_support: 0 2023-07-14T19:10:31.253 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:30 smithi191 ceph-mon[98311]: rocksdb: Options.inplace_update_num_locks: 10000 2023-07-14T19:10:31.253 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:30 smithi191 ceph-mon[98311]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2023-07-14T19:10:31.253 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:30 smithi191 ceph-mon[98311]: rocksdb: Options.memtable_whole_key_filtering: 0 2023-07-14T19:10:31.253 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:30 smithi191 ceph-mon[98311]: rocksdb: Options.memtable_huge_page_size: 0 2023-07-14T19:10:31.253 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:30 smithi191 ceph-mon[98311]: rocksdb: Options.bloom_locality: 0 2023-07-14T19:10:31.253 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:30 smithi191 ceph-mon[98311]: rocksdb: Options.max_successive_merges: 0 2023-07-14T19:10:31.254 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:30 smithi191 ceph-mon[98311]: rocksdb: Options.optimize_filters_for_hits: 0 2023-07-14T19:10:31.254 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:30 smithi191 ceph-mon[98311]: rocksdb: Options.paranoid_file_checks: 0 2023-07-14T19:10:31.254 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:30 smithi191 ceph-mon[98311]: rocksdb: Options.force_consistency_checks: 1 2023-07-14T19:10:31.254 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:30 smithi191 ceph-mon[98311]: rocksdb: Options.report_bg_io_stats: 0 2023-07-14T19:10:31.254 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:30 smithi191 ceph-mon[98311]: rocksdb: Options.ttl: 2592000 2023-07-14T19:10:31.254 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:30 smithi191 ceph-mon[98311]: rocksdb: Options.periodic_compaction_seconds: 0 2023-07-14T19:10:31.254 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:30 smithi191 ceph-mon[98311]: rocksdb: Options.preclude_last_level_data_seconds: 0 2023-07-14T19:10:31.254 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:30 smithi191 ceph-mon[98311]: rocksdb: Options.preserve_internal_time_seconds: 0 2023-07-14T19:10:31.255 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:30 smithi191 ceph-mon[98311]: rocksdb: Options.enable_blob_files: false 2023-07-14T19:10:31.255 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:30 smithi191 ceph-mon[98311]: rocksdb: Options.min_blob_size: 0 2023-07-14T19:10:31.255 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:30 smithi191 ceph-mon[98311]: rocksdb: Options.blob_file_size: 268435456 2023-07-14T19:10:31.255 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:30 smithi191 ceph-mon[98311]: rocksdb: Options.blob_compression_type: NoCompression 2023-07-14T19:10:31.255 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:30 smithi191 ceph-mon[98311]: rocksdb: Options.enable_blob_garbage_collection: false 2023-07-14T19:10:31.255 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:30 smithi191 ceph-mon[98311]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2023-07-14T19:10:31.255 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:30 smithi191 ceph-mon[98311]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2023-07-14T19:10:31.255 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:30 smithi191 ceph-mon[98311]: rocksdb: Options.blob_compaction_readahead_size: 0 2023-07-14T19:10:31.256 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:30 smithi191 ceph-mon[98311]: rocksdb: Options.blob_file_starting_level: 0 2023-07-14T19:10:31.256 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:30 smithi191 ceph-mon[98311]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2023-07-14T19:10:31.256 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:30 smithi191 ceph-mon[98311]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2023-07-14T19:10:31.256 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:30 smithi191 ceph-mon[98311]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 0 2023-07-14T19:10:31.256 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:30 smithi191 ceph-mon[98311]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 56292536-74d9-49c0-a3e7-63ab21e5590b 2023-07-14T19:10:31.256 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:30 smithi191 ceph-mon[98311]: rocksdb: EVENT_LOG_v1 {"time_micros": 1689361830864283, "job": 1, "event": "recovery_started", "wal_files": [4]} 2023-07-14T19:10:31.256 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:30 smithi191 ceph-mon[98311]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #4 mode 2 2023-07-14T19:10:31.257 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:30 smithi191 ceph-mon[98311]: rocksdb: EVENT_LOG_v1 {"time_micros": 1689361830864704, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1622, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1, "largest_seqno": 5, "table_properties": {"data_size": 523, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 401, "raw_average_value_size": 80, "num_data_blocks": 1, "num_entries": 5, "num_filter_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1689361830, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "56292536-74d9-49c0-a3e7-63ab21e5590b", "db_session_id": "S0M6V714I3O5UA97VU2H", "orig_file_number": 8, "seqno_to_time_mapping": "N/A"}} 2023-07-14T19:10:31.257 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:30 smithi191 ceph-mon[98311]: rocksdb: EVENT_LOG_v1 {"time_micros": 1689361830864776, "job": 1, "event": "recovery_finished"} 2023-07-14T19:10:31.257 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:30 smithi191 ceph-mon[98311]: rocksdb: [db/version_set.cc:5047] Creating manifest 10 2023-07-14T19:10:31.257 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:30 smithi191 ceph-mon[98311]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-c/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2023-07-14T19:10:31.257 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:30 smithi191 ceph-mon[98311]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x56165056e000 2023-07-14T19:10:31.257 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:30 smithi191 ceph-mon[98311]: rocksdb: DB pointer 0x56165055a000 2023-07-14T19:10:31.257 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:30 smithi191 ceph-mon[98311]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2023-07-14T19:10:31.258 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:30 smithi191 ceph-mon[98311]: rocksdb: [db/db_impl/db_impl.cc:1111] 2023-07-14T19:10:31.258 INFO:journalctl@ceph.mon.c.smithi191.stdout: ** DB Stats ** 2023-07-14T19:10:31.258 INFO:journalctl@ceph.mon.c.smithi191.stdout: Uptime(secs): 0.0 total, 0.0 interval 2023-07-14T19:10:31.258 INFO:journalctl@ceph.mon.c.smithi191.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2023-07-14T19:10:31.258 INFO:journalctl@ceph.mon.c.smithi191.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2023-07-14T19:10:31.258 INFO:journalctl@ceph.mon.c.smithi191.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2023-07-14T19:10:31.258 INFO:journalctl@ceph.mon.c.smithi191.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2023-07-14T19:10:31.258 INFO:journalctl@ceph.mon.c.smithi191.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2023-07-14T19:10:31.259 INFO:journalctl@ceph.mon.c.smithi191.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2023-07-14T19:10:31.259 INFO:journalctl@ceph.mon.c.smithi191.stdout: 2023-07-14T19:10:31.259 INFO:journalctl@ceph.mon.c.smithi191.stdout: ** Compaction Stats [default] ** 2023-07-14T19:10:31.259 INFO:journalctl@ceph.mon.c.smithi191.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2023-07-14T19:10:31.259 INFO:journalctl@ceph.mon.c.smithi191.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2023-07-14T19:10:31.259 INFO:journalctl@ceph.mon.c.smithi191.stdout: L0 1/0 1.58 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.8 0.00 0.00 1 0.000 0 0 0.0 0.0 2023-07-14T19:10:31.259 INFO:journalctl@ceph.mon.c.smithi191.stdout: Sum 1/0 1.58 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.8 0.00 0.00 1 0.000 0 0 0.0 0.0 2023-07-14T19:10:31.259 INFO:journalctl@ceph.mon.c.smithi191.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.8 0.00 0.00 1 0.000 0 0 0.0 0.0 2023-07-14T19:10:31.260 INFO:journalctl@ceph.mon.c.smithi191.stdout: 2023-07-14T19:10:31.260 INFO:journalctl@ceph.mon.c.smithi191.stdout: ** Compaction Stats [default] ** 2023-07-14T19:10:31.260 INFO:journalctl@ceph.mon.c.smithi191.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2023-07-14T19:10:31.260 INFO:journalctl@ceph.mon.c.smithi191.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2023-07-14T19:10:31.260 INFO:journalctl@ceph.mon.c.smithi191.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 3.8 0.00 0.00 1 0.000 0 0 0.0 0.0 2023-07-14T19:10:31.260 INFO:journalctl@ceph.mon.c.smithi191.stdout: 2023-07-14T19:10:31.260 INFO:journalctl@ceph.mon.c.smithi191.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2023-07-14T19:10:31.261 INFO:journalctl@ceph.mon.c.smithi191.stdout: 2023-07-14T19:10:31.261 INFO:journalctl@ceph.mon.c.smithi191.stdout: Uptime(secs): 0.0 total, 0.0 interval 2023-07-14T19:10:31.261 INFO:journalctl@ceph.mon.c.smithi191.stdout: Flush(GB): cumulative 0.000, interval 0.000 2023-07-14T19:10:31.261 INFO:journalctl@ceph.mon.c.smithi191.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2023-07-14T19:10:31.261 INFO:journalctl@ceph.mon.c.smithi191.stdout: AddFile(Total Files): cumulative 0, interval 0 2023-07-14T19:10:31.261 INFO:journalctl@ceph.mon.c.smithi191.stdout: AddFile(L0 Files): cumulative 0, interval 0 2023-07-14T19:10:31.261 INFO:journalctl@ceph.mon.c.smithi191.stdout: AddFile(Keys): cumulative 0, interval 0 2023-07-14T19:10:31.261 INFO:journalctl@ceph.mon.c.smithi191.stdout: Cumulative compaction: 0.00 GB write, 0.65 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-07-14T19:10:31.262 INFO:journalctl@ceph.mon.c.smithi191.stdout: Interval compaction: 0.00 GB write, 0.65 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-07-14T19:10:31.262 INFO:journalctl@ceph.mon.c.smithi191.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2023-07-14T19:10:31.262 INFO:journalctl@ceph.mon.c.smithi191.stdout: Block cache BinnedLRUCache@0x5616504d51f0#7 capacity: 512.00 MB usage: 0.22 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1e-05 secs_since: 0 2023-07-14T19:10:31.262 INFO:journalctl@ceph.mon.c.smithi191.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.11 KB,2.08616e-05%) IndexBlock(1,0.11 KB,2.08616e-05%) Misc(1,0.00 KB,0%) 2023-07-14T19:10:31.262 INFO:journalctl@ceph.mon.c.smithi191.stdout: 2023-07-14T19:10:31.262 INFO:journalctl@ceph.mon.c.smithi191.stdout: ** File Read Latency Histogram By Level [default] ** 2023-07-14T19:10:31.262 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:30 smithi191 ceph-mon[98311]: mon.c does not exist in monmap, will attempt to join an existing cluster 2023-07-14T19:10:31.262 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:30 smithi191 ceph-mon[98311]: using public_addr v2:172.21.15.191:0/0 -> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] 2023-07-14T19:10:31.263 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:30 smithi191 ceph-mon[98311]: starting mon.c rank -1 at public addrs [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] at bind addrs [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon_data /var/lib/ceph/mon/ceph-c fsid adc52f04-2279-11ee-9b33-001a4aab830c 2023-07-14T19:10:31.263 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:30 smithi191 ceph-mon[98311]: mon.c@-1(???) e0 preinit fsid adc52f04-2279-11ee-9b33-001a4aab830c 2023-07-14T19:10:31.263 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:30 smithi191 ceph-mon[98311]: mon.c@-1(synchronizing).mds e1 new map 2023-07-14T19:10:31.263 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:30 smithi191 ceph-mon[98311]: mon.c@-1(synchronizing).mds e1 print_map 2023-07-14T19:10:31.263 INFO:journalctl@ceph.mon.c.smithi191.stdout: e1 2023-07-14T19:10:31.263 INFO:journalctl@ceph.mon.c.smithi191.stdout: enable_multiple, ever_enabled_multiple: 1,1 2023-07-14T19:10:31.263 INFO:journalctl@ceph.mon.c.smithi191.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2023-07-14T19:10:31.264 INFO:journalctl@ceph.mon.c.smithi191.stdout: legacy client fscid: -1 2023-07-14T19:10:31.264 INFO:journalctl@ceph.mon.c.smithi191.stdout: 2023-07-14T19:10:31.264 INFO:journalctl@ceph.mon.c.smithi191.stdout: No filesystems configured 2023-07-14T19:10:31.264 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:30 smithi191 ceph-mon[98311]: mon.c@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2023-07-14T19:10:31.264 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:30 smithi191 ceph-mon[98311]: mon.c@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2023-07-14T19:10:31.264 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:30 smithi191 ceph-mon[98311]: mon.c@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2023-07-14T19:10:31.264 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:30 smithi191 ceph-mon[98311]: mon.c@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2023-07-14T19:10:31.264 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:30 smithi191 ceph-mon[98311]: mon.c@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2023-07-14T19:10:31.265 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:30 smithi191 ceph-mon[98311]: mon.c@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2023-07-14T19:10:31.265 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:30 smithi191 ceph-mon[98311]: mon.c@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2023-07-14T19:10:31.265 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:30 smithi191 ceph-mon[98311]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2023-07-14T19:10:31.265 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:30 smithi191 ceph-mon[98311]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2023-07-14T19:10:31.265 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:30 smithi191 ceph-mon[98311]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2023-07-14T19:10:31.265 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:30 smithi191 ceph-mon[98311]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T19:10:31.265 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:30 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:10:31.265 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:30 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:10:31.266 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:30 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:10:31.266 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:30 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:10:31.266 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:30 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi130", "name": "osd_memory_target"}]: dispatch 2023-07-14T19:10:31.266 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:30 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:10:31.266 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:30 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T19:10:31.266 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:30 smithi191 ceph-mon[98311]: Updating smithi130:/etc/ceph/ceph.conf 2023-07-14T19:10:31.266 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:30 smithi191 ceph-mon[98311]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi191", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T19:10:31.267 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:30 smithi191 ceph-mon[98311]: Updating smithi130:/var/lib/ceph/adc52f04-2279-11ee-9b33-001a4aab830c/config/ceph.conf 2023-07-14T19:10:31.267 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:30 smithi191 ceph-mon[98311]: Updating smithi130:/etc/ceph/ceph.client.admin.keyring 2023-07-14T19:10:31.267 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:30 smithi191 ceph-mon[98311]: Deploying cephadm binary to smithi191 2023-07-14T19:10:31.267 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:30 smithi191 ceph-mon[98311]: Updating smithi130:/var/lib/ceph/adc52f04-2279-11ee-9b33-001a4aab830c/config/ceph.client.admin.keyring 2023-07-14T19:10:31.267 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:30 smithi191 ceph-mon[98311]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T19:10:31.267 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:30 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:10:31.267 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:30 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:10:31.267 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:30 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:10:31.268 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:30 smithi191 ceph-mon[98311]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T19:10:31.268 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:30 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:10:31.268 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:30 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T19:10:31.268 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:30 smithi191 ceph-mon[98311]: Added host smithi191 2023-07-14T19:10:31.268 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:30 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:10:31.268 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:30 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:10:31.268 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:30 smithi191 ceph-mon[98311]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T19:10:31.268 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:30 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:10:31.269 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:30 smithi191 ceph-mon[98311]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-07-14T19:10:31.269 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:30 smithi191 ceph-mon[98311]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T19:10:31.269 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:30 smithi191 ceph-mon[98311]: from='client.? 172.21.15.40:0/1079880469' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2023-07-14T19:10:31.269 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:30 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:10:31.269 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:30 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:10:31.269 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:30 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:10:31.269 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:30 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:10:31.270 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:30 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi191", "name": "osd_memory_target"}]: dispatch 2023-07-14T19:10:31.270 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:30 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:10:31.270 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:30 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T19:10:31.270 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:30 smithi191 ceph-mon[98311]: Updating smithi191:/etc/ceph/ceph.conf 2023-07-14T19:10:31.270 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:30 smithi191 ceph-mon[98311]: Updating smithi191:/var/lib/ceph/adc52f04-2279-11ee-9b33-001a4aab830c/config/ceph.conf 2023-07-14T19:10:31.270 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:30 smithi191 ceph-mon[98311]: from='client.? 172.21.15.40:0/1079880469' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2023-07-14T19:10:31.270 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:30 smithi191 ceph-mon[98311]: osdmap e4: 0 total, 0 up, 0 in 2023-07-14T19:10:31.270 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:30 smithi191 ceph-mon[98311]: Updating smithi191:/etc/ceph/ceph.client.admin.keyring 2023-07-14T19:10:31.271 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:30 smithi191 ceph-mon[98311]: Updating smithi191:/var/lib/ceph/adc52f04-2279-11ee-9b33-001a4aab830c/config/ceph.client.admin.keyring 2023-07-14T19:10:31.271 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:30 smithi191 ceph-mon[98311]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T19:10:31.271 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:30 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:10:31.271 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:30 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:10:31.271 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:30 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:10:31.271 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:30 smithi191 ceph-mon[98311]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi040:172.21.15.40=a;smithi130:172.21.15.130=b;smithi191:172.21.15.191=c", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T19:10:31.271 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:30 smithi191 ceph-mon[98311]: Saving service mon spec with placement smithi040:172.21.15.40=a;smithi130:172.21.15.130=b;smithi191:172.21.15.191=c;count:3 2023-07-14T19:10:31.271 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:30 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:10:31.272 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:30 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T19:10:31.272 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:30 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:10:31.272 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:30 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T19:10:31.272 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:30 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:10:31.272 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:30 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-07-14T19:10:31.272 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:30 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:10:31.272 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:30 smithi191 ceph-mon[98311]: Deploying daemon mon.c on smithi191 2023-07-14T19:10:31.273 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:30 smithi191 ceph-mon[98311]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T19:10:31.273 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:30 smithi191 ceph-mon[98311]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T19:10:31.273 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:30 smithi191 ceph-mon[98311]: from='client.? 172.21.15.191:0/855704549' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-07-14T19:10:31.273 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:30 smithi191 ceph-mon[98311]: mon.c@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2023-07-14T19:10:31.273 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:30 smithi191 ceph-mon[98311]: mon.c@-1(synchronizing) e1 handle_conf_change mon_allow_pool_delete 2023-07-14T19:10:31.273 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:30 smithi191 ceph-mon[98311]: expand_channel_meta expand map: {default=false} 2023-07-14T19:10:31.273 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:30 smithi191 ceph-mon[98311]: expand_channel_meta from 'false' to 'false' 2023-07-14T19:10:31.274 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:30 smithi191 ceph-mon[98311]: expand_channel_meta expanded map: {default=false} 2023-07-14T19:10:31.274 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:30 smithi191 ceph-mon[98311]: expand_channel_meta expand map: {default=info} 2023-07-14T19:10:31.274 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:30 smithi191 ceph-mon[98311]: expand_channel_meta from 'info' to 'info' 2023-07-14T19:10:31.274 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:30 smithi191 ceph-mon[98311]: expand_channel_meta expanded map: {default=info} 2023-07-14T19:10:31.274 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:30 smithi191 ceph-mon[98311]: expand_channel_meta expand map: {default=daemon} 2023-07-14T19:10:31.274 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:30 smithi191 ceph-mon[98311]: expand_channel_meta from 'daemon' to 'daemon' 2023-07-14T19:10:31.274 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:30 smithi191 ceph-mon[98311]: expand_channel_meta expanded map: {default=daemon} 2023-07-14T19:10:31.274 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:30 smithi191 ceph-mon[98311]: expand_channel_meta expand map: {default=debug} 2023-07-14T19:10:31.275 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:30 smithi191 ceph-mon[98311]: expand_channel_meta from 'debug' to 'debug' 2023-07-14T19:10:31.275 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:30 smithi191 ceph-mon[98311]: expand_channel_meta expanded map: {default=debug} 2023-07-14T19:10:31.275 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:30 smithi191 systemd[1]: Started Ceph mon.c for adc52f04-2279-11ee-9b33-001a4aab830c. 2023-07-14T19:10:31.992 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2023-07-14T19:10:31.992 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid adc52f04-2279-11ee-9b33-001a4aab830c -- ceph mon dump -f json 2023-07-14T19:10:32.560 INFO:teuthology.orchestra.run.smithi191.stderr:Inferring config /var/lib/ceph/adc52f04-2279-11ee-9b33-001a4aab830c/mon.c/config 2023-07-14T19:10:36.472 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:10:35 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-07-14T19:10:36.472 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:10:35 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-14T19:10:36.472 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:10:35 smithi040 ceph-mon[96518]: mon.a calling monitor election 2023-07-14T19:10:36.472 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:10:35 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-14T19:10:36.472 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:10:35 smithi040 ceph-mon[96518]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T19:10:36.473 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:10:35 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-14T19:10:36.473 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:10:35 smithi040 ceph-mon[96518]: mon.c calling monitor election 2023-07-14T19:10:36.473 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:10:35 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-14T19:10:36.473 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:10:35 smithi040 ceph-mon[96518]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T19:10:36.473 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:10:35 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-14T19:10:36.473 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:10:35 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-14T19:10:36.473 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:10:35 smithi040 ceph-mon[96518]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2023-07-14T19:10:36.473 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:10:35 smithi040 ceph-mon[96518]: monmap e2: 2 mons at {a=[v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0],c=[v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0]} removed_ranks: {} 2023-07-14T19:10:36.474 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:10:35 smithi040 ceph-mon[96518]: fsmap 2023-07-14T19:10:36.474 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:10:35 smithi040 ceph-mon[96518]: osdmap e4: 0 total, 0 up, 0 in 2023-07-14T19:10:36.474 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:10:35 smithi040 ceph-mon[96518]: mgrmap e13: x(active, since 45s) 2023-07-14T19:10:36.474 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:10:35 smithi040 ceph-mon[96518]: overall HEALTH_OK 2023-07-14T19:10:36.474 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:10:35 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:10:36.474 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:10:35 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:10:36.474 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:10:35 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:10:36.475 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:10:35 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-07-14T19:10:36.475 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:10:35 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:10:36.475 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:35 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-07-14T19:10:36.475 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:35 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-14T19:10:36.475 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:35 smithi191 ceph-mon[98311]: mon.a calling monitor election 2023-07-14T19:10:36.476 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:35 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-14T19:10:36.476 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:35 smithi191 ceph-mon[98311]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T19:10:36.476 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:35 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-14T19:10:36.476 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:35 smithi191 ceph-mon[98311]: mon.c calling monitor election 2023-07-14T19:10:36.476 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:35 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-14T19:10:36.476 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:35 smithi191 ceph-mon[98311]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T19:10:36.476 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:35 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-14T19:10:36.477 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:35 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-14T19:10:36.477 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:35 smithi191 ceph-mon[98311]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2023-07-14T19:10:36.477 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:35 smithi191 ceph-mon[98311]: monmap e2: 2 mons at {a=[v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0],c=[v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0]} removed_ranks: {} 2023-07-14T19:10:36.477 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:35 smithi191 ceph-mon[98311]: fsmap 2023-07-14T19:10:36.477 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:35 smithi191 ceph-mon[98311]: osdmap e4: 0 total, 0 up, 0 in 2023-07-14T19:10:36.477 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:35 smithi191 ceph-mon[98311]: mgrmap e13: x(active, since 45s) 2023-07-14T19:10:36.477 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:35 smithi191 ceph-mon[98311]: overall HEALTH_OK 2023-07-14T19:10:36.478 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:35 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:10:36.478 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:35 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:10:36.478 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:35 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:10:36.478 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:35 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-07-14T19:10:36.478 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:35 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:10:36.741 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:10:36.738+0000 7f89b264c700 1 -- 172.21.15.191:0/2948770279 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f89940035c0 msgr2=0x7f8994005a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:10:36.741 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:10:36.738+0000 7f89b264c700 1 --2- 172.21.15.191:0/2948770279 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f89940035c0 0x7f8994005a40 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f89ac075150 tx=0x7f899c010ec0 comp rx=0 tx=0).stop 2023-07-14T19:10:36.741 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:10:36.738+0000 7f89b264c700 1 -- 172.21.15.191:0/2948770279 shutdown_connections 2023-07-14T19:10:36.741 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:10:36.738+0000 7f89b264c700 1 --2- 172.21.15.191:0/2948770279 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f89940035c0 0x7f8994005a40 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:10:36.742 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:10:36.738+0000 7f89b264c700 1 -- 172.21.15.191:0/2948770279 >> 172.21.15.191:0/2948770279 conn(0x7f89ac073ea0 msgr2=0x7f89ac0742a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:10:36.742 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:10:36.738+0000 7f89b264c700 1 -- 172.21.15.191:0/2948770279 shutdown_connections 2023-07-14T19:10:36.742 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:10:36.738+0000 7f89b264c700 1 -- 172.21.15.191:0/2948770279 wait complete. 2023-07-14T19:10:36.743 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:10:36.739+0000 7f89b264c700 1 Processor -- start 2023-07-14T19:10:36.743 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:10:36.739+0000 7f89b264c700 1 -- start start 2023-07-14T19:10:36.743 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:10:36.740+0000 7f89b264c700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f89940035c0 0x7f89ac194cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:10:36.743 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:10:36.740+0000 7f89b264c700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f89ac1951f0 0x7f89ac1985b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:10:36.744 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:10:36.740+0000 7f89b264c700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89ac195770 con 0x7f89940035c0 2023-07-14T19:10:36.744 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:10:36.740+0000 7f89b264c700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89ac1958b0 con 0x7f89ac1951f0 2023-07-14T19:10:36.744 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:10:36.740+0000 7f89ab7fe700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f89ac1951f0 0x7f89ac1985b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:10:36.744 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:10:36.740+0000 7f89ab7fe700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f89ac1951f0 0x7f89ac1985b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.191:59206/0 (socket says 172.21.15.191:59206) 2023-07-14T19:10:36.744 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:10:36.740+0000 7f89abfff700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f89940035c0 0x7f89ac194cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:10:36.744 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:10:36.740+0000 7f89ab7fe700 1 -- 172.21.15.191:0/1022453596 learned_addr learned my addr 172.21.15.191:0/1022453596 (peer_addr_for_me v2:172.21.15.191:0/0) 2023-07-14T19:10:36.744 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:10:36.740+0000 7f89ab7fe700 1 -- 172.21.15.191:0/1022453596 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f89ac1951f0 msgr2=0x7f89ac1985b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2023-07-14T19:10:36.745 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:10:36.740+0000 7f89ab7fe700 1 -- 172.21.15.191:0/1022453596 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f89ac1951f0 msgr2=0x7f89ac1985b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2023-07-14T19:10:36.745 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:10:36.740+0000 7f89ab7fe700 1 --2- 172.21.15.191:0/1022453596 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f89ac1951f0 0x7f89ac1985b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-07-14T19:10:36.745 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:10:36.740+0000 7f89ab7fe700 1 --2- 172.21.15.191:0/1022453596 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f89ac1951f0 0x7f89ac1985b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-07-14T19:10:36.745 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:10:36.740+0000 7f89abfff700 1 -- 172.21.15.191:0/1022453596 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f89ac1951f0 msgr2=0x7f89ac1985b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-07-14T19:10:36.745 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:10:36.741+0000 7f89abfff700 1 --2- 172.21.15.191:0/1022453596 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f89ac1951f0 0x7f89ac1985b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:10:36.745 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:10:36.741+0000 7f89abfff700 1 -- 172.21.15.191:0/1022453596 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f899c010910 con 0x7f89940035c0 2023-07-14T19:10:36.745 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:10:36.741+0000 7f89abfff700 1 --2- 172.21.15.191:0/1022453596 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f89940035c0 0x7f89ac194cb0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f89ac075150 tx=0x7f899c0055e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:10:36.746 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:10:36.741+0000 7f89a97fa700 1 -- 172.21.15.191:0/1022453596 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 309+0+0 (secure 0 0 0) 0x7f899c017040 con 0x7f89940035c0 2023-07-14T19:10:36.746 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:10:36.741+0000 7f89a97fa700 1 -- 172.21.15.191:0/1022453596 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f899c0044f0 con 0x7f89940035c0 2023-07-14T19:10:36.747 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:10:36.741+0000 7f89b264c700 1 -- 172.21.15.191:0/1022453596 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f89ac198af0 con 0x7f89940035c0 2023-07-14T19:10:36.747 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:10:36.741+0000 7f89a97fa700 1 -- 172.21.15.191:0/1022453596 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 309+0+0 (secure 0 0 0) 0x7f899c003e90 con 0x7f89940035c0 2023-07-14T19:10:36.747 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:10:36.741+0000 7f89b264c700 1 -- 172.21.15.191:0/1022453596 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f89ac1990c0 con 0x7f89940035c0 2023-07-14T19:10:36.747 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:10:36.742+0000 7f89a97fa700 1 -- 172.21.15.191:0/1022453596 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 13) v1 ==== 45539+0+0 (secure 0 0 0) 0x7f899c0244d0 con 0x7f89940035c0 2023-07-14T19:10:36.747 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:10:36.742+0000 7f89b264c700 1 -- 172.21.15.191:0/1022453596 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f89ac061c70 con 0x7f89940035c0 2023-07-14T19:10:36.747 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:10:36.742+0000 7f89a97fa700 1 --2- 172.21.15.191:0/1022453596 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f8998038580 0x7f899803aa30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:10:36.748 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:10:36.742+0000 7f89a97fa700 1 -- 172.21.15.191:0/1022453596 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f899c000e40 con 0x7f89940035c0 2023-07-14T19:10:36.748 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:10:36.743+0000 7f89ab7fe700 1 --2- 172.21.15.191:0/1022453596 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f8998038580 0x7f899803aa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:10:36.748 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:10:36.743+0000 7f89ab7fe700 1 --2- 172.21.15.191:0/1022453596 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f8998038580 0x7f899803aa30 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f89a0009180 tx=0x7f89a0008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:10:36.750 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:10:36.748+0000 7f89a97fa700 1 -- 172.21.15.191:0/1022453596 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f899c02b350 con 0x7f89940035c0 2023-07-14T19:10:37.014 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:10:37.011+0000 7f89b264c700 1 -- 172.21.15.191:0/1022453596 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f89ac19f930 con 0x7f89940035c0 2023-07-14T19:10:37.014 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:10:37.012+0000 7f89a97fa700 1 -- 172.21.15.191:0/1022453596 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 2 v2) v1 ==== 95+0+1006 (secure 0 0 0) 0x7f899c022b30 con 0x7f89940035c0 2023-07-14T19:10:37.015 INFO:teuthology.orchestra.run.smithi191.stdout: 2023-07-14T19:10:37.015 INFO:teuthology.orchestra.run.smithi191.stdout:{"epoch":2,"fsid":"adc52f04-2279-11ee-9b33-001a4aab830c","modified":"2023-07-14T19:10:30.892293Z","created":"2023-07-14T19:08:46.682427Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:3300","nonce":0},{"type":"v1","addr":"172.21.15.40:6789","nonce":0}]},"addr":"172.21.15.40:6789/0","public_addr":"172.21.15.40:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:3300","nonce":0},{"type":"v1","addr":"172.21.15.191:6789","nonce":0}]},"addr":"172.21.15.191:6789/0","public_addr":"172.21.15.191:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2023-07-14T19:10:37.016 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:10:37.014+0000 7f89b264c700 1 -- 172.21.15.191:0/1022453596 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f8998038580 msgr2=0x7f899803aa30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:10:37.017 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:10:37.014+0000 7f89b264c700 1 --2- 172.21.15.191:0/1022453596 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f8998038580 0x7f899803aa30 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f89a0009180 tx=0x7f89a0008040 comp rx=0 tx=0).stop 2023-07-14T19:10:37.017 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:10:37.014+0000 7f89b264c700 1 -- 172.21.15.191:0/1022453596 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f89940035c0 msgr2=0x7f89ac194cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:10:37.017 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:10:37.014+0000 7f89b264c700 1 --2- 172.21.15.191:0/1022453596 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f89940035c0 0x7f89ac194cb0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f89ac075150 tx=0x7f899c0055e0 comp rx=0 tx=0).stop 2023-07-14T19:10:37.017 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:10:37.014+0000 7f89b264c700 1 -- 172.21.15.191:0/1022453596 shutdown_connections 2023-07-14T19:10:37.017 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:10:37.014+0000 7f89b264c700 1 --2- 172.21.15.191:0/1022453596 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f8998038580 0x7f899803aa30 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:10:37.017 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:10:37.014+0000 7f89b264c700 1 --2- 172.21.15.191:0/1022453596 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f89940035c0 0x7f89ac194cb0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:10:37.017 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:10:37.014+0000 7f89b264c700 1 --2- 172.21.15.191:0/1022453596 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f89ac1951f0 0x7f89ac1985b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:10:37.018 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:10:37.014+0000 7f89b264c700 1 -- 172.21.15.191:0/1022453596 >> 172.21.15.191:0/1022453596 conn(0x7f89ac073ea0 msgr2=0x7f89ac0ffbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:10:37.018 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:10:37.014+0000 7f89b264c700 1 -- 172.21.15.191:0/1022453596 shutdown_connections 2023-07-14T19:10:37.018 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:10:37.014+0000 7f89b264c700 1 -- 172.21.15.191:0/1022453596 wait complete. 2023-07-14T19:10:37.018 INFO:teuthology.orchestra.run.smithi191.stderr:dumped monmap epoch 2 2023-07-14T19:10:37.722 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:37 smithi191 ceph-mon[98311]: Deploying daemon mon.b on smithi130 2023-07-14T19:10:37.723 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:37 smithi191 ceph-mon[98311]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T19:10:37.723 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:37 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-14T19:10:37.723 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:37 smithi191 ceph-mon[98311]: from='client.? 172.21.15.191:0/1022453596' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-07-14T19:10:37.896 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:10:37 smithi040 ceph-mon[96518]: Deploying daemon mon.b on smithi130 2023-07-14T19:10:37.896 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:10:37 smithi040 ceph-mon[96518]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T19:10:37.896 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:10:37 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-14T19:10:37.896 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:10:37 smithi040 ceph-mon[96518]: from='client.? 172.21.15.191:0/1022453596' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-07-14T19:10:38.221 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Jul 14 19:10:37 smithi040 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-mgr-x[96762]: 2023-07-14T19:10:37.892+0000 7f2fea5ba700 -1 mgr.server handle_report got status from non-daemon mon.c 2023-07-14T19:10:38.508 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2023-07-14T19:10:38.509 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid adc52f04-2279-11ee-9b33-001a4aab830c -- ceph mon dump -f json 2023-07-14T19:10:38.701 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:38 smithi130 systemd[1]: Starting Ceph mon.b for adc52f04-2279-11ee-9b33-001a4aab830c... 2023-07-14T19:10:38.952 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:38 smithi130 podman[97792]: 2023-07-14T19:10:38.952 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:38 smithi130 ceph-mon[97832]: set uid:gid to 167:167 (ceph:ceph) 2023-07-14T19:10:38.952 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:38 smithi130 ceph-mon[97832]: ceph version 18.0.0-4940-g81c9387d (81c9387d2e7576da065747ec9c16d1c7d69016ec) reef (dev), process ceph-mon, pid 7 2023-07-14T19:10:38.952 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:38 smithi130 ceph-mon[97832]: pidfile_write: ignore empty --pid-file 2023-07-14T19:10:38.952 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:38 smithi130 ceph-mon[97832]: load: jerasure load: lrc 2023-07-14T19:10:38.953 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:38 smithi130 ceph-mon[97832]: rocksdb: RocksDB version: 7.9.2 2023-07-14T19:10:38.953 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:38 smithi130 ceph-mon[97832]: rocksdb: Git sha 0 2023-07-14T19:10:38.953 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:38 smithi130 ceph-mon[97832]: rocksdb: Compile date 2023-07-14 14:45:42 2023-07-14T19:10:38.953 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:38 smithi130 ceph-mon[97832]: rocksdb: DB SUMMARY 2023-07-14T19:10:38.953 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:38 smithi130 ceph-mon[97832]: rocksdb: DB Session ID: JZ4PGUXVYBRBX79T5C8Y 2023-07-14T19:10:38.953 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:38 smithi130 ceph-mon[97832]: rocksdb: CURRENT file: CURRENT 2023-07-14T19:10:38.953 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:38 smithi130 ceph-mon[97832]: rocksdb: IDENTITY file: IDENTITY 2023-07-14T19:10:38.953 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:38 smithi130 ceph-mon[97832]: rocksdb: MANIFEST file: MANIFEST-000005 size: 59 Bytes 2023-07-14T19:10:38.954 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:38 smithi130 ceph-mon[97832]: rocksdb: SST files in /var/lib/ceph/mon/ceph-b/store.db dir, Total Num: 0, files: 2023-07-14T19:10:38.954 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:38 smithi130 ceph-mon[97832]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-b/store.db: 000004.log size: 636 ; 2023-07-14T19:10:38.954 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:38 smithi130 ceph-mon[97832]: rocksdb: Options.error_if_exists: 0 2023-07-14T19:10:38.954 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:38 smithi130 ceph-mon[97832]: rocksdb: Options.create_if_missing: 0 2023-07-14T19:10:38.954 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:38 smithi130 ceph-mon[97832]: rocksdb: Options.paranoid_checks: 1 2023-07-14T19:10:38.954 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:38 smithi130 ceph-mon[97832]: rocksdb: Options.flush_verify_memtable_count: 1 2023-07-14T19:10:38.954 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:38 smithi130 ceph-mon[97832]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2023-07-14T19:10:38.955 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:38 smithi130 ceph-mon[97832]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2023-07-14T19:10:38.955 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:38 smithi130 ceph-mon[97832]: rocksdb: Options.env: 0x55f446ddfc60 2023-07-14T19:10:38.955 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:38 smithi130 ceph-mon[97832]: rocksdb: Options.fs: PosixFileSystem 2023-07-14T19:10:38.955 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:38 smithi130 ceph-mon[97832]: rocksdb: Options.info_log: 0x55f4477192a0 2023-07-14T19:10:38.955 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:38 smithi130 ceph-mon[97832]: rocksdb: Options.max_file_opening_threads: 16 2023-07-14T19:10:38.955 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:38 smithi130 ceph-mon[97832]: rocksdb: Options.statistics: (nil) 2023-07-14T19:10:38.955 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:38 smithi130 ceph-mon[97832]: rocksdb: Options.use_fsync: 0 2023-07-14T19:10:38.955 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:38 smithi130 ceph-mon[97832]: rocksdb: Options.max_log_file_size: 0 2023-07-14T19:10:38.956 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:38 smithi130 ceph-mon[97832]: rocksdb: Options.max_manifest_file_size: 1073741824 2023-07-14T19:10:38.956 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:38 smithi130 ceph-mon[97832]: rocksdb: Options.log_file_time_to_roll: 0 2023-07-14T19:10:38.956 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:38 smithi130 ceph-mon[97832]: rocksdb: Options.keep_log_file_num: 1000 2023-07-14T19:10:38.956 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:38 smithi130 ceph-mon[97832]: rocksdb: Options.recycle_log_file_num: 0 2023-07-14T19:10:38.956 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:38 smithi130 ceph-mon[97832]: rocksdb: Options.allow_fallocate: 1 2023-07-14T19:10:38.956 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:38 smithi130 ceph-mon[97832]: rocksdb: Options.allow_mmap_reads: 0 2023-07-14T19:10:38.956 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:38 smithi130 ceph-mon[97832]: rocksdb: Options.allow_mmap_writes: 0 2023-07-14T19:10:38.957 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:38 smithi130 ceph-mon[97832]: rocksdb: Options.use_direct_reads: 0 2023-07-14T19:10:38.957 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:38 smithi130 ceph-mon[97832]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2023-07-14T19:10:38.957 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:38 smithi130 ceph-mon[97832]: rocksdb: Options.create_missing_column_families: 0 2023-07-14T19:10:38.957 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:38 smithi130 ceph-mon[97832]: rocksdb: Options.db_log_dir: 2023-07-14T19:10:38.957 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:38 smithi130 ceph-mon[97832]: rocksdb: Options.wal_dir: 2023-07-14T19:10:38.957 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:38 smithi130 ceph-mon[97832]: rocksdb: Options.table_cache_numshardbits: 6 2023-07-14T19:10:38.957 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:38 smithi130 ceph-mon[97832]: rocksdb: Options.WAL_ttl_seconds: 0 2023-07-14T19:10:38.957 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:38 smithi130 ceph-mon[97832]: rocksdb: Options.WAL_size_limit_MB: 0 2023-07-14T19:10:38.958 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:38 smithi130 ceph-mon[97832]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2023-07-14T19:10:38.958 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:38 smithi130 ceph-mon[97832]: rocksdb: Options.manifest_preallocation_size: 4194304 2023-07-14T19:10:38.958 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:38 smithi130 ceph-mon[97832]: rocksdb: Options.is_fd_close_on_exec: 1 2023-07-14T19:10:38.958 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:38 smithi130 ceph-mon[97832]: rocksdb: Options.advise_random_on_open: 1 2023-07-14T19:10:38.958 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:38 smithi130 ceph-mon[97832]: rocksdb: Options.db_write_buffer_size: 0 2023-07-14T19:10:38.958 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:38 smithi130 ceph-mon[97832]: rocksdb: Options.write_buffer_manager: 0x55f44763a3c0 2023-07-14T19:10:38.958 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:38 smithi130 ceph-mon[97832]: rocksdb: Options.access_hint_on_compaction_start: 1 2023-07-14T19:10:38.958 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:38 smithi130 ceph-mon[97832]: rocksdb: Options.random_access_max_buffer_size: 1048576 2023-07-14T19:10:38.959 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:38 smithi130 ceph-mon[97832]: rocksdb: Options.use_adaptive_mutex: 0 2023-07-14T19:10:38.959 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:38 smithi130 ceph-mon[97832]: rocksdb: Options.rate_limiter: (nil) 2023-07-14T19:10:38.959 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:38 smithi130 ceph-mon[97832]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2023-07-14T19:10:38.959 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:38 smithi130 ceph-mon[97832]: rocksdb: Options.wal_recovery_mode: 2 2023-07-14T19:10:38.959 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:38 smithi130 ceph-mon[97832]: rocksdb: Options.enable_thread_tracking: 0 2023-07-14T19:10:38.959 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:38 smithi130 ceph-mon[97832]: rocksdb: Options.enable_pipelined_write: 0 2023-07-14T19:10:38.959 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:38 smithi130 ceph-mon[97832]: rocksdb: Options.unordered_write: 0 2023-07-14T19:10:38.960 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:38 smithi130 ceph-mon[97832]: rocksdb: Options.allow_concurrent_memtable_write: 1 2023-07-14T19:10:38.960 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:38 smithi130 ceph-mon[97832]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2023-07-14T19:10:38.960 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:38 smithi130 ceph-mon[97832]: rocksdb: Options.write_thread_max_yield_usec: 100 2023-07-14T19:10:38.960 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:38 smithi130 ceph-mon[97832]: rocksdb: Options.write_thread_slow_yield_usec: 3 2023-07-14T19:10:38.960 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:38 smithi130 ceph-mon[97832]: rocksdb: Options.row_cache: None 2023-07-14T19:10:38.960 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:38 smithi130 ceph-mon[97832]: rocksdb: Options.wal_filter: None 2023-07-14T19:10:38.960 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:38 smithi130 ceph-mon[97832]: rocksdb: Options.avoid_flush_during_recovery: 0 2023-07-14T19:10:38.960 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:38 smithi130 ceph-mon[97832]: rocksdb: Options.allow_ingest_behind: 0 2023-07-14T19:10:38.961 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:38 smithi130 ceph-mon[97832]: rocksdb: Options.two_write_queues: 0 2023-07-14T19:10:38.961 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:38 smithi130 ceph-mon[97832]: rocksdb: Options.manual_wal_flush: 0 2023-07-14T19:10:38.961 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:38 smithi130 ceph-mon[97832]: rocksdb: Options.wal_compression: 0 2023-07-14T19:10:38.961 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:38 smithi130 ceph-mon[97832]: rocksdb: Options.atomic_flush: 0 2023-07-14T19:10:38.961 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:38 smithi130 ceph-mon[97832]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2023-07-14T19:10:38.961 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:38 smithi130 ceph-mon[97832]: rocksdb: Options.persist_stats_to_disk: 0 2023-07-14T19:10:38.961 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:38 smithi130 ceph-mon[97832]: rocksdb: Options.write_dbid_to_manifest: 0 2023-07-14T19:10:38.962 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:38 smithi130 ceph-mon[97832]: rocksdb: Options.log_readahead_size: 0 2023-07-14T19:10:38.962 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:38 smithi130 ceph-mon[97832]: rocksdb: Options.file_checksum_gen_factory: Unknown 2023-07-14T19:10:38.962 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:38 smithi130 ceph-mon[97832]: rocksdb: Options.best_efforts_recovery: 0 2023-07-14T19:10:38.962 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:38 smithi130 ceph-mon[97832]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2023-07-14T19:10:38.962 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:38 smithi130 ceph-mon[97832]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2023-07-14T19:10:38.962 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:38 smithi130 ceph-mon[97832]: rocksdb: Options.allow_data_in_errors: 0 2023-07-14T19:10:38.962 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:38 smithi130 ceph-mon[97832]: rocksdb: Options.db_host_id: __hostname__ 2023-07-14T19:10:38.962 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:38 smithi130 ceph-mon[97832]: rocksdb: Options.enforce_single_del_contracts: true 2023-07-14T19:10:38.963 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:38 smithi130 ceph-mon[97832]: rocksdb: Options.max_background_jobs: 2 2023-07-14T19:10:38.963 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:38 smithi130 ceph-mon[97832]: rocksdb: Options.max_background_compactions: -1 2023-07-14T19:10:38.963 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:38 smithi130 ceph-mon[97832]: rocksdb: Options.max_subcompactions: 1 2023-07-14T19:10:38.963 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:38 smithi130 ceph-mon[97832]: rocksdb: Options.avoid_flush_during_shutdown: 0 2023-07-14T19:10:38.963 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:38 smithi130 ceph-mon[97832]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2023-07-14T19:10:38.963 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:38 smithi130 ceph-mon[97832]: rocksdb: Options.delayed_write_rate : 16777216 2023-07-14T19:10:38.963 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:38 smithi130 ceph-mon[97832]: rocksdb: Options.max_total_wal_size: 0 2023-07-14T19:10:38.963 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:38 smithi130 ceph-mon[97832]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2023-07-14T19:10:38.964 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:38 smithi130 ceph-mon[97832]: rocksdb: Options.stats_dump_period_sec: 600 2023-07-14T19:10:38.964 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:38 smithi130 ceph-mon[97832]: rocksdb: Options.stats_persist_period_sec: 600 2023-07-14T19:10:38.964 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:38 smithi130 ceph-mon[97832]: rocksdb: Options.stats_history_buffer_size: 1048576 2023-07-14T19:10:38.964 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:38 smithi130 ceph-mon[97832]: rocksdb: Options.max_open_files: -1 2023-07-14T19:10:38.964 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:38 smithi130 ceph-mon[97832]: rocksdb: Options.bytes_per_sync: 0 2023-07-14T19:10:38.964 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:38 smithi130 ceph-mon[97832]: rocksdb: Options.wal_bytes_per_sync: 0 2023-07-14T19:10:38.964 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:38 smithi130 ceph-mon[97832]: rocksdb: Options.strict_bytes_per_sync: 0 2023-07-14T19:10:38.965 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:38 smithi130 ceph-mon[97832]: rocksdb: Options.compaction_readahead_size: 0 2023-07-14T19:10:38.965 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:38 smithi130 ceph-mon[97832]: rocksdb: Options.max_background_flushes: -1 2023-07-14T19:10:38.965 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:38 smithi130 ceph-mon[97832]: rocksdb: Compression algorithms supported: 2023-07-14T19:10:38.965 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:38 smithi130 ceph-mon[97832]: rocksdb: kZSTDNotFinalCompression supported: 0 2023-07-14T19:10:38.965 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:38 smithi130 ceph-mon[97832]: rocksdb: kZSTD supported: 0 2023-07-14T19:10:38.965 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:38 smithi130 ceph-mon[97832]: rocksdb: kXpressCompression supported: 0 2023-07-14T19:10:38.965 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:38 smithi130 ceph-mon[97832]: rocksdb: kLZ4HCCompression supported: 1 2023-07-14T19:10:38.965 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:38 smithi130 ceph-mon[97832]: rocksdb: kZlibCompression supported: 1 2023-07-14T19:10:38.966 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:38 smithi130 ceph-mon[97832]: rocksdb: kSnappyCompression supported: 1 2023-07-14T19:10:38.966 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:38 smithi130 ceph-mon[97832]: rocksdb: kLZ4Compression supported: 1 2023-07-14T19:10:38.966 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:38 smithi130 ceph-mon[97832]: rocksdb: kBZip2Compression supported: 0 2023-07-14T19:10:38.966 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:38 smithi130 ceph-mon[97832]: rocksdb: Fast CRC32 supported: Supported on x86 2023-07-14T19:10:38.966 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:38 smithi130 ceph-mon[97832]: rocksdb: DMutex implementation: pthread_mutex_t 2023-07-14T19:10:38.966 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:38 smithi130 bash[97792]: 5f00f7e99a365fe6de93fe52a82f50c0bee03d140d4d139ddf25b9b8f11f406b 2023-07-14T19:10:38.966 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:38 smithi130 ceph-mon[97832]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000005 2023-07-14T19:10:38.966 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:38 smithi130 ceph-mon[97832]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2023-07-14T19:10:38.967 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:38 smithi130 ceph-mon[97832]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2023-07-14T19:10:38.967 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:38 smithi130 ceph-mon[97832]: rocksdb: Options.merge_operator: 2023-07-14T19:10:38.967 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:38 smithi130 ceph-mon[97832]: rocksdb: Options.compaction_filter: None 2023-07-14T19:10:38.967 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:38 smithi130 ceph-mon[97832]: rocksdb: Options.compaction_filter_factory: None 2023-07-14T19:10:38.967 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:38 smithi130 ceph-mon[97832]: rocksdb: Options.sst_partitioner_factory: None 2023-07-14T19:10:38.967 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:38 smithi130 ceph-mon[97832]: rocksdb: Options.memtable_factory: SkipListFactory 2023-07-14T19:10:38.967 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:38 smithi130 ceph-mon[97832]: rocksdb: Options.table_factory: BlockBasedTable 2023-07-14T19:10:38.968 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:38 smithi130 ceph-mon[97832]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55f4477193a0) 2023-07-14T19:10:38.968 INFO:journalctl@ceph.mon.b.smithi130.stdout: cache_index_and_filter_blocks: 1 2023-07-14T19:10:38.968 INFO:journalctl@ceph.mon.b.smithi130.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2023-07-14T19:10:38.968 INFO:journalctl@ceph.mon.b.smithi130.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2023-07-14T19:10:38.968 INFO:journalctl@ceph.mon.b.smithi130.stdout: pin_top_level_index_and_filter: 1 2023-07-14T19:10:38.968 INFO:journalctl@ceph.mon.b.smithi130.stdout: index_type: 0 2023-07-14T19:10:38.968 INFO:journalctl@ceph.mon.b.smithi130.stdout: data_block_index_type: 0 2023-07-14T19:10:38.968 INFO:journalctl@ceph.mon.b.smithi130.stdout: index_shortening: 1 2023-07-14T19:10:38.969 INFO:journalctl@ceph.mon.b.smithi130.stdout: data_block_hash_table_util_ratio: 0.750000 2023-07-14T19:10:38.969 INFO:journalctl@ceph.mon.b.smithi130.stdout: checksum: 4 2023-07-14T19:10:38.969 INFO:journalctl@ceph.mon.b.smithi130.stdout: no_block_cache: 0 2023-07-14T19:10:38.969 INFO:journalctl@ceph.mon.b.smithi130.stdout: block_cache: 0x55f4476d91f0 2023-07-14T19:10:38.969 INFO:journalctl@ceph.mon.b.smithi130.stdout: block_cache_name: BinnedLRUCache 2023-07-14T19:10:38.969 INFO:journalctl@ceph.mon.b.smithi130.stdout: block_cache_options: 2023-07-14T19:10:38.969 INFO:journalctl@ceph.mon.b.smithi130.stdout: capacity : 536870912 2023-07-14T19:10:38.969 INFO:journalctl@ceph.mon.b.smithi130.stdout: num_shard_bits : 4 2023-07-14T19:10:38.970 INFO:journalctl@ceph.mon.b.smithi130.stdout: strict_capacity_limit : 0 2023-07-14T19:10:38.970 INFO:journalctl@ceph.mon.b.smithi130.stdout: high_pri_pool_ratio: 0.000 2023-07-14T19:10:38.970 INFO:journalctl@ceph.mon.b.smithi130.stdout: block_cache_compressed: (nil) 2023-07-14T19:10:38.970 INFO:journalctl@ceph.mon.b.smithi130.stdout: persistent_cache: (nil) 2023-07-14T19:10:38.970 INFO:journalctl@ceph.mon.b.smithi130.stdout: block_size: 4096 2023-07-14T19:10:38.970 INFO:journalctl@ceph.mon.b.smithi130.stdout: block_size_deviation: 10 2023-07-14T19:10:38.970 INFO:journalctl@ceph.mon.b.smithi130.stdout: block_restart_interval: 16 2023-07-14T19:10:38.970 INFO:journalctl@ceph.mon.b.smithi130.stdout: index_block_restart_interval: 1 2023-07-14T19:10:38.971 INFO:journalctl@ceph.mon.b.smithi130.stdout: metadata_block_size: 4096 2023-07-14T19:10:38.971 INFO:journalctl@ceph.mon.b.smithi130.stdout: partition_filters: 0 2023-07-14T19:10:38.971 INFO:journalctl@ceph.mon.b.smithi130.stdout: use_delta_encoding: 1 2023-07-14T19:10:38.971 INFO:journalctl@ceph.mon.b.smithi130.stdout: filter_policy: bloomfilter 2023-07-14T19:10:38.971 INFO:journalctl@ceph.mon.b.smithi130.stdout: whole_key_filtering: 1 2023-07-14T19:10:38.971 INFO:journalctl@ceph.mon.b.smithi130.stdout: verify_compression: 0 2023-07-14T19:10:38.971 INFO:journalctl@ceph.mon.b.smithi130.stdout: read_amp_bytes_per_bit: 0 2023-07-14T19:10:38.971 INFO:journalctl@ceph.mon.b.smithi130.stdout: format_version: 5 2023-07-14T19:10:38.972 INFO:journalctl@ceph.mon.b.smithi130.stdout: enable_index_compression: 1 2023-07-14T19:10:38.972 INFO:journalctl@ceph.mon.b.smithi130.stdout: block_align: 0 2023-07-14T19:10:38.972 INFO:journalctl@ceph.mon.b.smithi130.stdout: max_auto_readahead_size: 262144 2023-07-14T19:10:38.972 INFO:journalctl@ceph.mon.b.smithi130.stdout: prepopulate_block_cache: 0 2023-07-14T19:10:38.972 INFO:journalctl@ceph.mon.b.smithi130.stdout: initial_auto_readahead_size: 8192 2023-07-14T19:10:38.972 INFO:journalctl@ceph.mon.b.smithi130.stdout: num_file_reads_for_auto_readahead: 2 2023-07-14T19:10:38.972 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:38 smithi130 ceph-mon[97832]: rocksdb: Options.write_buffer_size: 33554432 2023-07-14T19:10:38.973 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:38 smithi130 ceph-mon[97832]: rocksdb: Options.max_write_buffer_number: 2 2023-07-14T19:10:38.973 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:38 smithi130 ceph-mon[97832]: rocksdb: Options.compression: NoCompression 2023-07-14T19:10:38.973 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:38 smithi130 ceph-mon[97832]: rocksdb: Options.bottommost_compression: Disabled 2023-07-14T19:10:38.973 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:38 smithi130 ceph-mon[97832]: rocksdb: Options.prefix_extractor: nullptr 2023-07-14T19:10:38.973 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:38 smithi130 ceph-mon[97832]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2023-07-14T19:10:38.973 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:38 smithi130 ceph-mon[97832]: rocksdb: Options.num_levels: 7 2023-07-14T19:10:38.973 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:38 smithi130 ceph-mon[97832]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2023-07-14T19:10:38.973 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:38 smithi130 ceph-mon[97832]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2023-07-14T19:10:38.974 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:38 smithi130 ceph-mon[97832]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2023-07-14T19:10:38.974 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:38 smithi130 ceph-mon[97832]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2023-07-14T19:10:38.974 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:38 smithi130 ceph-mon[97832]: rocksdb: Options.bottommost_compression_opts.level: 32767 2023-07-14T19:10:38.974 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:38 smithi130 ceph-mon[97832]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2023-07-14T19:10:38.974 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:38 smithi130 ceph-mon[97832]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2023-07-14T19:10:38.975 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:38 smithi130 ceph-mon[97832]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2023-07-14T19:10:38.975 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:38 smithi130 ceph-mon[97832]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2023-07-14T19:10:38.976 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:38 smithi130 ceph-mon[97832]: rocksdb: Options.bottommost_compression_opts.enabled: false 2023-07-14T19:10:38.976 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:38 smithi130 ceph-mon[97832]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2023-07-14T19:10:38.976 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:38 smithi130 ceph-mon[97832]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2023-07-14T19:10:38.976 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:38 smithi130 ceph-mon[97832]: rocksdb: Options.compression_opts.window_bits: -14 2023-07-14T19:10:38.976 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:38 smithi130 ceph-mon[97832]: rocksdb: Options.compression_opts.level: 32767 2023-07-14T19:10:38.976 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:38 smithi130 ceph-mon[97832]: rocksdb: Options.compression_opts.strategy: 0 2023-07-14T19:10:38.976 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:38 smithi130 ceph-mon[97832]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2023-07-14T19:10:38.977 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:38 smithi130 ceph-mon[97832]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2023-07-14T19:10:38.977 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:38 smithi130 ceph-mon[97832]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2023-07-14T19:10:38.977 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:38 smithi130 ceph-mon[97832]: rocksdb: Options.compression_opts.parallel_threads: 1 2023-07-14T19:10:38.977 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:38 smithi130 ceph-mon[97832]: rocksdb: Options.compression_opts.enabled: false 2023-07-14T19:10:38.977 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:38 smithi130 ceph-mon[97832]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2023-07-14T19:10:38.977 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:38 smithi130 ceph-mon[97832]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2023-07-14T19:10:38.977 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:38 smithi130 ceph-mon[97832]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2023-07-14T19:10:38.977 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:38 smithi130 ceph-mon[97832]: rocksdb: Options.level0_stop_writes_trigger: 36 2023-07-14T19:10:38.978 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:38 smithi130 ceph-mon[97832]: rocksdb: Options.target_file_size_base: 67108864 2023-07-14T19:10:38.978 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:38 smithi130 ceph-mon[97832]: rocksdb: Options.target_file_size_multiplier: 1 2023-07-14T19:10:38.978 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:38 smithi130 ceph-mon[97832]: rocksdb: Options.max_bytes_for_level_base: 268435456 2023-07-14T19:10:38.978 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:38 smithi130 ceph-mon[97832]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2023-07-14T19:10:38.978 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:38 smithi130 ceph-mon[97832]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2023-07-14T19:10:38.978 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:38 smithi130 ceph-mon[97832]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2023-07-14T19:10:38.978 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:38 smithi130 ceph-mon[97832]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2023-07-14T19:10:38.979 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:38 smithi130 ceph-mon[97832]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2023-07-14T19:10:38.979 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:38 smithi130 ceph-mon[97832]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2023-07-14T19:10:38.979 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:38 smithi130 ceph-mon[97832]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2023-07-14T19:10:38.979 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:38 smithi130 ceph-mon[97832]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2023-07-14T19:10:38.979 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:38 smithi130 ceph-mon[97832]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2023-07-14T19:10:38.979 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:38 smithi130 ceph-mon[97832]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2023-07-14T19:10:38.979 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:38 smithi130 ceph-mon[97832]: rocksdb: Options.max_compaction_bytes: 1677721600 2023-07-14T19:10:38.979 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:38 smithi130 ceph-mon[97832]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2023-07-14T19:10:38.980 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:38 smithi130 ceph-mon[97832]: rocksdb: Options.arena_block_size: 1048576 2023-07-14T19:10:38.980 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:38 smithi130 ceph-mon[97832]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2023-07-14T19:10:38.980 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:38 smithi130 ceph-mon[97832]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2023-07-14T19:10:38.980 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:38 smithi130 ceph-mon[97832]: rocksdb: Options.disable_auto_compactions: 0 2023-07-14T19:10:38.980 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:38 smithi130 ceph-mon[97832]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2023-07-14T19:10:38.980 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:38 smithi130 ceph-mon[97832]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2023-07-14T19:10:38.980 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:38 smithi130 ceph-mon[97832]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2023-07-14T19:10:38.980 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:38 smithi130 ceph-mon[97832]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2023-07-14T19:10:38.981 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:38 smithi130 ceph-mon[97832]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2023-07-14T19:10:38.981 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:38 smithi130 ceph-mon[97832]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2023-07-14T19:10:38.981 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:38 smithi130 ceph-mon[97832]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2023-07-14T19:10:38.981 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:38 smithi130 ceph-mon[97832]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2023-07-14T19:10:38.981 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:38 smithi130 ceph-mon[97832]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2023-07-14T19:10:38.981 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:38 smithi130 ceph-mon[97832]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2023-07-14T19:10:38.981 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:38 smithi130 ceph-mon[97832]: rocksdb: Options.table_properties_collectors: 2023-07-14T19:10:38.982 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:38 smithi130 ceph-mon[97832]: rocksdb: Options.inplace_update_support: 0 2023-07-14T19:10:38.982 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:38 smithi130 ceph-mon[97832]: rocksdb: Options.inplace_update_num_locks: 10000 2023-07-14T19:10:38.982 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:38 smithi130 ceph-mon[97832]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2023-07-14T19:10:38.982 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:38 smithi130 ceph-mon[97832]: rocksdb: Options.memtable_whole_key_filtering: 0 2023-07-14T19:10:38.982 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:38 smithi130 ceph-mon[97832]: rocksdb: Options.memtable_huge_page_size: 0 2023-07-14T19:10:38.982 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:38 smithi130 ceph-mon[97832]: rocksdb: Options.bloom_locality: 0 2023-07-14T19:10:38.982 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:38 smithi130 ceph-mon[97832]: rocksdb: Options.max_successive_merges: 0 2023-07-14T19:10:38.982 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:38 smithi130 ceph-mon[97832]: rocksdb: Options.optimize_filters_for_hits: 0 2023-07-14T19:10:38.983 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:38 smithi130 ceph-mon[97832]: rocksdb: Options.paranoid_file_checks: 0 2023-07-14T19:10:38.983 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:38 smithi130 ceph-mon[97832]: rocksdb: Options.force_consistency_checks: 1 2023-07-14T19:10:38.983 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:38 smithi130 ceph-mon[97832]: rocksdb: Options.report_bg_io_stats: 0 2023-07-14T19:10:38.983 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:38 smithi130 ceph-mon[97832]: rocksdb: Options.ttl: 2592000 2023-07-14T19:10:38.983 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:38 smithi130 ceph-mon[97832]: rocksdb: Options.periodic_compaction_seconds: 0 2023-07-14T19:10:38.983 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:38 smithi130 ceph-mon[97832]: rocksdb: Options.preclude_last_level_data_seconds: 0 2023-07-14T19:10:38.983 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:38 smithi130 ceph-mon[97832]: rocksdb: Options.preserve_internal_time_seconds: 0 2023-07-14T19:10:38.983 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:38 smithi130 ceph-mon[97832]: rocksdb: Options.enable_blob_files: false 2023-07-14T19:10:38.984 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:38 smithi130 ceph-mon[97832]: rocksdb: Options.min_blob_size: 0 2023-07-14T19:10:38.984 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:38 smithi130 ceph-mon[97832]: rocksdb: Options.blob_file_size: 268435456 2023-07-14T19:10:38.984 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:38 smithi130 ceph-mon[97832]: rocksdb: Options.blob_compression_type: NoCompression 2023-07-14T19:10:38.984 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:38 smithi130 ceph-mon[97832]: rocksdb: Options.enable_blob_garbage_collection: false 2023-07-14T19:10:38.984 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:38 smithi130 ceph-mon[97832]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2023-07-14T19:10:38.984 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:38 smithi130 ceph-mon[97832]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2023-07-14T19:10:38.984 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:38 smithi130 ceph-mon[97832]: rocksdb: Options.blob_compaction_readahead_size: 0 2023-07-14T19:10:38.985 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:38 smithi130 ceph-mon[97832]: rocksdb: Options.blob_file_starting_level: 0 2023-07-14T19:10:38.985 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:38 smithi130 ceph-mon[97832]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2023-07-14T19:10:38.985 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:38 smithi130 ceph-mon[97832]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2023-07-14T19:10:38.985 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:38 smithi130 ceph-mon[97832]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 0 2023-07-14T19:10:38.985 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:38 smithi130 ceph-mon[97832]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: cac9c12c-f160-4dc1-ab39-daff56f665b1 2023-07-14T19:10:38.985 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:38 smithi130 ceph-mon[97832]: rocksdb: EVENT_LOG_v1 {"time_micros": 1689361838924549, "job": 1, "event": "recovery_started", "wal_files": [4]} 2023-07-14T19:10:38.985 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:38 smithi130 ceph-mon[97832]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #4 mode 2 2023-07-14T19:10:38.985 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:38 smithi130 ceph-mon[97832]: rocksdb: EVENT_LOG_v1 {"time_micros": 1689361838924952, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1747, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1, "largest_seqno": 5, "table_properties": {"data_size": 648, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 526, "raw_average_value_size": 105, "num_data_blocks": 1, "num_entries": 5, "num_filter_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1689361838, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "cac9c12c-f160-4dc1-ab39-daff56f665b1", "db_session_id": "JZ4PGUXVYBRBX79T5C8Y", "orig_file_number": 8, "seqno_to_time_mapping": "N/A"}} 2023-07-14T19:10:38.986 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:38 smithi130 ceph-mon[97832]: rocksdb: EVENT_LOG_v1 {"time_micros": 1689361838925017, "job": 1, "event": "recovery_finished"} 2023-07-14T19:10:38.986 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:38 smithi130 ceph-mon[97832]: rocksdb: [db/version_set.cc:5047] Creating manifest 10 2023-07-14T19:10:38.986 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:38 smithi130 ceph-mon[97832]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-b/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2023-07-14T19:10:38.986 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:38 smithi130 ceph-mon[97832]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55f447772000 2023-07-14T19:10:38.986 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:38 smithi130 ceph-mon[97832]: rocksdb: DB pointer 0x55f44775e000 2023-07-14T19:10:38.986 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:38 smithi130 ceph-mon[97832]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2023-07-14T19:10:38.986 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:38 smithi130 ceph-mon[97832]: mon.b does not exist in monmap, will attempt to join an existing cluster 2023-07-14T19:10:38.987 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:38 smithi130 ceph-mon[97832]: rocksdb: [db/db_impl/db_impl.cc:1111] 2023-07-14T19:10:38.987 INFO:journalctl@ceph.mon.b.smithi130.stdout: ** DB Stats ** 2023-07-14T19:10:38.987 INFO:journalctl@ceph.mon.b.smithi130.stdout: Uptime(secs): 0.0 total, 0.0 interval 2023-07-14T19:10:38.987 INFO:journalctl@ceph.mon.b.smithi130.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2023-07-14T19:10:38.987 INFO:journalctl@ceph.mon.b.smithi130.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2023-07-14T19:10:38.987 INFO:journalctl@ceph.mon.b.smithi130.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2023-07-14T19:10:38.987 INFO:journalctl@ceph.mon.b.smithi130.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2023-07-14T19:10:38.987 INFO:journalctl@ceph.mon.b.smithi130.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2023-07-14T19:10:38.988 INFO:journalctl@ceph.mon.b.smithi130.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2023-07-14T19:10:38.988 INFO:journalctl@ceph.mon.b.smithi130.stdout: 2023-07-14T19:10:38.988 INFO:journalctl@ceph.mon.b.smithi130.stdout: ** Compaction Stats [default] ** 2023-07-14T19:10:38.988 INFO:journalctl@ceph.mon.b.smithi130.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2023-07-14T19:10:38.988 INFO:journalctl@ceph.mon.b.smithi130.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2023-07-14T19:10:38.988 INFO:journalctl@ceph.mon.b.smithi130.stdout: L0 1/0 1.71 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.3 0.00 0.00 1 0.000 0 0 0.0 0.0 2023-07-14T19:10:38.988 INFO:journalctl@ceph.mon.b.smithi130.stdout: Sum 1/0 1.71 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.3 0.00 0.00 1 0.000 0 0 0.0 0.0 2023-07-14T19:10:38.989 INFO:journalctl@ceph.mon.b.smithi130.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.3 0.00 0.00 1 0.000 0 0 0.0 0.0 2023-07-14T19:10:38.989 INFO:journalctl@ceph.mon.b.smithi130.stdout: 2023-07-14T19:10:38.989 INFO:journalctl@ceph.mon.b.smithi130.stdout: ** Compaction Stats [default] ** 2023-07-14T19:10:38.989 INFO:journalctl@ceph.mon.b.smithi130.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2023-07-14T19:10:38.989 INFO:journalctl@ceph.mon.b.smithi130.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2023-07-14T19:10:38.989 INFO:journalctl@ceph.mon.b.smithi130.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 4.3 0.00 0.00 1 0.000 0 0 0.0 0.0 2023-07-14T19:10:38.989 INFO:journalctl@ceph.mon.b.smithi130.stdout: 2023-07-14T19:10:38.989 INFO:journalctl@ceph.mon.b.smithi130.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2023-07-14T19:10:38.990 INFO:journalctl@ceph.mon.b.smithi130.stdout: 2023-07-14T19:10:38.990 INFO:journalctl@ceph.mon.b.smithi130.stdout: Uptime(secs): 0.0 total, 0.0 interval 2023-07-14T19:10:38.990 INFO:journalctl@ceph.mon.b.smithi130.stdout: Flush(GB): cumulative 0.000, interval 0.000 2023-07-14T19:10:38.990 INFO:journalctl@ceph.mon.b.smithi130.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2023-07-14T19:10:38.990 INFO:journalctl@ceph.mon.b.smithi130.stdout: AddFile(Total Files): cumulative 0, interval 0 2023-07-14T19:10:38.990 INFO:journalctl@ceph.mon.b.smithi130.stdout: AddFile(L0 Files): cumulative 0, interval 0 2023-07-14T19:10:38.990 INFO:journalctl@ceph.mon.b.smithi130.stdout: AddFile(Keys): cumulative 0, interval 0 2023-07-14T19:10:38.990 INFO:journalctl@ceph.mon.b.smithi130.stdout: Cumulative compaction: 0.00 GB write, 0.69 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-07-14T19:10:38.991 INFO:journalctl@ceph.mon.b.smithi130.stdout: Interval compaction: 0.00 GB write, 0.69 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-07-14T19:10:38.991 INFO:journalctl@ceph.mon.b.smithi130.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2023-07-14T19:10:38.991 INFO:journalctl@ceph.mon.b.smithi130.stdout: Block cache BinnedLRUCache@0x55f4476d91f0#7 capacity: 512.00 MB usage: 0.22 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 9e-06 secs_since: 0 2023-07-14T19:10:38.991 INFO:journalctl@ceph.mon.b.smithi130.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.11 KB,2.08616e-05%) IndexBlock(1,0.11 KB,2.08616e-05%) Misc(1,0.00 KB,0%) 2023-07-14T19:10:38.991 INFO:journalctl@ceph.mon.b.smithi130.stdout: 2023-07-14T19:10:38.991 INFO:journalctl@ceph.mon.b.smithi130.stdout: ** File Read Latency Histogram By Level [default] ** 2023-07-14T19:10:38.991 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:38 smithi130 ceph-mon[97832]: using public_addr v2:172.21.15.130:0/0 -> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] 2023-07-14T19:10:38.991 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:38 smithi130 ceph-mon[97832]: starting mon.b rank -1 at public addrs [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] at bind addrs [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] mon_data /var/lib/ceph/mon/ceph-b fsid adc52f04-2279-11ee-9b33-001a4aab830c 2023-07-14T19:10:38.992 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:38 smithi130 ceph-mon[97832]: mon.b@-1(???) e0 preinit fsid adc52f04-2279-11ee-9b33-001a4aab830c 2023-07-14T19:10:38.992 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:38 smithi130 ceph-mon[97832]: mon.b@-1(synchronizing).mds e1 new map 2023-07-14T19:10:38.992 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:38 smithi130 ceph-mon[97832]: mon.b@-1(synchronizing).mds e1 print_map 2023-07-14T19:10:38.992 INFO:journalctl@ceph.mon.b.smithi130.stdout: e1 2023-07-14T19:10:38.992 INFO:journalctl@ceph.mon.b.smithi130.stdout: enable_multiple, ever_enabled_multiple: 1,1 2023-07-14T19:10:38.992 INFO:journalctl@ceph.mon.b.smithi130.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2023-07-14T19:10:38.992 INFO:journalctl@ceph.mon.b.smithi130.stdout: legacy client fscid: -1 2023-07-14T19:10:38.993 INFO:journalctl@ceph.mon.b.smithi130.stdout: 2023-07-14T19:10:38.993 INFO:journalctl@ceph.mon.b.smithi130.stdout: No filesystems configured 2023-07-14T19:10:38.993 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:38 smithi130 ceph-mon[97832]: mon.b@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2023-07-14T19:10:38.993 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:38 smithi130 ceph-mon[97832]: mon.b@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2023-07-14T19:10:38.993 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:38 smithi130 ceph-mon[97832]: mon.b@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2023-07-14T19:10:38.993 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:38 smithi130 ceph-mon[97832]: mon.b@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2023-07-14T19:10:38.993 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:38 smithi130 ceph-mon[97832]: mon.b@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2023-07-14T19:10:38.993 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:38 smithi130 ceph-mon[97832]: mon.b@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2023-07-14T19:10:38.994 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:38 smithi130 ceph-mon[97832]: mon.b@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2023-07-14T19:10:38.994 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:38 smithi130 ceph-mon[97832]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2023-07-14T19:10:38.994 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:38 smithi130 ceph-mon[97832]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2023-07-14T19:10:38.994 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:38 smithi130 ceph-mon[97832]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2023-07-14T19:10:38.994 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:38 smithi130 ceph-mon[97832]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T19:10:38.994 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:38 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:10:38.994 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:38 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:10:38.994 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:38 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:10:38.995 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:38 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:10:38.995 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:38 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi130", "name": "osd_memory_target"}]: dispatch 2023-07-14T19:10:38.995 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:38 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:10:38.995 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:38 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T19:10:38.995 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:38 smithi130 ceph-mon[97832]: Updating smithi130:/etc/ceph/ceph.conf 2023-07-14T19:10:38.995 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:38 smithi130 ceph-mon[97832]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi191", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T19:10:38.995 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:38 smithi130 ceph-mon[97832]: Updating smithi130:/var/lib/ceph/adc52f04-2279-11ee-9b33-001a4aab830c/config/ceph.conf 2023-07-14T19:10:38.996 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:38 smithi130 ceph-mon[97832]: Updating smithi130:/etc/ceph/ceph.client.admin.keyring 2023-07-14T19:10:38.996 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:38 smithi130 ceph-mon[97832]: Deploying cephadm binary to smithi191 2023-07-14T19:10:38.996 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:38 smithi130 ceph-mon[97832]: Updating smithi130:/var/lib/ceph/adc52f04-2279-11ee-9b33-001a4aab830c/config/ceph.client.admin.keyring 2023-07-14T19:10:38.996 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:38 smithi130 ceph-mon[97832]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T19:10:38.996 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:38 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:10:38.996 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:38 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:10:38.996 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:38 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:10:38.996 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:38 smithi130 ceph-mon[97832]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T19:10:38.997 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:38 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:10:38.997 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:38 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T19:10:38.997 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:38 smithi130 ceph-mon[97832]: Added host smithi191 2023-07-14T19:10:38.997 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:38 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:10:38.997 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:38 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:10:38.997 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:38 smithi130 ceph-mon[97832]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T19:10:38.997 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:38 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:10:38.998 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:38 smithi130 ceph-mon[97832]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-07-14T19:10:38.998 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:38 smithi130 ceph-mon[97832]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T19:10:38.998 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:38 smithi130 ceph-mon[97832]: from='client.? 172.21.15.40:0/1079880469' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2023-07-14T19:10:38.998 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:38 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:10:38.998 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:38 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:10:38.998 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:38 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:10:38.998 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:38 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:10:38.998 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:38 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi191", "name": "osd_memory_target"}]: dispatch 2023-07-14T19:10:38.999 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:38 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:10:38.999 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:38 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T19:10:38.999 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:38 smithi130 ceph-mon[97832]: Updating smithi191:/etc/ceph/ceph.conf 2023-07-14T19:10:38.999 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:38 smithi130 ceph-mon[97832]: Updating smithi191:/var/lib/ceph/adc52f04-2279-11ee-9b33-001a4aab830c/config/ceph.conf 2023-07-14T19:10:38.999 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:38 smithi130 ceph-mon[97832]: from='client.? 172.21.15.40:0/1079880469' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2023-07-14T19:10:38.999 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:38 smithi130 ceph-mon[97832]: osdmap e4: 0 total, 0 up, 0 in 2023-07-14T19:10:38.999 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:38 smithi130 ceph-mon[97832]: Updating smithi191:/etc/ceph/ceph.client.admin.keyring 2023-07-14T19:10:38.999 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:38 smithi130 ceph-mon[97832]: Updating smithi191:/var/lib/ceph/adc52f04-2279-11ee-9b33-001a4aab830c/config/ceph.client.admin.keyring 2023-07-14T19:10:39.000 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:38 smithi130 ceph-mon[97832]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T19:10:39.000 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:38 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:10:39.000 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:38 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:10:39.000 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:38 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:10:39.000 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:38 smithi130 ceph-mon[97832]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi040:172.21.15.40=a;smithi130:172.21.15.130=b;smithi191:172.21.15.191=c", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T19:10:39.000 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:38 smithi130 ceph-mon[97832]: Saving service mon spec with placement smithi040:172.21.15.40=a;smithi130:172.21.15.130=b;smithi191:172.21.15.191=c;count:3 2023-07-14T19:10:39.000 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:38 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:10:39.001 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:38 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T19:10:39.001 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:38 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:10:39.001 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:38 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T19:10:39.001 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:38 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:10:39.001 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:38 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-07-14T19:10:39.001 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:38 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:10:39.001 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:38 smithi130 ceph-mon[97832]: Deploying daemon mon.c on smithi191 2023-07-14T19:10:39.001 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:38 smithi130 ceph-mon[97832]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T19:10:39.002 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:38 smithi130 ceph-mon[97832]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T19:10:39.002 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:38 smithi130 ceph-mon[97832]: from='client.? 172.21.15.191:0/855704549' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-07-14T19:10:39.002 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:38 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-07-14T19:10:39.002 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:38 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-14T19:10:39.002 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:38 smithi130 ceph-mon[97832]: mon.a calling monitor election 2023-07-14T19:10:39.002 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:38 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-14T19:10:39.002 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:38 smithi130 ceph-mon[97832]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T19:10:39.003 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:38 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-14T19:10:39.003 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:38 smithi130 ceph-mon[97832]: mon.c calling monitor election 2023-07-14T19:10:39.003 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:38 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-14T19:10:39.003 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:38 smithi130 ceph-mon[97832]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T19:10:39.003 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:38 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-14T19:10:39.003 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:38 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-14T19:10:39.003 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:38 smithi130 ceph-mon[97832]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2023-07-14T19:10:39.003 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:38 smithi130 ceph-mon[97832]: monmap e2: 2 mons at {a=[v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0],c=[v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0]} removed_ranks: {} 2023-07-14T19:10:39.004 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:38 smithi130 ceph-mon[97832]: fsmap 2023-07-14T19:10:39.004 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:38 smithi130 ceph-mon[97832]: osdmap e4: 0 total, 0 up, 0 in 2023-07-14T19:10:39.004 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:38 smithi130 ceph-mon[97832]: mgrmap e13: x(active, since 45s) 2023-07-14T19:10:39.004 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:38 smithi130 ceph-mon[97832]: overall HEALTH_OK 2023-07-14T19:10:39.004 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:38 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:10:39.004 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:38 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:10:39.004 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:38 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:10:39.004 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:38 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-07-14T19:10:39.005 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:38 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:10:39.005 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:38 smithi130 ceph-mon[97832]: Deploying daemon mon.b on smithi130 2023-07-14T19:10:39.005 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:38 smithi130 ceph-mon[97832]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T19:10:39.005 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:38 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-14T19:10:39.005 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:38 smithi130 ceph-mon[97832]: from='client.? 172.21.15.191:0/1022453596' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-07-14T19:10:39.005 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:38 smithi130 ceph-mon[97832]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T19:10:39.005 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:38 smithi130 ceph-mon[97832]: mon.b@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2023-07-14T19:10:39.006 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:38 smithi130 ceph-mon[97832]: mon.b@-1(synchronizing) e2 handle_conf_change mon_allow_pool_delete 2023-07-14T19:10:39.006 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:38 smithi130 ceph-mon[97832]: expand_channel_meta expand map: {default=false} 2023-07-14T19:10:39.006 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:38 smithi130 ceph-mon[97832]: expand_channel_meta from 'false' to 'false' 2023-07-14T19:10:39.006 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:38 smithi130 ceph-mon[97832]: expand_channel_meta expanded map: {default=false} 2023-07-14T19:10:39.006 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:38 smithi130 ceph-mon[97832]: expand_channel_meta expand map: {default=info} 2023-07-14T19:10:39.006 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:38 smithi130 ceph-mon[97832]: expand_channel_meta from 'info' to 'info' 2023-07-14T19:10:39.006 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:38 smithi130 ceph-mon[97832]: expand_channel_meta expanded map: {default=info} 2023-07-14T19:10:39.006 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:38 smithi130 ceph-mon[97832]: expand_channel_meta expand map: {default=daemon} 2023-07-14T19:10:39.007 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:38 smithi130 ceph-mon[97832]: expand_channel_meta from 'daemon' to 'daemon' 2023-07-14T19:10:39.007 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:38 smithi130 ceph-mon[97832]: expand_channel_meta expanded map: {default=daemon} 2023-07-14T19:10:39.007 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:38 smithi130 ceph-mon[97832]: expand_channel_meta expand map: {default=debug} 2023-07-14T19:10:39.007 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:38 smithi130 ceph-mon[97832]: expand_channel_meta from 'debug' to 'debug' 2023-07-14T19:10:39.007 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:38 smithi130 ceph-mon[97832]: expand_channel_meta expanded map: {default=debug} 2023-07-14T19:10:39.073 INFO:teuthology.orchestra.run.smithi191.stderr:Inferring config /var/lib/ceph/adc52f04-2279-11ee-9b33-001a4aab830c/mon.c/config 2023-07-14T19:10:39.221 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:10:38 smithi040 ceph-mon[96518]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T19:10:39.222 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:38 smithi191 ceph-mon[98311]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T19:10:39.242 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:38 smithi130 systemd[1]: Started Ceph mon.b for adc52f04-2279-11ee-9b33-001a4aab830c. 2023-07-14T19:10:44.472 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:10:44 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-07-14T19:10:44.472 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:10:44 smithi040 ceph-mon[96518]: mon.a calling monitor election 2023-07-14T19:10:44.472 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:10:44 smithi040 ceph-mon[96518]: mon.c calling monitor election 2023-07-14T19:10:44.472 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:10:44 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-14T19:10:44.472 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:10:44 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-14T19:10:44.473 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:10:44 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-14T19:10:44.473 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:10:44 smithi040 ceph-mon[96518]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T19:10:44.473 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:10:44 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-14T19:10:44.473 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:10:44 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-14T19:10:44.473 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:10:44 smithi040 ceph-mon[96518]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T19:10:44.473 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:10:44 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-14T19:10:44.473 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:10:44 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-14T19:10:44.474 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:10:44 smithi040 ceph-mon[96518]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2023-07-14T19:10:44.474 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:10:44 smithi040 ceph-mon[96518]: monmap e3: 3 mons at {a=[v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0],b=[v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0],c=[v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0]} removed_ranks: {} 2023-07-14T19:10:44.474 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:10:44 smithi040 ceph-mon[96518]: fsmap 2023-07-14T19:10:44.474 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:10:44 smithi040 ceph-mon[96518]: osdmap e4: 0 total, 0 up, 0 in 2023-07-14T19:10:44.474 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:10:44 smithi040 ceph-mon[96518]: mgrmap e13: x(active, since 53s) 2023-07-14T19:10:44.474 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:10:44 smithi040 ceph-mon[96518]: Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2023-07-14T19:10:44.474 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:10:44 smithi040 ceph-mon[96518]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2023-07-14T19:10:44.474 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:10:44 smithi040 ceph-mon[96518]: [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2023-07-14T19:10:44.475 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:10:44 smithi040 ceph-mon[96518]: mon.b (rank 2) addr [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] is down (out of quorum) 2023-07-14T19:10:44.475 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:10:44 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:10:44.475 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:10:44 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:10:44.475 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:10:44 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:10:44.475 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:10:44 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:10:44.476 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:44 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-07-14T19:10:44.476 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:44 smithi191 ceph-mon[98311]: mon.a calling monitor election 2023-07-14T19:10:44.476 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:44 smithi191 ceph-mon[98311]: mon.c calling monitor election 2023-07-14T19:10:44.476 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:44 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-14T19:10:44.476 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:44 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-14T19:10:44.476 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:44 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-14T19:10:44.476 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:44 smithi191 ceph-mon[98311]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T19:10:44.477 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:44 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-14T19:10:44.477 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:44 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-14T19:10:44.477 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:44 smithi191 ceph-mon[98311]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T19:10:44.477 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:44 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-14T19:10:44.477 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:44 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-14T19:10:44.477 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:44 smithi191 ceph-mon[98311]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2023-07-14T19:10:44.477 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:44 smithi191 ceph-mon[98311]: monmap e3: 3 mons at {a=[v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0],b=[v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0],c=[v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0]} removed_ranks: {} 2023-07-14T19:10:44.478 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:44 smithi191 ceph-mon[98311]: fsmap 2023-07-14T19:10:44.478 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:44 smithi191 ceph-mon[98311]: osdmap e4: 0 total, 0 up, 0 in 2023-07-14T19:10:44.478 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:44 smithi191 ceph-mon[98311]: mgrmap e13: x(active, since 53s) 2023-07-14T19:10:44.478 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:44 smithi191 ceph-mon[98311]: Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2023-07-14T19:10:44.478 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:44 smithi191 ceph-mon[98311]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2023-07-14T19:10:44.478 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:44 smithi191 ceph-mon[98311]: [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2023-07-14T19:10:44.478 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:44 smithi191 ceph-mon[98311]: mon.b (rank 2) addr [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] is down (out of quorum) 2023-07-14T19:10:44.478 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:44 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:10:44.479 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:44 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:10:44.479 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:44 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:10:44.479 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:44 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:10:44.659 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:10:44.656+0000 7f8366311700 1 -- 172.21.15.191:0/3509558118 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f834c0035c0 msgr2=0x7f834c005a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:10:44.659 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:10:44.656+0000 7f8366311700 1 --2- 172.21.15.191:0/3509558118 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f834c0035c0 0x7f834c005a40 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f8360075150 tx=0x7f8350010e70 comp rx=0 tx=0).stop 2023-07-14T19:10:44.659 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:10:44.656+0000 7f8366311700 1 -- 172.21.15.191:0/3509558118 shutdown_connections 2023-07-14T19:10:44.659 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:10:44.656+0000 7f8366311700 1 --2- 172.21.15.191:0/3509558118 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f834c0035c0 0x7f834c005a40 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:10:44.659 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:10:44.656+0000 7f8366311700 1 -- 172.21.15.191:0/3509558118 >> 172.21.15.191:0/3509558118 conn(0x7f8360073ea0 msgr2=0x7f83600742a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:10:44.659 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:10:44.656+0000 7f8366311700 1 -- 172.21.15.191:0/3509558118 shutdown_connections 2023-07-14T19:10:44.660 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:10:44.656+0000 7f8366311700 1 -- 172.21.15.191:0/3509558118 wait complete. 2023-07-14T19:10:44.661 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:10:44.657+0000 7f8366311700 1 Processor -- start 2023-07-14T19:10:44.661 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:10:44.657+0000 7f8366311700 1 -- start start 2023-07-14T19:10:44.662 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:10:44.657+0000 7f8366311700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f834c0035c0 0x7f8360194da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:10:44.662 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:10:44.657+0000 7f8366311700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f83601952e0 0x7f8360199750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:10:44.662 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:10:44.657+0000 7f8366311700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8360199c90 0x7f836019a130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:10:44.662 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:10:44.657+0000 7f8366311700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83601959b0 con 0x7f8360199c90 2023-07-14T19:10:44.662 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:10:44.657+0000 7f8366311700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83601a1a00 con 0x7f83601952e0 2023-07-14T19:10:44.662 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:10:44.657+0000 7f8366311700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83601a1b40 con 0x7f834c0035c0 2023-07-14T19:10:44.662 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:10:44.657+0000 7f835ffff700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f834c0035c0 0x7f8360194da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:10:44.663 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:10:44.657+0000 7f835ffff700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f834c0035c0 0x7f8360194da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.191:59232/0 (socket says 172.21.15.191:59232) 2023-07-14T19:10:44.663 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:10:44.657+0000 7f835ffff700 1 -- 172.21.15.191:0/2008133300 learned_addr learned my addr 172.21.15.191:0/2008133300 (peer_addr_for_me v2:172.21.15.191:0/0) 2023-07-14T19:10:44.663 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:10:44.658+0000 7f835ffff700 1 -- 172.21.15.191:0/2008133300 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f834c0035c0 msgr2=0x7f8360194da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2023-07-14T19:10:44.663 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:10:44.658+0000 7f835ffff700 1 -- 172.21.15.191:0/2008133300 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f834c0035c0 msgr2=0x7f8360194da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2023-07-14T19:10:44.663 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:10:44.658+0000 7f835ffff700 1 --2- 172.21.15.191:0/2008133300 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f834c0035c0 0x7f8360194da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-07-14T19:10:44.663 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:10:44.658+0000 7f835ffff700 1 --2- 172.21.15.191:0/2008133300 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f834c0035c0 0x7f8360194da0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-07-14T19:10:44.663 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:10:44.658+0000 7f835f7fe700 1 --2- 172.21.15.191:0/2008133300 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f83601952e0 0x7f8360199750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:10:44.664 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:10:44.658+0000 7f83648ae700 1 --2- 172.21.15.191:0/2008133300 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8360199c90 0x7f836019a130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:10:44.664 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:10:44.658+0000 7f835f7fe700 1 -- 172.21.15.191:0/2008133300 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f83601952e0 msgr2=0x7f8360199750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2023-07-14T19:10:44.664 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:10:44.658+0000 7f835f7fe700 1 -- 172.21.15.191:0/2008133300 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f83601952e0 msgr2=0x7f8360199750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2023-07-14T19:10:44.664 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:10:44.658+0000 7f835f7fe700 1 --2- 172.21.15.191:0/2008133300 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f83601952e0 0x7f8360199750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-07-14T19:10:44.664 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:10:44.658+0000 7f835f7fe700 1 --2- 172.21.15.191:0/2008133300 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f83601952e0 0x7f8360199750 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-07-14T19:10:44.664 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:10:44.658+0000 7f83648ae700 1 -- 172.21.15.191:0/2008133300 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f834c0035c0 msgr2=0x7f8360194da0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-07-14T19:10:44.664 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:10:44.658+0000 7f83648ae700 1 --2- 172.21.15.191:0/2008133300 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f834c0035c0 0x7f8360194da0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:10:44.664 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:10:44.658+0000 7f83648ae700 1 -- 172.21.15.191:0/2008133300 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f83601952e0 msgr2=0x7f8360199750 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-07-14T19:10:44.665 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:10:44.658+0000 7f83648ae700 1 --2- 172.21.15.191:0/2008133300 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f83601952e0 0x7f8360199750 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:10:44.665 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:10:44.658+0000 7f83648ae700 1 -- 172.21.15.191:0/2008133300 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83500108c0 con 0x7f8360199c90 2023-07-14T19:10:44.665 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:10:44.658+0000 7f83648ae700 1 --2- 172.21.15.191:0/2008133300 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8360199c90 0x7f836019a130 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f835800bd80 tx=0x7f835800be60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:10:44.665 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:10:44.658+0000 7f835d7fa700 1 -- 172.21.15.191:0/2008133300 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f835800cd10 con 0x7f8360199c90 2023-07-14T19:10:44.665 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:10:44.658+0000 7f8366311700 1 -- 172.21.15.191:0/2008133300 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f83601a1df0 con 0x7f8360199c90 2023-07-14T19:10:44.665 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:10:44.658+0000 7f835d7fa700 1 -- 172.21.15.191:0/2008133300 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f835800ce70 con 0x7f8360199c90 2023-07-14T19:10:44.665 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:10:44.658+0000 7f835d7fa700 1 -- 172.21.15.191:0/2008133300 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8358012880 con 0x7f8360199c90 2023-07-14T19:10:44.666 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:10:44.658+0000 7f8366311700 1 -- 172.21.15.191:0/2008133300 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f83601a22f0 con 0x7f8360199c90 2023-07-14T19:10:44.666 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:10:44.659+0000 7f835d7fa700 1 -- 172.21.15.191:0/2008133300 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 13) v1 ==== 45539+0+0 (secure 0 0 0) 0x7f8358012a00 con 0x7f8360199c90 2023-07-14T19:10:44.666 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:10:44.660+0000 7f835d7fa700 1 --2- 172.21.15.191:0/2008133300 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f8340038c30 0x7f834003b0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:10:44.667 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:10:44.660+0000 7f835d7fa700 1 -- 172.21.15.191:0/2008133300 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f8358014b00 con 0x7f8360199c90 2023-07-14T19:10:44.667 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:10:44.660+0000 7f8366311700 1 -- 172.21.15.191:0/2008133300 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f836018f180 con 0x7f8360199c90 2023-07-14T19:10:44.667 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:10:44.660+0000 7f835ffff700 1 --2- 172.21.15.191:0/2008133300 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f8340038c30 0x7f834003b0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:10:44.667 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:10:44.660+0000 7f835ffff700 1 --2- 172.21.15.191:0/2008133300 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f8340038c30 0x7f834003b0e0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f8350015040 tx=0x7f8350005a90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:10:44.667 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:10:44.665+0000 7f835d7fa700 1 -- 172.21.15.191:0/2008133300 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f8358019070 con 0x7f8360199c90 2023-07-14T19:10:44.952 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:10:44.949+0000 7f8366311700 1 -- 172.21.15.191:0/2008133300 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f836018eb60 con 0x7f8360199c90 2023-07-14T19:10:44.952 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:10:44.950+0000 7f835d7fa700 1 -- 172.21.15.191:0/2008133300 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 3 v3) v1 ==== 95+0+1274 (secure 0 0 0) 0x7f835801d3b0 con 0x7f8360199c90 2023-07-14T19:10:44.953 INFO:teuthology.orchestra.run.smithi191.stdout: 2023-07-14T19:10:44.953 INFO:teuthology.orchestra.run.smithi191.stdout:{"epoch":3,"fsid":"adc52f04-2279-11ee-9b33-001a4aab830c","modified":"2023-07-14T19:10:38.953239Z","created":"2023-07-14T19:08:46.682427Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:3300","nonce":0},{"type":"v1","addr":"172.21.15.40:6789","nonce":0}]},"addr":"172.21.15.40:6789/0","public_addr":"172.21.15.40:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:3300","nonce":0},{"type":"v1","addr":"172.21.15.191:6789","nonce":0}]},"addr":"172.21.15.191:6789/0","public_addr":"172.21.15.191:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:3300","nonce":0},{"type":"v1","addr":"172.21.15.130:6789","nonce":0}]},"addr":"172.21.15.130:6789/0","public_addr":"172.21.15.130:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2023-07-14T19:10:44.954 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:10:44.952+0000 7f8366311700 1 -- 172.21.15.191:0/2008133300 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f8340038c30 msgr2=0x7f834003b0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:10:44.955 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:10:44.952+0000 7f8366311700 1 --2- 172.21.15.191:0/2008133300 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f8340038c30 0x7f834003b0e0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f8350015040 tx=0x7f8350005a90 comp rx=0 tx=0).stop 2023-07-14T19:10:44.955 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:10:44.952+0000 7f8366311700 1 -- 172.21.15.191:0/2008133300 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8360199c90 msgr2=0x7f836019a130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:10:44.955 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:10:44.952+0000 7f8366311700 1 --2- 172.21.15.191:0/2008133300 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8360199c90 0x7f836019a130 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f835800bd80 tx=0x7f835800be60 comp rx=0 tx=0).stop 2023-07-14T19:10:44.955 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:10:44.952+0000 7f8366311700 1 -- 172.21.15.191:0/2008133300 shutdown_connections 2023-07-14T19:10:44.955 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:10:44.952+0000 7f8366311700 1 --2- 172.21.15.191:0/2008133300 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f8340038c30 0x7f834003b0e0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:10:44.955 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:10:44.952+0000 7f8366311700 1 --2- 172.21.15.191:0/2008133300 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f834c0035c0 0x7f8360194da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:10:44.956 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:10:44.952+0000 7f8366311700 1 --2- 172.21.15.191:0/2008133300 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8360199c90 0x7f836019a130 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:10:44.956 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:10:44.952+0000 7f8366311700 1 --2- 172.21.15.191:0/2008133300 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f83601952e0 0x7f8360199750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:10:44.956 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:10:44.952+0000 7f8366311700 1 -- 172.21.15.191:0/2008133300 >> 172.21.15.191:0/2008133300 conn(0x7f8360073ea0 msgr2=0x7f83600fd560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:10:44.956 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:10:44.952+0000 7f8366311700 1 -- 172.21.15.191:0/2008133300 shutdown_connections 2023-07-14T19:10:44.956 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:10:44.952+0000 7f8366311700 1 -- 172.21.15.191:0/2008133300 wait complete. 2023-07-14T19:10:44.956 INFO:teuthology.orchestra.run.smithi191.stderr:dumped monmap epoch 3 2023-07-14T19:10:45.401 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:45 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T19:10:45.401 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:45 smithi191 ceph-mon[98311]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T19:10:45.402 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:45 smithi191 ceph-mon[98311]: from='client.? 172.21.15.191:0/2008133300' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-07-14T19:10:45.402 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:45 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-14T19:10:45.428 INFO:tasks.cephadm:Generating final ceph.conf file... 2023-07-14T19:10:45.428 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid adc52f04-2279-11ee-9b33-001a4aab830c -- ceph config generate-minimal-conf 2023-07-14T19:10:45.461 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:10:45 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T19:10:45.462 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:10:45 smithi040 ceph-mon[96518]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T19:10:45.462 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:10:45 smithi040 ceph-mon[96518]: from='client.? 172.21.15.191:0/2008133300' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-07-14T19:10:45.462 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:10:45 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-14T19:10:46.009 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/adc52f04-2279-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T19:10:46.243 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:45 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-07-14T19:10:46.243 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:45 smithi130 ceph-mon[97832]: mon.a calling monitor election 2023-07-14T19:10:46.243 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:45 smithi130 ceph-mon[97832]: mon.c calling monitor election 2023-07-14T19:10:46.243 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:45 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-14T19:10:46.243 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:45 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-14T19:10:46.244 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:45 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-14T19:10:46.244 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:45 smithi130 ceph-mon[97832]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T19:10:46.244 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:45 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-14T19:10:46.244 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:45 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-14T19:10:46.244 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:45 smithi130 ceph-mon[97832]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T19:10:46.244 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:45 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-14T19:10:46.244 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:45 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-14T19:10:46.245 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:45 smithi130 ceph-mon[97832]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2023-07-14T19:10:46.245 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:45 smithi130 ceph-mon[97832]: monmap e3: 3 mons at {a=[v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0],b=[v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0],c=[v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0]} removed_ranks: {} 2023-07-14T19:10:46.246 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:45 smithi130 ceph-mon[97832]: fsmap 2023-07-14T19:10:46.246 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:45 smithi130 ceph-mon[97832]: osdmap e4: 0 total, 0 up, 0 in 2023-07-14T19:10:46.246 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:45 smithi130 ceph-mon[97832]: mgrmap e13: x(active, since 53s) 2023-07-14T19:10:46.246 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:45 smithi130 ceph-mon[97832]: Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2023-07-14T19:10:46.246 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:45 smithi130 ceph-mon[97832]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2023-07-14T19:10:46.246 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:45 smithi130 ceph-mon[97832]: [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2023-07-14T19:10:46.247 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:45 smithi130 ceph-mon[97832]: mon.b (rank 2) addr [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] is down (out of quorum) 2023-07-14T19:10:46.247 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:45 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:10:46.247 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:45 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:10:46.247 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:45 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:10:46.247 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:45 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:10:46.247 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:45 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T19:10:46.247 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:45 smithi130 ceph-mon[97832]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T19:10:46.248 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:45 smithi130 ceph-mon[97832]: from='client.? 172.21.15.191:0/2008133300' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-07-14T19:10:46.248 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:45 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-14T19:10:46.972 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:10:46 smithi040 ceph-mon[96518]: mon.b calling monitor election 2023-07-14T19:10:46.972 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:10:46 smithi040 ceph-mon[96518]: Updating smithi040:/etc/ceph/ceph.conf 2023-07-14T19:10:46.972 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:10:46 smithi040 ceph-mon[96518]: Updating smithi130:/etc/ceph/ceph.conf 2023-07-14T19:10:46.972 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:10:46 smithi040 ceph-mon[96518]: Updating smithi191:/etc/ceph/ceph.conf 2023-07-14T19:10:46.972 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:10:46 smithi040 ceph-mon[96518]: mon.b calling monitor election 2023-07-14T19:10:46.973 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:10:46 smithi040 ceph-mon[96518]: mon.c calling monitor election 2023-07-14T19:10:46.973 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:10:46 smithi040 ceph-mon[96518]: mon.a calling monitor election 2023-07-14T19:10:46.973 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:10:46 smithi040 ceph-mon[96518]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2023-07-14T19:10:46.973 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:10:46 smithi040 ceph-mon[96518]: monmap e3: 3 mons at {a=[v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0],b=[v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0],c=[v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0]} removed_ranks: {} 2023-07-14T19:10:46.973 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:10:46 smithi040 ceph-mon[96518]: fsmap 2023-07-14T19:10:46.973 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:10:46 smithi040 ceph-mon[96518]: osdmap e4: 0 total, 0 up, 0 in 2023-07-14T19:10:46.973 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:10:46 smithi040 ceph-mon[96518]: mgrmap e13: x(active, since 55s) 2023-07-14T19:10:46.974 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:10:46 smithi040 ceph-mon[96518]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2023-07-14T19:10:46.974 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:10:46 smithi040 ceph-mon[96518]: Cluster is now healthy 2023-07-14T19:10:46.974 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:10:46 smithi040 ceph-mon[96518]: overall HEALTH_OK 2023-07-14T19:10:46.974 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:10:46 smithi040 ceph-mon[96518]: Updating smithi130:/var/lib/ceph/adc52f04-2279-11ee-9b33-001a4aab830c/config/ceph.conf 2023-07-14T19:10:46.974 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:10:46 smithi040 ceph-mon[96518]: Updating smithi191:/var/lib/ceph/adc52f04-2279-11ee-9b33-001a4aab830c/config/ceph.conf 2023-07-14T19:10:46.974 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:10:46 smithi040 ceph-mon[96518]: Updating smithi040:/var/lib/ceph/adc52f04-2279-11ee-9b33-001a4aab830c/config/ceph.conf 2023-07-14T19:10:46.974 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:10:46 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:10:46.974 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:10:46 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:10:46.975 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:10:46 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:10:46.975 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:10:46 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:10:46.975 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:10:46 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:10:46.975 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:10:46 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:10:46.975 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:10:46 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:10:46.975 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:10:46 smithi040 ceph-mon[96518]: Reconfiguring mon.a (unknown last config time)... 2023-07-14T19:10:46.975 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:10:46 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-07-14T19:10:46.976 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:10:46 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2023-07-14T19:10:46.976 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:10:46 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:10:46.976 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:10:46 smithi040 ceph-mon[96518]: Reconfiguring daemon mon.a on smithi040 2023-07-14T19:10:46.976 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:10:46 smithi040 ceph-mon[96518]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T19:10:47.222 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:46 smithi191 ceph-mon[98311]: mon.b calling monitor election 2023-07-14T19:10:47.223 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:46 smithi191 ceph-mon[98311]: Updating smithi040:/etc/ceph/ceph.conf 2023-07-14T19:10:47.223 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:46 smithi191 ceph-mon[98311]: Updating smithi130:/etc/ceph/ceph.conf 2023-07-14T19:10:47.223 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:46 smithi191 ceph-mon[98311]: Updating smithi191:/etc/ceph/ceph.conf 2023-07-14T19:10:47.223 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:46 smithi191 ceph-mon[98311]: mon.b calling monitor election 2023-07-14T19:10:47.223 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:46 smithi191 ceph-mon[98311]: mon.c calling monitor election 2023-07-14T19:10:47.223 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:46 smithi191 ceph-mon[98311]: mon.a calling monitor election 2023-07-14T19:10:47.224 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:46 smithi191 ceph-mon[98311]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2023-07-14T19:10:47.224 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:46 smithi191 ceph-mon[98311]: monmap e3: 3 mons at {a=[v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0],b=[v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0],c=[v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0]} removed_ranks: {} 2023-07-14T19:10:47.224 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:46 smithi191 ceph-mon[98311]: fsmap 2023-07-14T19:10:47.224 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:46 smithi191 ceph-mon[98311]: osdmap e4: 0 total, 0 up, 0 in 2023-07-14T19:10:47.224 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:46 smithi191 ceph-mon[98311]: mgrmap e13: x(active, since 55s) 2023-07-14T19:10:47.224 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:46 smithi191 ceph-mon[98311]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2023-07-14T19:10:47.224 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:46 smithi191 ceph-mon[98311]: Cluster is now healthy 2023-07-14T19:10:47.225 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:46 smithi191 ceph-mon[98311]: overall HEALTH_OK 2023-07-14T19:10:47.225 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:46 smithi191 ceph-mon[98311]: Updating smithi130:/var/lib/ceph/adc52f04-2279-11ee-9b33-001a4aab830c/config/ceph.conf 2023-07-14T19:10:47.225 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:46 smithi191 ceph-mon[98311]: Updating smithi191:/var/lib/ceph/adc52f04-2279-11ee-9b33-001a4aab830c/config/ceph.conf 2023-07-14T19:10:47.225 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:46 smithi191 ceph-mon[98311]: Updating smithi040:/var/lib/ceph/adc52f04-2279-11ee-9b33-001a4aab830c/config/ceph.conf 2023-07-14T19:10:47.225 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:46 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:10:47.225 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:46 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:10:47.225 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:46 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:10:47.225 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:46 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:10:47.226 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:46 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:10:47.226 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:46 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:10:47.226 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:46 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:10:47.226 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:46 smithi191 ceph-mon[98311]: Reconfiguring mon.a (unknown last config time)... 2023-07-14T19:10:47.226 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:46 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-07-14T19:10:47.226 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:46 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2023-07-14T19:10:47.226 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:46 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:10:47.227 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:46 smithi191 ceph-mon[98311]: Reconfiguring daemon mon.a on smithi040 2023-07-14T19:10:47.227 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:46 smithi191 ceph-mon[98311]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T19:10:47.243 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:46 smithi130 ceph-mon[97832]: mon.b calling monitor election 2023-07-14T19:10:47.243 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:46 smithi130 ceph-mon[97832]: Updating smithi040:/etc/ceph/ceph.conf 2023-07-14T19:10:47.243 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:46 smithi130 ceph-mon[97832]: Updating smithi130:/etc/ceph/ceph.conf 2023-07-14T19:10:47.243 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:46 smithi130 ceph-mon[97832]: Updating smithi191:/etc/ceph/ceph.conf 2023-07-14T19:10:47.243 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:46 smithi130 ceph-mon[97832]: mon.b calling monitor election 2023-07-14T19:10:47.244 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:46 smithi130 ceph-mon[97832]: mon.c calling monitor election 2023-07-14T19:10:47.244 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:46 smithi130 ceph-mon[97832]: mon.a calling monitor election 2023-07-14T19:10:47.244 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:46 smithi130 ceph-mon[97832]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2023-07-14T19:10:47.244 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:46 smithi130 ceph-mon[97832]: monmap e3: 3 mons at {a=[v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0],b=[v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0],c=[v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0]} removed_ranks: {} 2023-07-14T19:10:47.244 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:46 smithi130 ceph-mon[97832]: fsmap 2023-07-14T19:10:47.244 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:46 smithi130 ceph-mon[97832]: osdmap e4: 0 total, 0 up, 0 in 2023-07-14T19:10:47.244 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:46 smithi130 ceph-mon[97832]: mgrmap e13: x(active, since 55s) 2023-07-14T19:10:47.245 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:46 smithi130 ceph-mon[97832]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2023-07-14T19:10:47.245 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:46 smithi130 ceph-mon[97832]: Cluster is now healthy 2023-07-14T19:10:47.245 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:46 smithi130 ceph-mon[97832]: overall HEALTH_OK 2023-07-14T19:10:47.245 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:46 smithi130 ceph-mon[97832]: Updating smithi130:/var/lib/ceph/adc52f04-2279-11ee-9b33-001a4aab830c/config/ceph.conf 2023-07-14T19:10:47.245 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:46 smithi130 ceph-mon[97832]: Updating smithi191:/var/lib/ceph/adc52f04-2279-11ee-9b33-001a4aab830c/config/ceph.conf 2023-07-14T19:10:47.245 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:46 smithi130 ceph-mon[97832]: Updating smithi040:/var/lib/ceph/adc52f04-2279-11ee-9b33-001a4aab830c/config/ceph.conf 2023-07-14T19:10:47.245 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:46 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:10:47.245 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:46 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:10:47.246 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:46 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:10:47.246 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:46 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:10:47.246 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:46 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:10:47.246 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:46 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:10:47.246 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:46 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:10:47.246 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:46 smithi130 ceph-mon[97832]: Reconfiguring mon.a (unknown last config time)... 2023-07-14T19:10:47.246 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:46 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-07-14T19:10:47.246 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:46 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2023-07-14T19:10:47.247 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:46 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:10:47.247 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:46 smithi130 ceph-mon[97832]: Reconfiguring daemon mon.a on smithi040 2023-07-14T19:10:47.247 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:46 smithi130 ceph-mon[97832]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T19:10:47.422 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:10:47.392+0000 7f224f920700 1 -- 172.21.15.40:0/288088330 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f22481057c0 msgr2=0x7f2248105b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:10:47.423 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:10:47.392+0000 7f224f920700 1 --2- 172.21.15.40:0/288088330 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f22481057c0 0x7f2248105b90 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f2244009910 tx=0x7f2244009c20 comp rx=0 tx=0).stop 2023-07-14T19:10:47.423 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:10:47.392+0000 7f224f920700 1 -- 172.21.15.40:0/288088330 shutdown_connections 2023-07-14T19:10:47.423 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:10:47.392+0000 7f224f920700 1 --2- 172.21.15.40:0/288088330 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f22481057c0 0x7f2248105b90 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:10:47.423 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:10:47.392+0000 7f224f920700 1 -- 172.21.15.40:0/288088330 >> 172.21.15.40:0/288088330 conn(0x7f22480743e0 msgr2=0x7f22480747e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:10:47.423 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:10:47.393+0000 7f224f920700 1 -- 172.21.15.40:0/288088330 shutdown_connections 2023-07-14T19:10:47.424 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:10:47.393+0000 7f224f920700 1 -- 172.21.15.40:0/288088330 wait complete. 2023-07-14T19:10:47.424 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:10:47.393+0000 7f224f920700 1 Processor -- start 2023-07-14T19:10:47.424 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:10:47.393+0000 7f224f920700 1 -- start start 2023-07-14T19:10:47.424 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:10:47.394+0000 7f224f920700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f22481057c0 0x7f2248194d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:10:47.424 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:10:47.394+0000 7f224f920700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f22481952a0 0x7f2248199710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:10:47.425 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:10:47.394+0000 7f224f920700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2248199c50 0x7f224819a0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:10:47.425 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:10:47.394+0000 7f224f920700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2248195970 con 0x7f2248199c50 2023-07-14T19:10:47.425 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:10:47.394+0000 7f224f920700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22481a19c0 con 0x7f22481952a0 2023-07-14T19:10:47.426 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:10:47.394+0000 7f224f920700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22481a1b00 con 0x7f22481057c0 2023-07-14T19:10:47.426 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:10:47.394+0000 7f224debd700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2248199c50 0x7f224819a0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:10:47.426 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:10:47.394+0000 7f224debd700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2248199c50 0x7f224819a0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:60180/0 (socket says 172.21.15.40:60180) 2023-07-14T19:10:47.426 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:10:47.394+0000 7f224debd700 1 -- 172.21.15.40:0/1372780812 learned_addr learned my addr 172.21.15.40:0/1372780812 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-07-14T19:10:47.427 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:10:47.394+0000 7f224cebb700 1 --2- 172.21.15.40:0/1372780812 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f22481952a0 0x7f2248199710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:10:47.427 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:10:47.394+0000 7f224d6bc700 1 --2- 172.21.15.40:0/1372780812 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f22481057c0 0x7f2248194d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:10:47.427 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:10:47.394+0000 7f224debd700 1 -- 172.21.15.40:0/1372780812 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f22481057c0 msgr2=0x7f2248194d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:10:47.428 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:10:47.394+0000 7f224debd700 1 --2- 172.21.15.40:0/1372780812 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f22481057c0 0x7f2248194d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:10:47.428 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:10:47.394+0000 7f224debd700 1 -- 172.21.15.40:0/1372780812 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f22481952a0 msgr2=0x7f2248199710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:10:47.428 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:10:47.394+0000 7f224debd700 1 --2- 172.21.15.40:0/1372780812 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f22481952a0 0x7f2248199710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:10:47.429 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:10:47.394+0000 7f224debd700 1 -- 172.21.15.40:0/1372780812 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22440095f0 con 0x7f2248199c50 2023-07-14T19:10:47.429 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:10:47.394+0000 7f224debd700 1 --2- 172.21.15.40:0/1372780812 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2248199c50 0x7f224819a0f0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f224000ba80 tx=0x7f224000bd90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:10:47.429 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:10:47.395+0000 7f223e7fc700 1 -- 172.21.15.40:0/1372780812 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f224000c9b0 con 0x7f2248199c50 2023-07-14T19:10:47.430 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:10:47.395+0000 7f223e7fc700 1 -- 172.21.15.40:0/1372780812 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f2240014440 con 0x7f2248199c50 2023-07-14T19:10:47.430 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:10:47.395+0000 7f224f920700 1 -- 172.21.15.40:0/1372780812 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f22481a1db0 con 0x7f2248199c50 2023-07-14T19:10:47.430 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:10:47.395+0000 7f223e7fc700 1 -- 172.21.15.40:0/1372780812 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2240007590 con 0x7f2248199c50 2023-07-14T19:10:47.430 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:10:47.395+0000 7f224f920700 1 -- 172.21.15.40:0/1372780812 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f22481a2280 con 0x7f2248199c50 2023-07-14T19:10:47.431 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:10:47.396+0000 7f223e7fc700 1 -- 172.21.15.40:0/1372780812 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 13) v1 ==== 45539+0+0 (secure 0 0 0) 0x7f22400077b0 con 0x7f2248199c50 2023-07-14T19:10:47.431 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:10:47.396+0000 7f224f920700 1 -- 172.21.15.40:0/1372780812 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2248061c70 con 0x7f2248199c50 2023-07-14T19:10:47.431 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:10:47.396+0000 7f223e7fc700 1 --2- 172.21.15.40:0/1372780812 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f2234038c30 0x7f223403b0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:10:47.432 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:10:47.396+0000 7f223e7fc700 1 -- 172.21.15.40:0/1372780812 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f224004ab60 con 0x7f2248199c50 2023-07-14T19:10:47.432 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:10:47.396+0000 7f224d6bc700 1 --2- 172.21.15.40:0/1372780812 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f2234038c30 0x7f223403b0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:10:47.432 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:10:47.396+0000 7f224d6bc700 1 --2- 172.21.15.40:0/1372780812 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f2234038c30 0x7f223403b0e0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f224400b5c0 tx=0x7f2244005da0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:10:47.432 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:10:47.401+0000 7f223e7fc700 1 -- 172.21.15.40:0/1372780812 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f224001b8b0 con 0x7f2248199c50 2023-07-14T19:10:47.615 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:10:47.613+0000 7f224f920700 1 -- 172.21.15.40:0/1372780812 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) v1 -- 0x7f22481960c0 con 0x7f2248199c50 2023-07-14T19:10:47.616 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:10:47.613+0000 7f223e7fc700 1 -- 172.21.15.40:0/1372780812 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v9) v1 ==== 76+0+275 (secure 0 0 0) 0x7f2240012340 con 0x7f2248199c50 2023-07-14T19:10:47.616 INFO:teuthology.orchestra.run.smithi040.stdout:# minimal ceph.conf for adc52f04-2279-11ee-9b33-001a4aab830c 2023-07-14T19:10:47.616 INFO:teuthology.orchestra.run.smithi040.stdout:[global] 2023-07-14T19:10:47.617 INFO:teuthology.orchestra.run.smithi040.stdout: fsid = adc52f04-2279-11ee-9b33-001a4aab830c 2023-07-14T19:10:47.617 INFO:teuthology.orchestra.run.smithi040.stdout: mon_host = [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] 2023-07-14T19:10:47.618 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:10:47.615+0000 7f224f920700 1 -- 172.21.15.40:0/1372780812 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f2234038c30 msgr2=0x7f223403b0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:10:47.618 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:10:47.615+0000 7f224f920700 1 --2- 172.21.15.40:0/1372780812 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f2234038c30 0x7f223403b0e0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f224400b5c0 tx=0x7f2244005da0 comp rx=0 tx=0).stop 2023-07-14T19:10:47.618 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:10:47.615+0000 7f224f920700 1 -- 172.21.15.40:0/1372780812 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2248199c50 msgr2=0x7f224819a0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:10:47.618 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:10:47.615+0000 7f224f920700 1 --2- 172.21.15.40:0/1372780812 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2248199c50 0x7f224819a0f0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f224000ba80 tx=0x7f224000bd90 comp rx=0 tx=0).stop 2023-07-14T19:10:47.619 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:10:47.616+0000 7f224f920700 1 -- 172.21.15.40:0/1372780812 shutdown_connections 2023-07-14T19:10:47.619 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:10:47.616+0000 7f224f920700 1 --2- 172.21.15.40:0/1372780812 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f2234038c30 0x7f223403b0e0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:10:47.619 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:10:47.616+0000 7f224f920700 1 --2- 172.21.15.40:0/1372780812 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f22481057c0 0x7f2248194d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:10:47.619 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:10:47.616+0000 7f224f920700 1 --2- 172.21.15.40:0/1372780812 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2248199c50 0x7f224819a0f0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:10:47.619 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:10:47.616+0000 7f224f920700 1 --2- 172.21.15.40:0/1372780812 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f22481952a0 0x7f2248199710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:10:47.619 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:10:47.616+0000 7f224f920700 1 -- 172.21.15.40:0/1372780812 >> 172.21.15.40:0/1372780812 conn(0x7f22480743e0 msgr2=0x7f2248107020 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:10:47.619 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:10:47.616+0000 7f224f920700 1 -- 172.21.15.40:0/1372780812 shutdown_connections 2023-07-14T19:10:47.620 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:10:47.616+0000 7f224f920700 1 -- 172.21.15.40:0/1372780812 wait complete. 2023-07-14T19:10:48.221 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:10:47 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-14T19:10:48.222 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:10:47 smithi040 ceph-mon[96518]: from='client.? 172.21.15.40:0/1372780812' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:10:48.222 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:47 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-14T19:10:48.222 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:47 smithi191 ceph-mon[98311]: from='client.? 172.21.15.40:0/1372780812' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:10:48.243 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:47 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-14T19:10:48.243 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:47 smithi130 ceph-mon[97832]: from='client.? 172.21.15.40:0/1372780812' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:10:48.262 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2023-07-14T19:10:48.262 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2023-07-14T19:10:48.262 DEBUG:teuthology.orchestra.run.smithi040:> sudo dd of=/etc/ceph/ceph.conf 2023-07-14T19:10:48.334 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2023-07-14T19:10:48.335 DEBUG:teuthology.orchestra.run.smithi040:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2023-07-14T19:10:48.402 DEBUG:teuthology.orchestra.run.smithi130:> set -ex 2023-07-14T19:10:48.402 DEBUG:teuthology.orchestra.run.smithi130:> sudo dd of=/etc/ceph/ceph.conf 2023-07-14T19:10:48.436 DEBUG:teuthology.orchestra.run.smithi130:> set -ex 2023-07-14T19:10:48.437 DEBUG:teuthology.orchestra.run.smithi130:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2023-07-14T19:10:48.503 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2023-07-14T19:10:48.504 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd of=/etc/ceph/ceph.conf 2023-07-14T19:10:48.540 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2023-07-14T19:10:48.540 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2023-07-14T19:10:48.612 INFO:tasks.cephadm:Adding mgr.x on smithi040 2023-07-14T19:10:48.612 INFO:tasks.cephadm:Adding mgr.y on smithi130 2023-07-14T19:10:48.613 INFO:tasks.cephadm:Adding mgr.z on smithi191 2023-07-14T19:10:48.613 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid adc52f04-2279-11ee-9b33-001a4aab830c -- ceph orch apply mgr '3;smithi040=x;smithi130=y;smithi191=z' 2023-07-14T19:10:49.205 INFO:teuthology.orchestra.run.smithi191.stderr:Inferring config /var/lib/ceph/adc52f04-2279-11ee-9b33-001a4aab830c/mon.c/config 2023-07-14T19:10:49.644 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:49 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:10:49.644 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:49 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:10:49.645 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:49 smithi191 ceph-mon[98311]: Reconfiguring mon.b (monmap changed)... 2023-07-14T19:10:49.645 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:49 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-07-14T19:10:49.645 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:49 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2023-07-14T19:10:49.645 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:49 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:10:49.645 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:49 smithi191 ceph-mon[98311]: Reconfiguring daemon mon.b on smithi130 2023-07-14T19:10:49.645 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:49 smithi191 ceph-mon[98311]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T19:10:49.721 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:10:49 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:10:49.722 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:10:49 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:10:49.722 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:10:49 smithi040 ceph-mon[96518]: Reconfiguring mon.b (monmap changed)... 2023-07-14T19:10:49.722 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:10:49 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-07-14T19:10:49.722 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:10:49 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2023-07-14T19:10:49.722 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:10:49 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:10:49.723 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:10:49 smithi040 ceph-mon[96518]: Reconfiguring daemon mon.b on smithi130 2023-07-14T19:10:49.723 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:10:49 smithi040 ceph-mon[96518]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T19:10:49.743 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:49 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:10:49.743 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:49 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:10:49.743 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:49 smithi130 ceph-mon[97832]: Reconfiguring mon.b (monmap changed)... 2023-07-14T19:10:49.743 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:49 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-07-14T19:10:49.743 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:49 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2023-07-14T19:10:49.744 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:49 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:10:49.744 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:49 smithi130 ceph-mon[97832]: Reconfiguring daemon mon.b on smithi130 2023-07-14T19:10:49.744 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:49 smithi130 ceph-mon[97832]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T19:10:50.722 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:50 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:10:50.723 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:50 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:10:50.723 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:50 smithi191 ceph-mon[98311]: Reconfiguring mon.c (monmap changed)... 2023-07-14T19:10:50.723 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:50 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-07-14T19:10:50.723 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:50 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2023-07-14T19:10:50.723 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:50 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:10:50.723 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:50 smithi191 ceph-mon[98311]: Reconfiguring daemon mon.c on smithi191 2023-07-14T19:10:50.775 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:10:50.773+0000 7f82e0f97700 1 -- 172.21.15.191:0/3568841244 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f82dc0fe730 msgr2=0x7f82dc0feb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:10:50.776 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:10:50.773+0000 7f82e0f97700 1 --2- 172.21.15.191:0/3568841244 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f82dc0fe730 0x7f82dc0feb40 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f82c4009910 tx=0x7f82c4009c20 comp rx=0 tx=0).stop 2023-07-14T19:10:50.776 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:10:50.773+0000 7f82e0f97700 1 -- 172.21.15.191:0/3568841244 shutdown_connections 2023-07-14T19:10:50.776 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:10:50.773+0000 7f82e0f97700 1 --2- 172.21.15.191:0/3568841244 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f82dc0fe730 0x7f82dc0feb40 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:10:50.776 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:10:50.773+0000 7f82e0f97700 1 -- 172.21.15.191:0/3568841244 >> 172.21.15.191:0/3568841244 conn(0x7f82dc0f9ce0 msgr2=0x7f82dc0fc110 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:10:50.776 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:10:50.773+0000 7f82e0f97700 1 -- 172.21.15.191:0/3568841244 shutdown_connections 2023-07-14T19:10:50.776 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:10:50.773+0000 7f82e0f97700 1 -- 172.21.15.191:0/3568841244 wait complete. 2023-07-14T19:10:50.778 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:10:50.774+0000 7f82e0f97700 1 Processor -- start 2023-07-14T19:10:50.778 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:10:50.774+0000 7f82e0f97700 1 -- start start 2023-07-14T19:10:50.778 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:10:50.774+0000 7f82e0f97700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f82dc0fe730 0x7f82dc196d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:10:50.778 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:10:50.774+0000 7f82e0f97700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f82dc197260 0x7f82dc19c290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:10:50.779 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:10:50.774+0000 7f82e0f97700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f82dc1976d0 0x7f82dc19e820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:10:50.779 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:10:50.774+0000 7f82e0f97700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82dc19eff0 con 0x7f82dc1976d0 2023-07-14T19:10:50.779 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:10:50.774+0000 7f82e0f97700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82dc19f160 con 0x7f82dc197260 2023-07-14T19:10:50.779 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:10:50.774+0000 7f82e0f97700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82dc19f320 con 0x7f82dc0fe730 2023-07-14T19:10:50.779 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:10:50.774+0000 7f82da59c700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f82dc0fe730 0x7f82dc196d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:10:50.779 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:10:50.774+0000 7f82da59c700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f82dc0fe730 0x7f82dc196d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.191:59242/0 (socket says 172.21.15.191:59242) 2023-07-14T19:10:50.779 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:10:50.774+0000 7f82da59c700 1 -- 172.21.15.191:0/2289972291 learned_addr learned my addr 172.21.15.191:0/2289972291 (peer_addr_for_me v2:172.21.15.191:0/0) 2023-07-14T19:10:50.780 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:10:50.774+0000 7f82dad9d700 1 --2- 172.21.15.191:0/2289972291 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f82dc1976d0 0x7f82dc19e820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:10:50.780 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:10:50.774+0000 7f82d9d9b700 1 --2- 172.21.15.191:0/2289972291 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f82dc197260 0x7f82dc19c290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:10:50.780 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:10:50.774+0000 7f82da59c700 1 -- 172.21.15.191:0/2289972291 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f82dc197260 msgr2=0x7f82dc19c290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:10:50.780 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:10:50.774+0000 7f82da59c700 1 --2- 172.21.15.191:0/2289972291 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f82dc197260 0x7f82dc19c290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:10:50.780 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:10:50.774+0000 7f82da59c700 1 -- 172.21.15.191:0/2289972291 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f82dc1976d0 msgr2=0x7f82dc19e820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:10:50.780 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:10:50.774+0000 7f82da59c700 1 --2- 172.21.15.191:0/2289972291 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f82dc1976d0 0x7f82dc19e820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:10:50.780 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:10:50.774+0000 7f82da59c700 1 -- 172.21.15.191:0/2289972291 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82c40095f0 con 0x7f82dc0fe730 2023-07-14T19:10:50.781 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:10:50.774+0000 7f82da59c700 1 --2- 172.21.15.191:0/2289972291 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f82dc0fe730 0x7f82dc196d20 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f82c4006010 tx=0x7f82c4004cb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:10:50.781 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:10:50.775+0000 7f82d37fe700 1 -- 172.21.15.191:0/2289972291 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82c401d070 con 0x7f82dc0fe730 2023-07-14T19:10:50.781 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:10:50.775+0000 7f82e0f97700 1 -- 172.21.15.191:0/2289972291 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f82dc19f5a0 con 0x7f82dc0fe730 2023-07-14T19:10:50.781 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:10:50.775+0000 7f82d37fe700 1 -- 172.21.15.191:0/2289972291 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f82c400be20 con 0x7f82dc0fe730 2023-07-14T19:10:50.781 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:10:50.775+0000 7f82d37fe700 1 -- 172.21.15.191:0/2289972291 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82c400f460 con 0x7f82dc0fe730 2023-07-14T19:10:50.781 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:10:50.775+0000 7f82e0f97700 1 -- 172.21.15.191:0/2289972291 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f82dc19fb10 con 0x7f82dc0fe730 2023-07-14T19:10:50.781 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:10:50.775+0000 7f82d37fe700 1 -- 172.21.15.191:0/2289972291 <== mon.1 v2:172.21.15.191:3300/0 4 ==== mgrmap(e 13) v1 ==== 45539+0+0 (secure 0 0 0) 0x7f82c400f5c0 con 0x7f82dc0fe730 2023-07-14T19:10:50.781 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:10:50.776+0000 7f82d37fe700 1 --2- 172.21.15.191:0/2289972291 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f82c8038c80 0x7f82c803b130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:10:50.782 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:10:50.776+0000 7f82d37fe700 1 -- 172.21.15.191:0/2289972291 <== mon.1 v2:172.21.15.191:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f82c404d770 con 0x7f82dc0fe730 2023-07-14T19:10:50.782 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:10:50.776+0000 7f82e0f97700 1 -- 172.21.15.191:0/2289972291 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f82dc190e20 con 0x7f82dc0fe730 2023-07-14T19:10:50.782 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:10:50.776+0000 7f82d9d9b700 1 --2- 172.21.15.191:0/2289972291 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f82c8038c80 0x7f82c803b130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:10:50.782 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:10:50.776+0000 7f82d9d9b700 1 --2- 172.21.15.191:0/2289972291 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f82c8038c80 0x7f82c803b130 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f82cc009850 tx=0x7f82cc008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:10:50.783 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:10:50.781+0000 7f82d37fe700 1 -- 172.21.15.191:0/2289972291 <== mon.1 v2:172.21.15.191:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f82c4020960 con 0x7f82dc0fe730 2023-07-14T19:10:50.972 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:10:50 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:10:50.972 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:10:50 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:10:50.972 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:10:50 smithi040 ceph-mon[96518]: Reconfiguring mon.c (monmap changed)... 2023-07-14T19:10:50.972 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:10:50 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-07-14T19:10:50.972 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:10:50 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2023-07-14T19:10:50.972 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:10:50 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:10:50.973 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:10:50 smithi040 ceph-mon[96518]: Reconfiguring daemon mon.c on smithi191 2023-07-14T19:10:50.986 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:10:50.984+0000 7f82e0f97700 1 -- 172.21.15.191:0/2289972291 --> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi040=x;smithi130=y;smithi191=z", "target": ["mon-mgr", ""]}) v1 -- 0x7f82dc060a80 con 0x7f82c8038c80 2023-07-14T19:10:50.992 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:50 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:10:50.993 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:50 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:10:50.993 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:50 smithi130 ceph-mon[97832]: Reconfiguring mon.c (monmap changed)... 2023-07-14T19:10:50.993 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:50 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-07-14T19:10:50.993 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:50 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2023-07-14T19:10:50.993 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:50 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:10:50.993 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:50 smithi130 ceph-mon[97832]: Reconfiguring daemon mon.c on smithi191 2023-07-14T19:10:50.994 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:10:50.991+0000 7f82d37fe700 1 -- 172.21.15.191:0/2289972291 <== mgr.14152 v2:172.21.15.40:6800/1693027118 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f82dc060a80 con 0x7f82c8038c80 2023-07-14T19:10:50.994 INFO:teuthology.orchestra.run.smithi191.stdout:Scheduled mgr update... 2023-07-14T19:10:50.995 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:10:50.993+0000 7f82e0f97700 1 -- 172.21.15.191:0/2289972291 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f82c8038c80 msgr2=0x7f82c803b130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:10:50.995 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:10:50.993+0000 7f82e0f97700 1 --2- 172.21.15.191:0/2289972291 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f82c8038c80 0x7f82c803b130 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f82cc009850 tx=0x7f82cc008040 comp rx=0 tx=0).stop 2023-07-14T19:10:50.995 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:10:50.993+0000 7f82e0f97700 1 -- 172.21.15.191:0/2289972291 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f82dc0fe730 msgr2=0x7f82dc196d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:10:50.996 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:10:50.993+0000 7f82e0f97700 1 --2- 172.21.15.191:0/2289972291 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f82dc0fe730 0x7f82dc196d20 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f82c4006010 tx=0x7f82c4004cb0 comp rx=0 tx=0).stop 2023-07-14T19:10:50.996 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:10:50.993+0000 7f82e0f97700 1 -- 172.21.15.191:0/2289972291 shutdown_connections 2023-07-14T19:10:50.996 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:10:50.993+0000 7f82e0f97700 1 --2- 172.21.15.191:0/2289972291 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f82c8038c80 0x7f82c803b130 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:10:50.996 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:10:50.993+0000 7f82e0f97700 1 --2- 172.21.15.191:0/2289972291 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f82dc0fe730 0x7f82dc196d20 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:10:50.996 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:10:50.993+0000 7f82e0f97700 1 --2- 172.21.15.191:0/2289972291 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f82dc1976d0 0x7f82dc19e820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:10:50.996 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:10:50.993+0000 7f82e0f97700 1 --2- 172.21.15.191:0/2289972291 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f82dc197260 0x7f82dc19c290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:10:50.996 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:10:50.993+0000 7f82e0f97700 1 -- 172.21.15.191:0/2289972291 >> 172.21.15.191:0/2289972291 conn(0x7f82dc0f9ce0 msgr2=0x7f82dc18df60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:10:50.997 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:10:50.993+0000 7f82e0f97700 1 -- 172.21.15.191:0/2289972291 shutdown_connections 2023-07-14T19:10:50.997 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:10:50.993+0000 7f82e0f97700 1 -- 172.21.15.191:0/2289972291 wait complete. 2023-07-14T19:10:51.497 DEBUG:teuthology.orchestra.run.smithi130:mgr.y> sudo journalctl -f -n 0 -u ceph-adc52f04-2279-11ee-9b33-001a4aab830c@mgr.y.service 2023-07-14T19:10:51.499 DEBUG:teuthology.orchestra.run.smithi191:mgr.z> sudo journalctl -f -n 0 -u ceph-adc52f04-2279-11ee-9b33-001a4aab830c@mgr.z.service 2023-07-14T19:10:51.502 INFO:tasks.cephadm:Deploying OSDs... 2023-07-14T19:10:51.502 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2023-07-14T19:10:51.502 DEBUG:teuthology.orchestra.run.smithi040:> dd if=/scratch_devs of=/dev/stdout 2023-07-14T19:10:51.523 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-07-14T19:10:51.523 DEBUG:teuthology.orchestra.run.smithi040:> stat /dev/vg_nvme/lv_1 2023-07-14T19:10:51.532 INFO:journalctl@ceph.mgr.z.smithi191.stdout:-- Logs begin at Fri 2023-07-14 18:50:54 UTC. -- 2023-07-14T19:10:51.534 INFO:journalctl@ceph.mgr.y.smithi130.stdout:-- Logs begin at Fri 2023-07-14 18:50:32 UTC. -- 2023-07-14T19:10:51.583 INFO:teuthology.orchestra.run.smithi040.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-07-14T19:10:51.584 INFO:teuthology.orchestra.run.smithi040.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-07-14T19:10:51.584 INFO:teuthology.orchestra.run.smithi040.stdout:Device: 6h/6d Inode: 373247 Links: 1 2023-07-14T19:10:51.584 INFO:teuthology.orchestra.run.smithi040.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-07-14T19:10:51.584 INFO:teuthology.orchestra.run.smithi040.stdout:Context: system_u:object_r:device_t:s0 2023-07-14T19:10:51.584 INFO:teuthology.orchestra.run.smithi040.stdout:Access: 2023-07-14 19:10:02.384263510 +0000 2023-07-14T19:10:51.585 INFO:teuthology.orchestra.run.smithi040.stdout:Modify: 2023-07-14 19:02:56.672923272 +0000 2023-07-14T19:10:51.585 INFO:teuthology.orchestra.run.smithi040.stdout:Change: 2023-07-14 19:02:56.672923272 +0000 2023-07-14T19:10:51.585 INFO:teuthology.orchestra.run.smithi040.stdout: Birth: - 2023-07-14T19:10:51.585 DEBUG:teuthology.orchestra.run.smithi040:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-07-14T19:10:51.655 INFO:teuthology.orchestra.run.smithi040.stderr:1+0 records in 2023-07-14T19:10:51.655 INFO:teuthology.orchestra.run.smithi040.stderr:1+0 records out 2023-07-14T19:10:51.655 INFO:teuthology.orchestra.run.smithi040.stderr:512 bytes copied, 0.000264143 s, 1.9 MB/s 2023-07-14T19:10:51.656 DEBUG:teuthology.orchestra.run.smithi040:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-07-14T19:10:51.716 DEBUG:teuthology.orchestra.run.smithi040:> stat /dev/vg_nvme/lv_2 2023-07-14T19:10:51.722 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:51 smithi191 ceph-mon[98311]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T19:10:51.722 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:51 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:10:51.722 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:51 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:10:51.723 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:51 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:10:51.723 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:51 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T19:10:51.723 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:51 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:10:51.723 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:51 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T19:10:51.723 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:51 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:10:51.723 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:51 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2023-07-14T19:10:51.723 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:51 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2023-07-14T19:10:51.724 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:51 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2023-07-14T19:10:51.724 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:51 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:10:51.777 INFO:teuthology.orchestra.run.smithi040.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-07-14T19:10:51.777 INFO:teuthology.orchestra.run.smithi040.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-07-14T19:10:51.777 INFO:teuthology.orchestra.run.smithi040.stdout:Device: 6h/6d Inode: 380493 Links: 1 2023-07-14T19:10:51.777 INFO:teuthology.orchestra.run.smithi040.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-07-14T19:10:51.778 INFO:teuthology.orchestra.run.smithi040.stdout:Context: system_u:object_r:device_t:s0 2023-07-14T19:10:51.778 INFO:teuthology.orchestra.run.smithi040.stdout:Access: 2023-07-14 19:10:02.384263510 +0000 2023-07-14T19:10:51.778 INFO:teuthology.orchestra.run.smithi040.stdout:Modify: 2023-07-14 19:02:57.218916460 +0000 2023-07-14T19:10:51.778 INFO:teuthology.orchestra.run.smithi040.stdout:Change: 2023-07-14 19:02:57.218916460 +0000 2023-07-14T19:10:51.778 INFO:teuthology.orchestra.run.smithi040.stdout: Birth: - 2023-07-14T19:10:51.778 DEBUG:teuthology.orchestra.run.smithi040:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-07-14T19:10:51.843 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:10:51 smithi040 ceph-mon[96518]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T19:10:51.844 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:10:51 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:10:51.844 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:10:51 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:10:51.844 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:10:51 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:10:51.844 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:10:51 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T19:10:51.844 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:10:51 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:10:51.845 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:10:51 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T19:10:51.845 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:10:51 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:10:51.845 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:10:51 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2023-07-14T19:10:51.845 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:10:51 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2023-07-14T19:10:51.845 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:10:51 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2023-07-14T19:10:51.845 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:10:51 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:10:51.846 INFO:teuthology.orchestra.run.smithi040.stderr:1+0 records in 2023-07-14T19:10:51.846 INFO:teuthology.orchestra.run.smithi040.stderr:1+0 records out 2023-07-14T19:10:51.846 INFO:teuthology.orchestra.run.smithi040.stderr:512 bytes copied, 0.000298647 s, 1.7 MB/s 2023-07-14T19:10:51.847 DEBUG:teuthology.orchestra.run.smithi040:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-07-14T19:10:51.907 DEBUG:teuthology.orchestra.run.smithi040:> stat /dev/vg_nvme/lv_3 2023-07-14T19:10:51.966 INFO:teuthology.orchestra.run.smithi040.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-07-14T19:10:51.967 INFO:teuthology.orchestra.run.smithi040.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-07-14T19:10:51.967 INFO:teuthology.orchestra.run.smithi040.stdout:Device: 6h/6d Inode: 373320 Links: 1 2023-07-14T19:10:51.967 INFO:teuthology.orchestra.run.smithi040.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-07-14T19:10:51.967 INFO:teuthology.orchestra.run.smithi040.stdout:Context: system_u:object_r:device_t:s0 2023-07-14T19:10:51.967 INFO:teuthology.orchestra.run.smithi040.stdout:Access: 2023-07-14 19:10:02.384263510 +0000 2023-07-14T19:10:51.967 INFO:teuthology.orchestra.run.smithi040.stdout:Modify: 2023-07-14 19:02:57.748909848 +0000 2023-07-14T19:10:51.967 INFO:teuthology.orchestra.run.smithi040.stdout:Change: 2023-07-14 19:02:57.748909848 +0000 2023-07-14T19:10:51.968 INFO:teuthology.orchestra.run.smithi040.stdout: Birth: - 2023-07-14T19:10:51.968 DEBUG:teuthology.orchestra.run.smithi040:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-07-14T19:10:51.993 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:51 smithi130 ceph-mon[97832]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T19:10:51.993 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:51 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:10:51.993 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:51 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:10:51.993 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:51 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:10:51.993 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:51 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T19:10:51.993 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:51 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:10:51.994 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:51 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T19:10:51.994 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:51 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:10:51.994 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:51 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2023-07-14T19:10:51.994 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:51 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2023-07-14T19:10:51.994 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:51 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2023-07-14T19:10:51.994 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:51 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:10:52.034 INFO:teuthology.orchestra.run.smithi040.stderr:1+0 records in 2023-07-14T19:10:52.034 INFO:teuthology.orchestra.run.smithi040.stderr:1+0 records out 2023-07-14T19:10:52.035 INFO:teuthology.orchestra.run.smithi040.stderr:512 bytes copied, 0.000249472 s, 2.1 MB/s 2023-07-14T19:10:52.035 DEBUG:teuthology.orchestra.run.smithi040:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-07-14T19:10:52.094 DEBUG:teuthology.orchestra.run.smithi040:> stat /dev/vg_nvme/lv_4 2023-07-14T19:10:52.152 INFO:teuthology.orchestra.run.smithi040.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-07-14T19:10:52.153 INFO:teuthology.orchestra.run.smithi040.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-07-14T19:10:52.153 INFO:teuthology.orchestra.run.smithi040.stdout:Device: 6h/6d Inode: 375270 Links: 1 2023-07-14T19:10:52.153 INFO:teuthology.orchestra.run.smithi040.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-07-14T19:10:52.153 INFO:teuthology.orchestra.run.smithi040.stdout:Context: system_u:object_r:device_t:s0 2023-07-14T19:10:52.153 INFO:teuthology.orchestra.run.smithi040.stdout:Access: 2023-07-14 19:10:02.384263510 +0000 2023-07-14T19:10:52.153 INFO:teuthology.orchestra.run.smithi040.stdout:Modify: 2023-07-14 19:02:58.285903148 +0000 2023-07-14T19:10:52.153 INFO:teuthology.orchestra.run.smithi040.stdout:Change: 2023-07-14 19:02:58.285903148 +0000 2023-07-14T19:10:52.153 INFO:teuthology.orchestra.run.smithi040.stdout: Birth: - 2023-07-14T19:10:52.154 DEBUG:teuthology.orchestra.run.smithi040:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-07-14T19:10:52.219 INFO:teuthology.orchestra.run.smithi040.stderr:1+0 records in 2023-07-14T19:10:52.219 INFO:teuthology.orchestra.run.smithi040.stderr:1+0 records out 2023-07-14T19:10:52.219 INFO:teuthology.orchestra.run.smithi040.stderr:512 bytes copied, 0.000186496 s, 2.7 MB/s 2023-07-14T19:10:52.220 DEBUG:teuthology.orchestra.run.smithi040:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-07-14T19:10:52.278 DEBUG:teuthology.orchestra.run.smithi130:> set -ex 2023-07-14T19:10:52.279 DEBUG:teuthology.orchestra.run.smithi130:> dd if=/scratch_devs of=/dev/stdout 2023-07-14T19:10:52.299 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-07-14T19:10:52.299 DEBUG:teuthology.orchestra.run.smithi130:> stat /dev/vg_nvme/lv_1 2023-07-14T19:10:52.359 INFO:teuthology.orchestra.run.smithi130.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-07-14T19:10:52.359 INFO:teuthology.orchestra.run.smithi130.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-07-14T19:10:52.359 INFO:teuthology.orchestra.run.smithi130.stdout:Device: 6h/6d Inode: 378480 Links: 1 2023-07-14T19:10:52.360 INFO:teuthology.orchestra.run.smithi130.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-07-14T19:10:52.360 INFO:teuthology.orchestra.run.smithi130.stdout:Context: system_u:object_r:device_t:s0 2023-07-14T19:10:52.360 INFO:teuthology.orchestra.run.smithi130.stdout:Access: 2023-07-14 19:10:38.361437247 +0000 2023-07-14T19:10:52.360 INFO:teuthology.orchestra.run.smithi130.stdout:Modify: 2023-07-14 19:02:35.279983376 +0000 2023-07-14T19:10:52.360 INFO:teuthology.orchestra.run.smithi130.stdout:Change: 2023-07-14 19:02:35.279983376 +0000 2023-07-14T19:10:52.360 INFO:teuthology.orchestra.run.smithi130.stdout: Birth: - 2023-07-14T19:10:52.361 DEBUG:teuthology.orchestra.run.smithi130:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-07-14T19:10:52.429 INFO:teuthology.orchestra.run.smithi130.stderr:1+0 records in 2023-07-14T19:10:52.430 INFO:teuthology.orchestra.run.smithi130.stderr:1+0 records out 2023-07-14T19:10:52.430 INFO:teuthology.orchestra.run.smithi130.stderr:512 bytes copied, 0.000281696 s, 1.8 MB/s 2023-07-14T19:10:52.431 DEBUG:teuthology.orchestra.run.smithi130:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-07-14T19:10:52.492 DEBUG:teuthology.orchestra.run.smithi130:> stat /dev/vg_nvme/lv_2 2023-07-14T19:10:52.552 INFO:teuthology.orchestra.run.smithi130.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-07-14T19:10:52.552 INFO:teuthology.orchestra.run.smithi130.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-07-14T19:10:52.552 INFO:teuthology.orchestra.run.smithi130.stdout:Device: 6h/6d Inode: 382221 Links: 1 2023-07-14T19:10:52.553 INFO:teuthology.orchestra.run.smithi130.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-07-14T19:10:52.553 INFO:teuthology.orchestra.run.smithi130.stdout:Context: system_u:object_r:device_t:s0 2023-07-14T19:10:52.553 INFO:teuthology.orchestra.run.smithi130.stdout:Access: 2023-07-14 19:10:38.361437247 +0000 2023-07-14T19:10:52.553 INFO:teuthology.orchestra.run.smithi130.stdout:Modify: 2023-07-14 19:02:35.802973316 +0000 2023-07-14T19:10:52.553 INFO:teuthology.orchestra.run.smithi130.stdout:Change: 2023-07-14 19:02:35.802973316 +0000 2023-07-14T19:10:52.553 INFO:teuthology.orchestra.run.smithi130.stdout: Birth: - 2023-07-14T19:10:52.553 DEBUG:teuthology.orchestra.run.smithi130:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-07-14T19:10:52.620 INFO:teuthology.orchestra.run.smithi130.stderr:1+0 records in 2023-07-14T19:10:52.621 INFO:teuthology.orchestra.run.smithi130.stderr:1+0 records out 2023-07-14T19:10:52.621 INFO:teuthology.orchestra.run.smithi130.stderr:512 bytes copied, 0.000233093 s, 2.2 MB/s 2023-07-14T19:10:52.621 DEBUG:teuthology.orchestra.run.smithi130:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-07-14T19:10:52.681 DEBUG:teuthology.orchestra.run.smithi130:> stat /dev/vg_nvme/lv_3 2023-07-14T19:10:52.739 INFO:teuthology.orchestra.run.smithi130.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-07-14T19:10:52.740 INFO:teuthology.orchestra.run.smithi130.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-07-14T19:10:52.740 INFO:teuthology.orchestra.run.smithi130.stdout:Device: 6h/6d Inode: 377305 Links: 1 2023-07-14T19:10:52.740 INFO:teuthology.orchestra.run.smithi130.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-07-14T19:10:52.740 INFO:teuthology.orchestra.run.smithi130.stdout:Context: system_u:object_r:device_t:s0 2023-07-14T19:10:52.740 INFO:teuthology.orchestra.run.smithi130.stdout:Access: 2023-07-14 19:10:38.361437247 +0000 2023-07-14T19:10:52.741 INFO:teuthology.orchestra.run.smithi130.stdout:Modify: 2023-07-14 19:02:36.316963428 +0000 2023-07-14T19:10:52.741 INFO:teuthology.orchestra.run.smithi130.stdout:Change: 2023-07-14 19:02:36.316963428 +0000 2023-07-14T19:10:52.741 INFO:teuthology.orchestra.run.smithi130.stdout: Birth: - 2023-07-14T19:10:52.741 DEBUG:teuthology.orchestra.run.smithi130:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-07-14T19:10:52.742 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:52 smithi130 ceph-mon[97832]: from='client.24100 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi040=x;smithi130=y;smithi191=z", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T19:10:52.743 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:52 smithi130 ceph-mon[97832]: Saving service mgr spec with placement smithi040=x;smithi130=y;smithi191=z;count:3 2023-07-14T19:10:52.743 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:52 smithi130 ceph-mon[97832]: Deploying daemon mgr.z on smithi191 2023-07-14T19:10:52.769 INFO:teuthology.orchestra.run.smithi130.stderr:1+0 records in 2023-07-14T19:10:52.769 INFO:teuthology.orchestra.run.smithi130.stderr:1+0 records out 2023-07-14T19:10:52.770 INFO:teuthology.orchestra.run.smithi130.stderr:512 bytes copied, 0.000266072 s, 1.9 MB/s 2023-07-14T19:10:52.770 DEBUG:teuthology.orchestra.run.smithi130:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-07-14T19:10:52.829 DEBUG:teuthology.orchestra.run.smithi130:> stat /dev/vg_nvme/lv_4 2023-07-14T19:10:52.886 INFO:teuthology.orchestra.run.smithi130.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-07-14T19:10:52.886 INFO:teuthology.orchestra.run.smithi130.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-07-14T19:10:52.887 INFO:teuthology.orchestra.run.smithi130.stdout:Device: 6h/6d Inode: 383314 Links: 1 2023-07-14T19:10:52.887 INFO:teuthology.orchestra.run.smithi130.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-07-14T19:10:52.887 INFO:teuthology.orchestra.run.smithi130.stdout:Context: system_u:object_r:device_t:s0 2023-07-14T19:10:52.887 INFO:teuthology.orchestra.run.smithi130.stdout:Access: 2023-07-14 19:10:38.361437247 +0000 2023-07-14T19:10:52.887 INFO:teuthology.orchestra.run.smithi130.stdout:Modify: 2023-07-14 19:02:36.826953618 +0000 2023-07-14T19:10:52.887 INFO:teuthology.orchestra.run.smithi130.stdout:Change: 2023-07-14 19:02:36.826953618 +0000 2023-07-14T19:10:52.887 INFO:teuthology.orchestra.run.smithi130.stdout: Birth: - 2023-07-14T19:10:52.888 DEBUG:teuthology.orchestra.run.smithi130:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-07-14T19:10:52.952 INFO:teuthology.orchestra.run.smithi130.stderr:1+0 records in 2023-07-14T19:10:52.952 INFO:teuthology.orchestra.run.smithi130.stderr:1+0 records out 2023-07-14T19:10:52.952 INFO:teuthology.orchestra.run.smithi130.stderr:512 bytes copied, 0.000200479 s, 2.6 MB/s 2023-07-14T19:10:52.953 DEBUG:teuthology.orchestra.run.smithi130:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-07-14T19:10:52.971 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:10:52 smithi040 ceph-mon[96518]: from='client.24100 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi040=x;smithi130=y;smithi191=z", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T19:10:52.972 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:10:52 smithi040 ceph-mon[96518]: Saving service mgr spec with placement smithi040=x;smithi130=y;smithi191=z;count:3 2023-07-14T19:10:52.972 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:10:52 smithi040 ceph-mon[96518]: Deploying daemon mgr.z on smithi191 2023-07-14T19:10:52.972 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:52 smithi191 ceph-mon[98311]: from='client.24100 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi040=x;smithi130=y;smithi191=z", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T19:10:52.973 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:52 smithi191 ceph-mon[98311]: Saving service mgr spec with placement smithi040=x;smithi130=y;smithi191=z;count:3 2023-07-14T19:10:52.973 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:52 smithi191 ceph-mon[98311]: Deploying daemon mgr.z on smithi191 2023-07-14T19:10:52.973 INFO:journalctl@ceph.mgr.z.smithi191.stdout:Jul 14 19:10:52 smithi191 systemd[1]: Starting Ceph mgr.z for adc52f04-2279-11ee-9b33-001a4aab830c... 2023-07-14T19:10:53.011 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2023-07-14T19:10:53.011 DEBUG:teuthology.orchestra.run.smithi191:> dd if=/scratch_devs of=/dev/stdout 2023-07-14T19:10:53.029 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-07-14T19:10:53.029 DEBUG:teuthology.orchestra.run.smithi191:> stat /dev/vg_nvme/lv_1 2023-07-14T19:10:53.088 INFO:teuthology.orchestra.run.smithi191.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-07-14T19:10:53.089 INFO:teuthology.orchestra.run.smithi191.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-07-14T19:10:53.089 INFO:teuthology.orchestra.run.smithi191.stdout:Device: 6h/6d Inode: 371530 Links: 1 2023-07-14T19:10:53.089 INFO:teuthology.orchestra.run.smithi191.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-07-14T19:10:53.089 INFO:teuthology.orchestra.run.smithi191.stdout:Context: system_u:object_r:device_t:s0 2023-07-14T19:10:53.089 INFO:teuthology.orchestra.run.smithi191.stdout:Access: 2023-07-14 19:10:52.653657459 +0000 2023-07-14T19:10:53.089 INFO:teuthology.orchestra.run.smithi191.stdout:Modify: 2023-07-14 19:02:47.915710100 +0000 2023-07-14T19:10:53.090 INFO:teuthology.orchestra.run.smithi191.stdout:Change: 2023-07-14 19:02:47.915710100 +0000 2023-07-14T19:10:53.090 INFO:teuthology.orchestra.run.smithi191.stdout: Birth: - 2023-07-14T19:10:53.090 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-07-14T19:10:53.156 INFO:teuthology.orchestra.run.smithi191.stderr:1+0 records in 2023-07-14T19:10:53.157 INFO:teuthology.orchestra.run.smithi191.stderr:1+0 records out 2023-07-14T19:10:53.157 INFO:teuthology.orchestra.run.smithi191.stderr:512 bytes copied, 0.000260572 s, 2.0 MB/s 2023-07-14T19:10:53.158 DEBUG:teuthology.orchestra.run.smithi191:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-07-14T19:10:53.215 DEBUG:teuthology.orchestra.run.smithi191:> stat /dev/vg_nvme/lv_2 2023-07-14T19:10:53.272 INFO:teuthology.orchestra.run.smithi191.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-07-14T19:10:53.272 INFO:teuthology.orchestra.run.smithi191.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-07-14T19:10:53.272 INFO:teuthology.orchestra.run.smithi191.stdout:Device: 6h/6d Inode: 381093 Links: 1 2023-07-14T19:10:53.272 INFO:teuthology.orchestra.run.smithi191.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-07-14T19:10:53.273 INFO:teuthology.orchestra.run.smithi191.stdout:Context: system_u:object_r:device_t:s0 2023-07-14T19:10:53.273 INFO:teuthology.orchestra.run.smithi191.stdout:Access: 2023-07-14 19:10:52.653657459 +0000 2023-07-14T19:10:53.273 INFO:teuthology.orchestra.run.smithi191.stdout:Modify: 2023-07-14 19:02:48.436700656 +0000 2023-07-14T19:10:53.273 INFO:teuthology.orchestra.run.smithi191.stdout:Change: 2023-07-14 19:02:48.436700656 +0000 2023-07-14T19:10:53.273 INFO:teuthology.orchestra.run.smithi191.stdout: Birth: - 2023-07-14T19:10:53.274 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-07-14T19:10:53.275 INFO:journalctl@ceph.mgr.z.smithi191.stdout:Jul 14 19:10:52 smithi191 podman[100137]: 2023-07-14T19:10:53.303 INFO:teuthology.orchestra.run.smithi191.stderr:1+0 records in 2023-07-14T19:10:53.303 INFO:teuthology.orchestra.run.smithi191.stderr:1+0 records out 2023-07-14T19:10:53.303 INFO:teuthology.orchestra.run.smithi191.stderr:512 bytes copied, 0.000209986 s, 2.4 MB/s 2023-07-14T19:10:53.304 DEBUG:teuthology.orchestra.run.smithi191:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-07-14T19:10:53.363 DEBUG:teuthology.orchestra.run.smithi191:> stat /dev/vg_nvme/lv_3 2023-07-14T19:10:53.421 INFO:teuthology.orchestra.run.smithi191.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-07-14T19:10:53.421 INFO:teuthology.orchestra.run.smithi191.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-07-14T19:10:53.421 INFO:teuthology.orchestra.run.smithi191.stdout:Device: 6h/6d Inode: 377331 Links: 1 2023-07-14T19:10:53.422 INFO:teuthology.orchestra.run.smithi191.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-07-14T19:10:53.422 INFO:teuthology.orchestra.run.smithi191.stdout:Context: system_u:object_r:device_t:s0 2023-07-14T19:10:53.422 INFO:teuthology.orchestra.run.smithi191.stdout:Access: 2023-07-14 19:10:52.653657459 +0000 2023-07-14T19:10:53.422 INFO:teuthology.orchestra.run.smithi191.stdout:Modify: 2023-07-14 19:02:48.964691084 +0000 2023-07-14T19:10:53.422 INFO:teuthology.orchestra.run.smithi191.stdout:Change: 2023-07-14 19:02:48.964691084 +0000 2023-07-14T19:10:53.422 INFO:teuthology.orchestra.run.smithi191.stdout: Birth: - 2023-07-14T19:10:53.423 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-07-14T19:10:53.487 INFO:teuthology.orchestra.run.smithi191.stderr:1+0 records in 2023-07-14T19:10:53.488 INFO:teuthology.orchestra.run.smithi191.stderr:1+0 records out 2023-07-14T19:10:53.488 INFO:teuthology.orchestra.run.smithi191.stderr:512 bytes copied, 0.000230756 s, 2.2 MB/s 2023-07-14T19:10:53.489 DEBUG:teuthology.orchestra.run.smithi191:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-07-14T19:10:53.548 DEBUG:teuthology.orchestra.run.smithi191:> stat /dev/vg_nvme/lv_4 2023-07-14T19:10:53.590 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:53 smithi191 ceph-mon[98311]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T19:10:53.591 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:53 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:10:53.591 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:53 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:10:53.591 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:53 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:10:53.591 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:53 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2023-07-14T19:10:53.591 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:53 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2023-07-14T19:10:53.592 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:53 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2023-07-14T19:10:53.592 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:53 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:10:53.592 INFO:journalctl@ceph.mgr.z.smithi191.stdout:Jul 14 19:10:53 smithi191 bash[100137]: d34005d85306de40da8c5afa82503383f69b379eb763fb25f8dc11f1589549d4 2023-07-14T19:10:53.592 INFO:journalctl@ceph.mgr.z.smithi191.stdout:Jul 14 19:10:53 smithi191 ceph-mgr[100258]: -- 172.21.15.191:0/2324907297 <== mon.1 v2:172.21.15.191:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55eb0c1d31e0 con 0x55eb0c1a8800 2023-07-14T19:10:53.592 INFO:journalctl@ceph.mgr.z.smithi191.stdout:Jul 14 19:10:53 smithi191 systemd[1]: Started Ceph mgr.z for adc52f04-2279-11ee-9b33-001a4aab830c. 2023-07-14T19:10:53.592 INFO:journalctl@ceph.mgr.z.smithi191.stdout:Jul 14 19:10:53 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-mgr-z[100172]: 2023-07-14T19:10:53.410+0000 7f1798049200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-07-14T19:10:53.593 INFO:journalctl@ceph.mgr.z.smithi191.stdout:Jul 14 19:10:53 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-mgr-z[100172]: 2023-07-14T19:10:53.493+0000 7f1798049200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-07-14T19:10:53.593 INFO:journalctl@ceph.mgr.z.smithi191.stdout:Jul 14 19:10:53 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-mgr-z[100172]: 2023-07-14T19:10:53.587+0000 7f1798049200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-07-14T19:10:53.606 INFO:teuthology.orchestra.run.smithi191.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-07-14T19:10:53.606 INFO:teuthology.orchestra.run.smithi191.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-07-14T19:10:53.606 INFO:teuthology.orchestra.run.smithi191.stdout:Device: 6h/6d Inode: 381302 Links: 1 2023-07-14T19:10:53.607 INFO:teuthology.orchestra.run.smithi191.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-07-14T19:10:53.607 INFO:teuthology.orchestra.run.smithi191.stdout:Context: system_u:object_r:device_t:s0 2023-07-14T19:10:53.607 INFO:teuthology.orchestra.run.smithi191.stdout:Access: 2023-07-14 19:10:52.653657459 +0000 2023-07-14T19:10:53.607 INFO:teuthology.orchestra.run.smithi191.stdout:Modify: 2023-07-14 19:02:49.469681929 +0000 2023-07-14T19:10:53.608 INFO:teuthology.orchestra.run.smithi191.stdout:Change: 2023-07-14 19:02:49.469681929 +0000 2023-07-14T19:10:53.608 INFO:teuthology.orchestra.run.smithi191.stdout: Birth: - 2023-07-14T19:10:53.609 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-07-14T19:10:53.672 INFO:teuthology.orchestra.run.smithi191.stderr:1+0 records in 2023-07-14T19:10:53.672 INFO:teuthology.orchestra.run.smithi191.stderr:1+0 records out 2023-07-14T19:10:53.672 INFO:teuthology.orchestra.run.smithi191.stderr:512 bytes copied, 0.000173334 s, 3.0 MB/s 2023-07-14T19:10:53.673 DEBUG:teuthology.orchestra.run.smithi191:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-07-14T19:10:53.731 INFO:tasks.cephadm:Deploying osd.0 on smithi040 with /dev/vg_nvme/lv_4... 2023-07-14T19:10:53.731 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid adc52f04-2279-11ee-9b33-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2023-07-14T19:10:53.761 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:10:53 smithi040 ceph-mon[96518]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T19:10:53.761 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:10:53 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:10:53.761 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:10:53 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:10:53.761 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:10:53 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:10:53.761 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:10:53 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2023-07-14T19:10:53.762 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:10:53 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2023-07-14T19:10:53.762 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:10:53 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2023-07-14T19:10:53.762 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:10:53 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:10:53.859 INFO:journalctl@ceph.mgr.z.smithi191.stdout:Jul 14 19:10:53 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-mgr-z[100172]: 2023-07-14T19:10:53.686+0000 7f1798049200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-07-14T19:10:53.993 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:53 smithi130 ceph-mon[97832]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T19:10:53.993 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:53 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:10:53.993 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:53 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:10:53.993 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:53 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:10:53.993 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:53 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2023-07-14T19:10:53.994 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:53 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2023-07-14T19:10:53.994 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:53 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2023-07-14T19:10:53.994 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:53 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:10:54.164 INFO:journalctl@ceph.mgr.z.smithi191.stdout:Jul 14 19:10:53 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-mgr-z[100172]: 2023-07-14T19:10:53.857+0000 7f1798049200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-07-14T19:10:54.300 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/adc52f04-2279-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T19:10:54.472 INFO:journalctl@ceph.mgr.z.smithi191.stdout:Jul 14 19:10:54 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-mgr-z[100172]: 2023-07-14T19:10:54.161+0000 7f1798049200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-07-14T19:10:54.473 INFO:journalctl@ceph.mgr.z.smithi191.stdout:Jul 14 19:10:54 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-mgr-z[100172]: 2023-07-14T19:10:54.250+0000 7f1798049200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-07-14T19:10:54.473 INFO:journalctl@ceph.mgr.z.smithi191.stdout:Jul 14 19:10:54 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-mgr-z[100172]: 2023-07-14T19:10:54.378+0000 7f1798049200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-07-14T19:10:54.708 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:54 smithi130 ceph-mon[97832]: Deploying daemon mgr.y on smithi130 2023-07-14T19:10:54.816 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:10:54 smithi040 ceph-mon[96518]: Deploying daemon mgr.y on smithi130 2023-07-14T19:10:54.889 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:54 smithi191 ceph-mon[98311]: Deploying daemon mgr.y on smithi130 2023-07-14T19:10:54.890 INFO:journalctl@ceph.mgr.z.smithi191.stdout:Jul 14 19:10:54 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-mgr-z[100172]: 2023-07-14T19:10:54.470+0000 7f1798049200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-07-14T19:10:54.993 INFO:journalctl@ceph.mgr.y.smithi130.stdout:Jul 14 19:10:54 smithi130 systemd[1]: Starting Ceph mgr.y for adc52f04-2279-11ee-9b33-001a4aab830c... 2023-07-14T19:10:55.222 INFO:journalctl@ceph.mgr.z.smithi191.stdout:Jul 14 19:10:54 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-mgr-z[100172]: 2023-07-14T19:10:54.887+0000 7f1798049200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-07-14T19:10:55.500 INFO:journalctl@ceph.mgr.y.smithi130.stdout:Jul 14 19:10:55 smithi130 podman[99281]: 2023-07-14T19:10:55.722 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:55 smithi191 ceph-mon[98311]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T19:10:55.722 INFO:journalctl@ceph.mgr.z.smithi191.stdout:Jul 14 19:10:55 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-mgr-z[100172]: 2023-07-14T19:10:55.457+0000 7f1798049200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-07-14T19:10:55.723 INFO:journalctl@ceph.mgr.z.smithi191.stdout:Jul 14 19:10:55 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-mgr-z[100172]: 2023-07-14T19:10:55.630+0000 7f1798049200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-07-14T19:10:55.798 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:55 smithi130 ceph-mon[97832]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T19:10:55.798 INFO:journalctl@ceph.mgr.y.smithi130.stdout:Jul 14 19:10:55 smithi130 bash[99281]: 52d2499424a3c13357589504dfdfb033237a1e3c88e23010523404d596fd9461 2023-07-14T19:10:55.798 INFO:journalctl@ceph.mgr.y.smithi130.stdout:Jul 14 19:10:55 smithi130 ceph-mgr[99322]: -- 172.21.15.130:0/2898155702 <== mon.2 v2:172.21.15.130:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x560563eb11e0 con 0x560563e87000 2023-07-14T19:10:55.798 INFO:journalctl@ceph.mgr.y.smithi130.stdout:Jul 14 19:10:55 smithi130 systemd[1]: Started Ceph mgr.y for adc52f04-2279-11ee-9b33-001a4aab830c. 2023-07-14T19:10:55.799 INFO:journalctl@ceph.mgr.y.smithi130.stdout:Jul 14 19:10:55 smithi130 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-mgr-y[99297]: 2023-07-14T19:10:55.625+0000 7f5a22a1f200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-07-14T19:10:55.799 INFO:journalctl@ceph.mgr.y.smithi130.stdout:Jul 14 19:10:55 smithi130 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-mgr-y[99297]: 2023-07-14T19:10:55.706+0000 7f5a22a1f200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-07-14T19:10:55.799 INFO:journalctl@ceph.mgr.y.smithi130.stdout:Jul 14 19:10:55 smithi130 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-mgr-y[99297]: 2023-07-14T19:10:55.796+0000 7f5a22a1f200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-07-14T19:10:55.971 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:10:55 smithi040 ceph-mon[96518]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T19:10:56.083 INFO:journalctl@ceph.mgr.y.smithi130.stdout:Jul 14 19:10:55 smithi130 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-mgr-y[99297]: 2023-07-14T19:10:55.897+0000 7f5a22a1f200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-07-14T19:10:56.083 INFO:journalctl@ceph.mgr.y.smithi130.stdout:Jul 14 19:10:56 smithi130 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-mgr-y[99297]: 2023-07-14T19:10:56.081+0000 7f5a22a1f200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-07-14T19:10:56.111 INFO:teuthology.orchestra.run.smithi040.stderr:--> Zapping: /dev/vg_nvme/lv_4 2023-07-14T19:10:56.111 INFO:teuthology.orchestra.run.smithi040.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2023-07-14T19:10:56.111 INFO:teuthology.orchestra.run.smithi040.stderr: stderr: 10+0 records in 2023-07-14T19:10:56.111 INFO:teuthology.orchestra.run.smithi040.stderr:10+0 records out 2023-07-14T19:10:56.112 INFO:teuthology.orchestra.run.smithi040.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0108038 s, 971 MB/s 2023-07-14T19:10:56.112 INFO:teuthology.orchestra.run.smithi040.stderr:--> Zapping successful for: 2023-07-14T19:10:56.648 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid adc52f04-2279-11ee-9b33-001a4aab830c -- ceph orch daemon add osd smithi040:vg_nvme/lv_4 2023-07-14T19:10:56.705 INFO:journalctl@ceph.mgr.z.smithi191.stdout:Jul 14 19:10:56 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-mgr-z[100172]: 2023-07-14T19:10:56.432+0000 7f1798049200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-07-14T19:10:56.705 INFO:journalctl@ceph.mgr.z.smithi191.stdout:Jul 14 19:10:56 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-mgr-z[100172]: 2023-07-14T19:10:56.523+0000 7f1798049200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-07-14T19:10:56.705 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:56 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:10:56.705 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:56 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:10:56.706 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:56 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:10:56.706 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:56 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:10:56.706 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:56 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T19:10:56.706 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:56 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:10:56.738 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:56 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:10:56.738 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:56 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:10:56.739 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:56 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:10:56.739 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:56 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:10:56.739 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:56 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T19:10:56.740 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:56 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:10:56.740 INFO:journalctl@ceph.mgr.y.smithi130.stdout:Jul 14 19:10:56 smithi130 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-mgr-y[99297]: 2023-07-14T19:10:56.405+0000 7f5a22a1f200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-07-14T19:10:56.740 INFO:journalctl@ceph.mgr.y.smithi130.stdout:Jul 14 19:10:56 smithi130 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-mgr-y[99297]: 2023-07-14T19:10:56.502+0000 7f5a22a1f200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-07-14T19:10:56.741 INFO:journalctl@ceph.mgr.y.smithi130.stdout:Jul 14 19:10:56 smithi130 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-mgr-y[99297]: 2023-07-14T19:10:56.638+0000 7f5a22a1f200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-07-14T19:10:56.741 INFO:journalctl@ceph.mgr.y.smithi130.stdout:Jul 14 19:10:56 smithi130 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-mgr-y[99297]: 2023-07-14T19:10:56.736+0000 7f5a22a1f200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-07-14T19:10:56.971 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:10:56 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:10:56.972 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:10:56 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:10:56.972 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:10:56 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:10:56.972 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:10:56 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:10:56.972 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:10:56 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T19:10:56.972 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:10:56 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:10:56.973 INFO:journalctl@ceph.mgr.z.smithi191.stdout:Jul 14 19:10:56 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-mgr-z[100172]: 2023-07-14T19:10:56.702+0000 7f1798049200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-07-14T19:10:57.249 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/adc52f04-2279-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T19:10:57.397 INFO:journalctl@ceph.mgr.z.smithi191.stdout:Jul 14 19:10:56 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-mgr-z[100172]: 2023-07-14T19:10:56.995+0000 7f1798049200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-07-14T19:10:57.397 INFO:journalctl@ceph.mgr.z.smithi191.stdout:Jul 14 19:10:57 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-mgr-z[100172]: 2023-07-14T19:10:57.231+0000 7f1798049200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-07-14T19:10:57.492 INFO:journalctl@ceph.mgr.y.smithi130.stdout:Jul 14 19:10:57 smithi130 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-mgr-y[99297]: 2023-07-14T19:10:57.183+0000 7f5a22a1f200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-07-14T19:10:57.662 INFO:journalctl@ceph.mgr.z.smithi191.stdout:Jul 14 19:10:57 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-mgr-z[100172]: 2023-07-14T19:10:57.395+0000 7f1798049200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-07-14T19:10:57.663 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:57 smithi191 ceph-mon[98311]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T19:10:57.663 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:57 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:10:57.663 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:57 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:10:57.663 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:57 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:10:57.663 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:57 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:10:57.663 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:57 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:10:57.663 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:57 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T19:10:57.664 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:57 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:10:57.664 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:57 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2023-07-14T19:10:57.664 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:57 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2023-07-14T19:10:57.664 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:57 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:10:57.721 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:10:57 smithi040 ceph-mon[96518]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T19:10:57.722 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:10:57 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:10:57.722 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:10:57 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:10:57.722 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:10:57 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:10:57.722 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:10:57 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:10:57.722 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:10:57 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:10:57.723 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:10:57 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T19:10:57.723 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:10:57 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:10:57.723 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:10:57 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2023-07-14T19:10:57.723 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:10:57 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2023-07-14T19:10:57.723 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:10:57 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:10:57.934 INFO:journalctl@ceph.mgr.y.smithi130.stdout:Jul 14 19:10:57 smithi130 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-mgr-y[99297]: 2023-07-14T19:10:57.762+0000 7f5a22a1f200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-07-14T19:10:57.934 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:57 smithi130 ceph-mon[97832]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T19:10:57.934 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:57 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:10:57.934 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:57 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:10:57.935 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:57 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:10:57.935 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:57 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:10:57.935 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:57 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:10:57.935 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:57 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T19:10:57.935 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:57 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:10:57.935 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:57 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2023-07-14T19:10:57.936 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:57 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2023-07-14T19:10:57.936 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:57 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:10:57.972 INFO:journalctl@ceph.mgr.z.smithi191.stdout:Jul 14 19:10:57 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-mgr-z[100172]: 2023-07-14T19:10:57.660+0000 7f1798049200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-07-14T19:10:58.242 INFO:journalctl@ceph.mgr.y.smithi130.stdout:Jul 14 19:10:57 smithi130 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-mgr-y[99297]: 2023-07-14T19:10:57.932+0000 7f5a22a1f200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-07-14T19:10:58.678 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:58 smithi191 ceph-mon[98311]: Reconfiguring mgr.x (unknown last config time)... 2023-07-14T19:10:58.678 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:58 smithi191 ceph-mon[98311]: Reconfiguring daemon mgr.x on smithi040 2023-07-14T19:10:58.678 INFO:journalctl@ceph.mgr.z.smithi191.stdout:Jul 14 19:10:58 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-mgr-z[100172]: 2023-07-14T19:10:58.356+0000 7f1798049200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-07-14T19:10:58.679 INFO:journalctl@ceph.mgr.z.smithi191.stdout:Jul 14 19:10:58 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-mgr-z[100172]: 2023-07-14T19:10:58.485+0000 7f1798049200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-07-14T19:10:58.806 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:10:58 smithi040 ceph-mon[96518]: Reconfiguring mgr.x (unknown last config time)... 2023-07-14T19:10:58.806 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:10:58 smithi040 ceph-mon[96518]: Reconfiguring daemon mgr.x on smithi040 2023-07-14T19:10:58.958 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:58 smithi130 ceph-mon[97832]: Reconfiguring mgr.x (unknown last config time)... 2023-07-14T19:10:58.958 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:58 smithi130 ceph-mon[97832]: Reconfiguring daemon mgr.x on smithi040 2023-07-14T19:10:58.958 INFO:journalctl@ceph.mgr.y.smithi130.stdout:Jul 14 19:10:58 smithi130 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-mgr-y[99297]: 2023-07-14T19:10:58.695+0000 7f5a22a1f200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-07-14T19:10:58.958 INFO:journalctl@ceph.mgr.y.smithi130.stdout:Jul 14 19:10:58 smithi130 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-mgr-y[99297]: 2023-07-14T19:10:58.785+0000 7f5a22a1f200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-07-14T19:10:58.972 INFO:journalctl@ceph.mgr.z.smithi191.stdout:Jul 14 19:10:58 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-mgr-z[100172]: 2023-07-14T19:10:58.675+0000 7f1798049200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-07-14T19:10:58.972 INFO:journalctl@ceph.mgr.z.smithi191.stdout:Jul 14 19:10:58 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-mgr-z[100172]: 2023-07-14T19:10:58.788+0000 7f1798049200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-07-14T19:10:59.109 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:10:59.106+0000 7f0304cad700 1 -- 172.21.15.40:0/1211970460 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f03000febb0 msgr2=0x7f03000fefc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:10:59.109 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:10:59.106+0000 7f0304cad700 1 --2- 172.21.15.40:0/1211970460 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f03000febb0 0x7f03000fefc0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f02f0009990 tx=0x7f02f0009ca0 comp rx=0 tx=0).stop 2023-07-14T19:10:59.109 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:10:59.106+0000 7f0304cad700 1 -- 172.21.15.40:0/1211970460 shutdown_connections 2023-07-14T19:10:59.109 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:10:59.106+0000 7f0304cad700 1 --2- 172.21.15.40:0/1211970460 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0300100740 0x7f0300107000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:10:59.110 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:10:59.106+0000 7f0304cad700 1 --2- 172.21.15.40:0/1211970460 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f03000febb0 0x7f03000fefc0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:10:59.110 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:10:59.106+0000 7f0304cad700 1 --2- 172.21.15.40:0/1211970460 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f03000ffdb0 0x7f0300100200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:10:59.110 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:10:59.106+0000 7f0304cad700 1 -- 172.21.15.40:0/1211970460 >> 172.21.15.40:0/1211970460 conn(0x7f03000fa1e0 msgr2=0x7f03000fc630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:10:59.110 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:10:59.106+0000 7f0304cad700 1 -- 172.21.15.40:0/1211970460 shutdown_connections 2023-07-14T19:10:59.110 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:10:59.106+0000 7f0304cad700 1 -- 172.21.15.40:0/1211970460 wait complete. 2023-07-14T19:10:59.112 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:10:59.107+0000 7f0304cad700 1 Processor -- start 2023-07-14T19:10:59.112 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:10:59.107+0000 7f0304cad700 1 -- start start 2023-07-14T19:10:59.112 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:10:59.107+0000 7f0304cad700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f03000febb0 0x7f0300068c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:10:59.112 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:10:59.107+0000 7f0304cad700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f03000ffdb0 0x7f030019a770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:10:59.113 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:10:59.107+0000 7f0304cad700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0300100740 0x7f030019cdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:10:59.113 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:10:59.107+0000 7f0304cad700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f030019d560 con 0x7f0300100740 2023-07-14T19:10:59.113 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:10:59.107+0000 7f0304cad700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f030019d6d0 con 0x7f03000ffdb0 2023-07-14T19:10:59.113 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:10:59.107+0000 7f0304cad700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f030019d890 con 0x7f03000febb0 2023-07-14T19:10:59.113 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:10:59.107+0000 7f02fed9d700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0300100740 0x7f030019cdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:10:59.113 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:10:59.107+0000 7f02fe59c700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f03000febb0 0x7f0300068c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:10:59.114 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:10:59.107+0000 7f02fdd9b700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f03000ffdb0 0x7f030019a770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:10:59.114 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:10:59.107+0000 7f02fed9d700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0300100740 0x7f030019cdc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:60194/0 (socket says 172.21.15.40:60194) 2023-07-14T19:10:59.114 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:10:59.107+0000 7f02fed9d700 1 -- 172.21.15.40:0/2975404528 learned_addr learned my addr 172.21.15.40:0/2975404528 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-07-14T19:10:59.114 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:10:59.108+0000 7f02fed9d700 1 -- 172.21.15.40:0/2975404528 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f03000febb0 msgr2=0x7f0300068c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:10:59.114 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:10:59.108+0000 7f02fed9d700 1 --2- 172.21.15.40:0/2975404528 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f03000febb0 0x7f0300068c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:10:59.114 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:10:59.108+0000 7f02fed9d700 1 -- 172.21.15.40:0/2975404528 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f03000ffdb0 msgr2=0x7f030019a770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:10:59.114 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:10:59.108+0000 7f02fed9d700 1 --2- 172.21.15.40:0/2975404528 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f03000ffdb0 0x7f030019a770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:10:59.115 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:10:59.108+0000 7f02fed9d700 1 -- 172.21.15.40:0/2975404528 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02f00095f0 con 0x7f0300100740 2023-07-14T19:10:59.115 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:10:59.108+0000 7f02fdd9b700 1 --2- 172.21.15.40:0/2975404528 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f03000ffdb0 0x7f030019a770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-14T19:10:59.115 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:10:59.108+0000 7f02fed9d700 1 --2- 172.21.15.40:0/2975404528 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0300100740 0x7f030019cdc0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f02f8007ee0 tx=0x7f02f800d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:10:59.115 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:10:59.108+0000 7f02f77fe700 1 -- 172.21.15.40:0/2975404528 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02f800dd20 con 0x7f0300100740 2023-07-14T19:10:59.115 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:10:59.108+0000 7f02f77fe700 1 -- 172.21.15.40:0/2975404528 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f02f800f040 con 0x7f0300100740 2023-07-14T19:10:59.115 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:10:59.108+0000 7f0304cad700 1 -- 172.21.15.40:0/2975404528 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f030019db70 con 0x7f0300100740 2023-07-14T19:10:59.115 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:10:59.108+0000 7f02f77fe700 1 -- 172.21.15.40:0/2975404528 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02f80048a0 con 0x7f0300100740 2023-07-14T19:10:59.116 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:10:59.108+0000 7f0304cad700 1 -- 172.21.15.40:0/2975404528 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f030019e070 con 0x7f0300100740 2023-07-14T19:10:59.116 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:10:59.109+0000 7f02f77fe700 1 -- 172.21.15.40:0/2975404528 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 13) v1 ==== 45539+0+0 (secure 0 0 0) 0x7f02f8015400 con 0x7f0300100740 2023-07-14T19:10:59.116 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:10:59.109+0000 7f0304cad700 1 -- 172.21.15.40:0/2975404528 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f03001061f0 con 0x7f0300100740 2023-07-14T19:10:59.116 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:10:59.109+0000 7f02f77fe700 1 --2- 172.21.15.40:0/2975404528 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f02ec0389e0 0x7f02ec03ae90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:10:59.116 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:10:59.109+0000 7f02f77fe700 1 -- 172.21.15.40:0/2975404528 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f02f804aac0 con 0x7f0300100740 2023-07-14T19:10:59.116 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:10:59.109+0000 7f02fe59c700 1 --2- 172.21.15.40:0/2975404528 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f02ec0389e0 0x7f02ec03ae90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:10:59.116 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:10:59.110+0000 7f02fe59c700 1 --2- 172.21.15.40:0/2975404528 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f02ec0389e0 0x7f02ec03ae90 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f02f000b2e0 tx=0x7f02f0005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:10:59.117 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:10:59.114+0000 7f02f77fe700 1 -- 172.21.15.40:0/2975404528 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f02f804ce60 con 0x7f0300100740 2023-07-14T19:10:59.235 INFO:journalctl@ceph.mgr.y.smithi130.stdout:Jul 14 19:10:58 smithi130 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-mgr-y[99297]: 2023-07-14T19:10:58.956+0000 7f5a22a1f200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-07-14T19:10:59.324 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:10:59.321+0000 7f0304cad700 1 -- 172.21.15.40:0/2975404528 --> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi040:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) v1 -- 0x7f0300060a80 con 0x7f02ec0389e0 2023-07-14T19:10:59.472 INFO:journalctl@ceph.mgr.z.smithi191.stdout:Jul 14 19:10:59 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-mgr-z[100172]: 2023-07-14T19:10:59.004+0000 7f1798049200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-07-14T19:10:59.493 INFO:journalctl@ceph.mgr.y.smithi130.stdout:Jul 14 19:10:59 smithi130 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-mgr-y[99297]: 2023-07-14T19:10:59.234+0000 7f5a22a1f200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-07-14T19:10:59.493 INFO:journalctl@ceph.mgr.y.smithi130.stdout:Jul 14 19:10:59 smithi130 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-mgr-y[99297]: 2023-07-14T19:10:59.461+0000 7f5a22a1f200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-07-14T19:10:59.869 INFO:journalctl@ceph.mgr.y.smithi130.stdout:Jul 14 19:10:59 smithi130 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-mgr-y[99297]: 2023-07-14T19:10:59.609+0000 7f5a22a1f200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-07-14T19:10:59.870 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:59 smithi130 ceph-mon[97832]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T19:10:59.870 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:59 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:10:59.870 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:59 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:10:59.870 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:59 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T19:10:59.870 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:59 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:10:59.870 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:59 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T19:10:59.871 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:59 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:10:59.871 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:59 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-14T19:10:59.871 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:59 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-14T19:10:59.871 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:10:59 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:10:59.878 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:59 smithi191 ceph-mon[98311]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T19:10:59.878 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:59 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:10:59.879 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:59 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:10:59.879 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:59 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T19:10:59.879 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:59 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:10:59.879 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:59 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T19:10:59.879 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:59 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:10:59.880 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:59 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-14T19:10:59.880 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:59 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-14T19:10:59.880 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:10:59 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:10:59.971 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:10:59 smithi040 ceph-mon[96518]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T19:10:59.972 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:10:59 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:10:59.972 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:10:59 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:10:59.972 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:10:59 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T19:10:59.972 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:10:59 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:10:59.972 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:10:59 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T19:10:59.973 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:10:59 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:10:59.973 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:10:59 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-14T19:10:59.973 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:10:59 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-14T19:10:59.973 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:10:59 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:11:00.222 INFO:journalctl@ceph.mgr.z.smithi191.stdout:Jul 14 19:10:59 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-mgr-z[100172]: 2023-07-14T19:10:59.876+0000 7f1798049200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-07-14T19:11:00.242 INFO:journalctl@ceph.mgr.y.smithi130.stdout:Jul 14 19:10:59 smithi130 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-mgr-y[99297]: 2023-07-14T19:10:59.868+0000 7f5a22a1f200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-07-14T19:11:00.589 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:11:00.586+0000 7f02f77fe700 1 -- 172.21.15.40:0/2975404528 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mgrmap(e 14) v1 ==== 90599+0+0 (secure 0 0 0) 0x7f02f80143a0 con 0x7f0300100740 2023-07-14T19:11:00.793 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:11:00 smithi040 ceph-mon[96518]: from='client.14220 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi040:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T19:11:00.794 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:11:00 smithi040 ceph-mon[96518]: Standby manager daemon z started 2023-07-14T19:11:00.794 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:11:00 smithi040 ceph-mon[96518]: from='mgr.? 172.21.15.191:0/3532168858' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2023-07-14T19:11:00.794 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:11:00 smithi040 ceph-mon[96518]: from='mgr.? 172.21.15.191:0/3532168858' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2023-07-14T19:11:00.794 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:11:00 smithi040 ceph-mon[96518]: from='mgr.? 172.21.15.191:0/3532168858' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2023-07-14T19:11:00.794 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:11:00 smithi040 ceph-mon[96518]: from='mgr.? 172.21.15.191:0/3532168858' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2023-07-14T19:11:00.886 INFO:journalctl@ceph.mgr.y.smithi130.stdout:Jul 14 19:11:00 smithi130 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-mgr-y[99297]: 2023-07-14T19:11:00.566+0000 7f5a22a1f200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-07-14T19:11:00.886 INFO:journalctl@ceph.mgr.y.smithi130.stdout:Jul 14 19:11:00 smithi130 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-mgr-y[99297]: 2023-07-14T19:11:00.697+0000 7f5a22a1f200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-07-14T19:11:00.887 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:11:00 smithi130 ceph-mon[97832]: from='client.14220 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi040:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T19:11:00.887 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:11:00 smithi130 ceph-mon[97832]: Standby manager daemon z started 2023-07-14T19:11:00.887 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:11:00 smithi130 ceph-mon[97832]: from='mgr.? 172.21.15.191:0/3532168858' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2023-07-14T19:11:00.888 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:11:00 smithi130 ceph-mon[97832]: from='mgr.? 172.21.15.191:0/3532168858' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2023-07-14T19:11:00.888 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:11:00 smithi130 ceph-mon[97832]: from='mgr.? 172.21.15.191:0/3532168858' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2023-07-14T19:11:00.889 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:11:00 smithi130 ceph-mon[97832]: from='mgr.? 172.21.15.191:0/3532168858' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2023-07-14T19:11:00.972 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:11:00 smithi191 ceph-mon[98311]: from='client.14220 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi040:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T19:11:00.973 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:11:00 smithi191 ceph-mon[98311]: Standby manager daemon z started 2023-07-14T19:11:00.973 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:11:00 smithi191 ceph-mon[98311]: from='mgr.? 172.21.15.191:0/3532168858' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2023-07-14T19:11:00.973 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:11:00 smithi191 ceph-mon[98311]: from='mgr.? 172.21.15.191:0/3532168858' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2023-07-14T19:11:00.973 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:11:00 smithi191 ceph-mon[98311]: from='mgr.? 172.21.15.191:0/3532168858' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2023-07-14T19:11:00.973 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:11:00 smithi191 ceph-mon[98311]: from='mgr.? 172.21.15.191:0/3532168858' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2023-07-14T19:11:01.211 INFO:journalctl@ceph.mgr.y.smithi130.stdout:Jul 14 19:11:00 smithi130 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-mgr-y[99297]: 2023-07-14T19:11:00.884+0000 7f5a22a1f200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-07-14T19:11:01.212 INFO:journalctl@ceph.mgr.y.smithi130.stdout:Jul 14 19:11:00 smithi130 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-mgr-y[99297]: 2023-07-14T19:11:00.996+0000 7f5a22a1f200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-07-14T19:11:01.492 INFO:journalctl@ceph.mgr.y.smithi130.stdout:Jul 14 19:11:01 smithi130 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-mgr-y[99297]: 2023-07-14T19:11:01.209+0000 7f5a22a1f200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-07-14T19:11:01.971 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:11:01 smithi040 ceph-mon[96518]: mgrmap e14: x(active, since 69s), standbys: z 2023-07-14T19:11:01.971 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:11:01 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2023-07-14T19:11:01.972 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:11:01 smithi040 ceph-mon[96518]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T19:11:01.972 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:11:01 smithi191 ceph-mon[98311]: mgrmap e14: x(active, since 69s), standbys: z 2023-07-14T19:11:01.973 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:11:01 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2023-07-14T19:11:01.973 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:11:01 smithi191 ceph-mon[98311]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T19:11:01.992 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:11:01 smithi130 ceph-mon[97832]: mgrmap e14: x(active, since 69s), standbys: z 2023-07-14T19:11:01.993 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:11:01 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2023-07-14T19:11:01.993 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:11:01 smithi130 ceph-mon[97832]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T19:11:02.492 INFO:journalctl@ceph.mgr.y.smithi130.stdout:Jul 14 19:11:02 smithi130 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-mgr-y[99297]: 2023-07-14T19:11:02.063+0000 7f5a22a1f200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-07-14T19:11:02.802 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:11:02.799+0000 7f02f77fe700 1 -- 172.21.15.40:0/2975404528 <== mon.0 v2:172.21.15.40:3300/0 8 ==== mgrmap(e 15) v1 ==== 135659+0+0 (secure 0 0 0) 0x7f02f801cc30 con 0x7f0300100740 2023-07-14T19:11:02.971 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:11:02 smithi040 ceph-mon[96518]: from='client.? 172.21.15.40:0/405477386' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0d79356a-95d2-49ad-857b-2092ae9f7e13"}]: dispatch 2023-07-14T19:11:02.972 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:11:02 smithi040 ceph-mon[96518]: from='client.? 172.21.15.40:0/405477386' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "0d79356a-95d2-49ad-857b-2092ae9f7e13"}]': finished 2023-07-14T19:11:02.972 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:11:02 smithi040 ceph-mon[96518]: osdmap e5: 1 total, 0 up, 1 in 2023-07-14T19:11:02.972 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:11:02 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-07-14T19:11:02.972 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:11:02 smithi040 ceph-mon[96518]: Standby manager daemon y started 2023-07-14T19:11:02.972 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:11:02 smithi040 ceph-mon[96518]: from='mgr.? 172.21.15.130:0/3536911733' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2023-07-14T19:11:02.972 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:11:02 smithi040 ceph-mon[96518]: from='mgr.? 172.21.15.130:0/3536911733' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2023-07-14T19:11:02.972 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:11:02 smithi040 ceph-mon[96518]: from='mgr.? 172.21.15.130:0/3536911733' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2023-07-14T19:11:02.973 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:11:02 smithi040 ceph-mon[96518]: from='mgr.? 172.21.15.130:0/3536911733' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2023-07-14T19:11:02.973 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:11:02 smithi040 ceph-mon[96518]: from='client.? 172.21.15.40:0/668310271' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-14T19:11:02.973 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:11:02 smithi191 ceph-mon[98311]: from='client.? 172.21.15.40:0/405477386' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0d79356a-95d2-49ad-857b-2092ae9f7e13"}]: dispatch 2023-07-14T19:11:02.973 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:11:02 smithi191 ceph-mon[98311]: from='client.? 172.21.15.40:0/405477386' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "0d79356a-95d2-49ad-857b-2092ae9f7e13"}]': finished 2023-07-14T19:11:02.974 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:11:02 smithi191 ceph-mon[98311]: osdmap e5: 1 total, 0 up, 1 in 2023-07-14T19:11:02.974 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:11:02 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-07-14T19:11:02.974 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:11:02 smithi191 ceph-mon[98311]: Standby manager daemon y started 2023-07-14T19:11:02.974 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:11:02 smithi191 ceph-mon[98311]: from='mgr.? 172.21.15.130:0/3536911733' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2023-07-14T19:11:02.974 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:11:02 smithi191 ceph-mon[98311]: from='mgr.? 172.21.15.130:0/3536911733' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2023-07-14T19:11:02.974 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:11:02 smithi191 ceph-mon[98311]: from='mgr.? 172.21.15.130:0/3536911733' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2023-07-14T19:11:02.974 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:11:02 smithi191 ceph-mon[98311]: from='mgr.? 172.21.15.130:0/3536911733' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2023-07-14T19:11:02.974 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:11:02 smithi191 ceph-mon[98311]: from='client.? 172.21.15.40:0/668310271' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-14T19:11:02.992 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:11:02 smithi130 ceph-mon[97832]: from='client.? 172.21.15.40:0/405477386' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0d79356a-95d2-49ad-857b-2092ae9f7e13"}]: dispatch 2023-07-14T19:11:02.993 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:11:02 smithi130 ceph-mon[97832]: from='client.? 172.21.15.40:0/405477386' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "0d79356a-95d2-49ad-857b-2092ae9f7e13"}]': finished 2023-07-14T19:11:02.993 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:11:02 smithi130 ceph-mon[97832]: osdmap e5: 1 total, 0 up, 1 in 2023-07-14T19:11:02.993 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:11:02 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-07-14T19:11:02.993 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:11:02 smithi130 ceph-mon[97832]: Standby manager daemon y started 2023-07-14T19:11:02.993 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:11:02 smithi130 ceph-mon[97832]: from='mgr.? 172.21.15.130:0/3536911733' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2023-07-14T19:11:02.993 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:11:02 smithi130 ceph-mon[97832]: from='mgr.? 172.21.15.130:0/3536911733' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2023-07-14T19:11:02.994 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:11:02 smithi130 ceph-mon[97832]: from='mgr.? 172.21.15.130:0/3536911733' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2023-07-14T19:11:02.994 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:11:02 smithi130 ceph-mon[97832]: from='mgr.? 172.21.15.130:0/3536911733' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2023-07-14T19:11:02.994 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:11:02 smithi130 ceph-mon[97832]: from='client.? 172.21.15.40:0/668310271' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-14T19:11:03.971 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:11:03 smithi040 ceph-mon[96518]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T19:11:03.972 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:11:03 smithi040 ceph-mon[96518]: mgrmap e15: x(active, since 72s), standbys: z, y 2023-07-14T19:11:03.972 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:11:03 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2023-07-14T19:11:03.972 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:11:03 smithi191 ceph-mon[98311]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T19:11:03.972 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:11:03 smithi191 ceph-mon[98311]: mgrmap e15: x(active, since 72s), standbys: z, y 2023-07-14T19:11:03.973 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:11:03 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2023-07-14T19:11:03.992 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:11:03 smithi130 ceph-mon[97832]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T19:11:03.993 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:11:03 smithi130 ceph-mon[97832]: mgrmap e15: x(active, since 72s), standbys: z, y 2023-07-14T19:11:03.993 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:11:03 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2023-07-14T19:11:05.222 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:11:04 smithi040 ceph-mon[96518]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T19:11:05.222 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:11:04 smithi191 ceph-mon[98311]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T19:11:05.242 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:11:04 smithi130 ceph-mon[97832]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T19:11:07.722 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:11:07 smithi040 ceph-mon[96518]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T19:11:07.972 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:11:07 smithi191 ceph-mon[98311]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T19:11:07.992 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:11:07 smithi130 ceph-mon[97832]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T19:11:08.971 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:11:08 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2023-07-14T19:11:08.972 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:11:08 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:11:08.972 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:11:08 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2023-07-14T19:11:08.972 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:11:08 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:11:08.992 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:11:08 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2023-07-14T19:11:08.993 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:11:08 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:11:09.971 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:11:09 smithi040 ceph-mon[96518]: Deploying daemon osd.0 on smithi040 2023-07-14T19:11:09.972 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:11:09 smithi040 ceph-mon[96518]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T19:11:09.993 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:11:09 smithi130 ceph-mon[97832]: Deploying daemon osd.0 on smithi040 2023-07-14T19:11:09.993 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:11:09 smithi130 ceph-mon[97832]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T19:11:10.222 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:11:09 smithi191 ceph-mon[98311]: Deploying daemon osd.0 on smithi040 2023-07-14T19:11:10.223 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:11:09 smithi191 ceph-mon[98311]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T19:11:10.941 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:11:10 smithi040 ceph-mon[96518]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T19:11:11.222 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:11:10 smithi191 ceph-mon[98311]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T19:11:11.242 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:11:10 smithi130 ceph-mon[97832]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T19:11:13.169 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:11:12 smithi040 ceph-mon[96518]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T19:11:13.222 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:11:12 smithi191 ceph-mon[98311]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T19:11:13.242 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:11:12 smithi130 ceph-mon[97832]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T19:11:14.721 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:11:14 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:11:14.722 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:11:14 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:11:14.722 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:11:14 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T19:11:14.722 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:11:14 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:11:14.723 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:11:14 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:11:14.723 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:11:14 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T19:11:14.742 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:11:14 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:11:14.743 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:11:14 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:11:14.743 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:11:14 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T19:11:15.510 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:11:15 smithi040 ceph-mon[96518]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T19:11:15.722 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:11:15 smithi191 ceph-mon[98311]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T19:11:15.742 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:11:15 smithi130 ceph-mon[97832]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T19:11:16.480 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:11:16.479+0000 7f02f77fe700 1 -- 172.21.15.40:0/2975404528 <== mgr.14152 v2:172.21.15.40:6800/1693027118 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f0300060a80 con 0x7f02ec0389e0 2023-07-14T19:11:16.481 INFO:teuthology.orchestra.run.smithi040.stdout:Created osd(s) 0 on host 'smithi040' 2023-07-14T19:11:16.483 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:11:16.481+0000 7f0304cad700 1 -- 172.21.15.40:0/2975404528 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f02ec0389e0 msgr2=0x7f02ec03ae90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:11:16.483 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:11:16.481+0000 7f0304cad700 1 --2- 172.21.15.40:0/2975404528 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f02ec0389e0 0x7f02ec03ae90 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f02f000b2e0 tx=0x7f02f0005fb0 comp rx=0 tx=0).stop 2023-07-14T19:11:16.483 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:11:16.481+0000 7f0304cad700 1 -- 172.21.15.40:0/2975404528 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0300100740 msgr2=0x7f030019cdc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:11:16.483 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:11:16.481+0000 7f0304cad700 1 --2- 172.21.15.40:0/2975404528 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0300100740 0x7f030019cdc0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f02f8007ee0 tx=0x7f02f800d3b0 comp rx=0 tx=0).stop 2023-07-14T19:11:16.483 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:11:16.481+0000 7f0304cad700 1 -- 172.21.15.40:0/2975404528 shutdown_connections 2023-07-14T19:11:16.483 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:11:16.481+0000 7f0304cad700 1 --2- 172.21.15.40:0/2975404528 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f02ec0389e0 0x7f02ec03ae90 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:11:16.484 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:11:16.481+0000 7f0304cad700 1 --2- 172.21.15.40:0/2975404528 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f03000febb0 0x7f0300068c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:11:16.484 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:11:16.481+0000 7f0304cad700 1 --2- 172.21.15.40:0/2975404528 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0300100740 0x7f030019cdc0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:11:16.484 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:11:16.481+0000 7f0304cad700 1 --2- 172.21.15.40:0/2975404528 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f03000ffdb0 0x7f030019a770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:11:16.484 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:11:16.481+0000 7f0304cad700 1 -- 172.21.15.40:0/2975404528 >> 172.21.15.40:0/2975404528 conn(0x7f03000fa1e0 msgr2=0x7f0300103330 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:11:16.484 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:11:16.481+0000 7f0304cad700 1 -- 172.21.15.40:0/2975404528 shutdown_connections 2023-07-14T19:11:16.484 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:11:16.481+0000 7f0304cad700 1 -- 172.21.15.40:0/2975404528 wait complete. 2023-07-14T19:11:16.725 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:11:16 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:11:16.725 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:11:16 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:11:16.726 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:11:16 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:11:16.726 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:11:16 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T19:11:16.726 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:11:16 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:11:16.727 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:11:16 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:11:16.727 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:11:16 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:11:16.727 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:11:16 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:11:16.727 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:11:16 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T19:11:16.728 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:11:16 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:11:16.743 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:11:16 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:11:16.743 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:11:16 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:11:16.743 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:11:16 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:11:16.743 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:11:16 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T19:11:16.744 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:11:16 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:11:16.930 DEBUG:teuthology.orchestra.run.smithi040:osd.0> sudo journalctl -f -n 0 -u ceph-adc52f04-2279-11ee-9b33-001a4aab830c@osd.0.service 2023-07-14T19:11:16.933 INFO:tasks.cephadm:Deploying osd.1 on smithi130 with /dev/vg_nvme/lv_4... 2023-07-14T19:11:16.933 DEBUG:teuthology.orchestra.run.smithi130:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid adc52f04-2279-11ee-9b33-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2023-07-14T19:11:16.965 INFO:journalctl@ceph.osd.0.smithi040.stdout:-- Logs begin at Fri 2023-07-14 18:50:26 UTC. -- 2023-07-14T19:11:17.495 INFO:teuthology.orchestra.run.smithi130.stderr:Inferring config /var/lib/ceph/adc52f04-2279-11ee-9b33-001a4aab830c/mon.b/config 2023-07-14T19:11:17.611 INFO:journalctl@ceph.osd.0.smithi040.stdout:Jul 14 19:11:17 smithi040 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-0[109451]: 2023-07-14T19:11:17.331+0000 7f7283913640 -1 osd.0 0 log_to_monitors true 2023-07-14T19:11:17.611 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:11:17 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:11:17.611 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:11:17 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:11:17.611 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:11:17 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T19:11:17.612 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:11:17 smithi040 ceph-mon[96518]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T19:11:17.612 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:11:17 smithi040 ceph-mon[96518]: from='osd.0 [v2:172.21.15.40:6802/3599703125,v1:172.21.15.40:6803/3599703125]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2023-07-14T19:11:17.721 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:11:17 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:11:17.721 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:11:17 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:11:17.721 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:11:17 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T19:11:17.721 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:11:17 smithi130 ceph-mon[97832]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T19:11:17.722 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:11:17 smithi130 ceph-mon[97832]: from='osd.0 [v2:172.21.15.40:6802/3599703125,v1:172.21.15.40:6803/3599703125]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2023-07-14T19:11:17.722 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:11:17 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:11:17.722 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:11:17 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:11:17.722 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:11:17 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T19:11:17.723 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:11:17 smithi191 ceph-mon[98311]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T19:11:17.723 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:11:17 smithi191 ceph-mon[98311]: from='osd.0 [v2:172.21.15.40:6802/3599703125,v1:172.21.15.40:6803/3599703125]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2023-07-14T19:11:18.596 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:11:18 smithi130 ceph-mon[97832]: from='osd.0 [v2:172.21.15.40:6802/3599703125,v1:172.21.15.40:6803/3599703125]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2023-07-14T19:11:18.596 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:11:18 smithi130 ceph-mon[97832]: osdmap e6: 1 total, 0 up, 1 in 2023-07-14T19:11:18.596 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:11:18 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-07-14T19:11:18.596 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:11:18 smithi130 ceph-mon[97832]: from='osd.0 [v2:172.21.15.40:6802/3599703125,v1:172.21.15.40:6803/3599703125]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi040", "root=default"]}]: dispatch 2023-07-14T19:11:18.764 INFO:journalctl@ceph.osd.0.smithi040.stdout:Jul 14 19:11:18 smithi040 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-0[109451]: 2023-07-14T19:11:18.504+0000 7f7278f8b700 -1 osd.0 0 waiting for initial osdmap 2023-07-14T19:11:18.765 INFO:journalctl@ceph.osd.0.smithi040.stdout:Jul 14 19:11:18 smithi040 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-0[109451]: 2023-07-14T19:11:18.508+0000 7f7274580700 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-07-14T19:11:18.765 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:11:18 smithi040 ceph-mon[96518]: from='osd.0 [v2:172.21.15.40:6802/3599703125,v1:172.21.15.40:6803/3599703125]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2023-07-14T19:11:18.765 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:11:18 smithi040 ceph-mon[96518]: osdmap e6: 1 total, 0 up, 1 in 2023-07-14T19:11:18.765 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:11:18 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-07-14T19:11:18.765 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:11:18 smithi040 ceph-mon[96518]: from='osd.0 [v2:172.21.15.40:6802/3599703125,v1:172.21.15.40:6803/3599703125]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi040", "root=default"]}]: dispatch 2023-07-14T19:11:18.972 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:11:18 smithi191 ceph-mon[98311]: from='osd.0 [v2:172.21.15.40:6802/3599703125,v1:172.21.15.40:6803/3599703125]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2023-07-14T19:11:18.973 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:11:18 smithi191 ceph-mon[98311]: osdmap e6: 1 total, 0 up, 1 in 2023-07-14T19:11:18.973 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:11:18 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-07-14T19:11:18.973 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:11:18 smithi191 ceph-mon[98311]: from='osd.0 [v2:172.21.15.40:6802/3599703125,v1:172.21.15.40:6803/3599703125]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi040", "root=default"]}]: dispatch 2023-07-14T19:11:19.094 INFO:teuthology.orchestra.run.smithi130.stderr:--> Zapping: /dev/vg_nvme/lv_4 2023-07-14T19:11:19.095 INFO:teuthology.orchestra.run.smithi130.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2023-07-14T19:11:19.095 INFO:teuthology.orchestra.run.smithi130.stderr: stderr: 10+0 records in 2023-07-14T19:11:19.095 INFO:teuthology.orchestra.run.smithi130.stderr:10+0 records out 2023-07-14T19:11:19.095 INFO:teuthology.orchestra.run.smithi130.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.010879 s, 964 MB/s 2023-07-14T19:11:19.095 INFO:teuthology.orchestra.run.smithi130.stderr:--> Zapping successful for: 2023-07-14T19:11:19.447 DEBUG:teuthology.orchestra.run.smithi130:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid adc52f04-2279-11ee-9b33-001a4aab830c -- ceph orch daemon add osd smithi130:vg_nvme/lv_4 2023-07-14T19:11:19.743 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:11:19 smithi130 ceph-mon[97832]: from='osd.0 [v2:172.21.15.40:6802/3599703125,v1:172.21.15.40:6803/3599703125]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi040", "root=default"]}]': finished 2023-07-14T19:11:19.743 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:11:19 smithi130 ceph-mon[97832]: osdmap e7: 1 total, 0 up, 1 in 2023-07-14T19:11:19.743 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:11:19 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-07-14T19:11:19.743 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:11:19 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-07-14T19:11:19.743 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:11:19 smithi130 ceph-mon[97832]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T19:11:19.744 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:11:19 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:11:19.744 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:11:19 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:11:19.744 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:11:19 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-07-14T19:11:19.744 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:11:19 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:11:19.744 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:11:19 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:11:19.744 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:11:19 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T19:11:19.744 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:11:19 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:11:19.971 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:11:19 smithi040 ceph-mon[96518]: from='osd.0 [v2:172.21.15.40:6802/3599703125,v1:172.21.15.40:6803/3599703125]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi040", "root=default"]}]': finished 2023-07-14T19:11:19.972 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:11:19 smithi040 ceph-mon[96518]: osdmap e7: 1 total, 0 up, 1 in 2023-07-14T19:11:19.972 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:11:19 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-07-14T19:11:19.972 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:11:19 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-07-14T19:11:19.972 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:11:19 smithi040 ceph-mon[96518]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T19:11:19.973 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:11:19 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:11:19.973 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:11:19 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:11:19.973 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:11:19 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-07-14T19:11:19.973 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:11:19 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:11:19.973 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:11:19 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:11:19.973 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:11:19 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T19:11:19.973 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:11:19 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:11:19.974 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:11:19 smithi191 ceph-mon[98311]: from='osd.0 [v2:172.21.15.40:6802/3599703125,v1:172.21.15.40:6803/3599703125]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi040", "root=default"]}]': finished 2023-07-14T19:11:19.974 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:11:19 smithi191 ceph-mon[98311]: osdmap e7: 1 total, 0 up, 1 in 2023-07-14T19:11:19.974 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:11:19 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-07-14T19:11:19.974 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:11:19 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-07-14T19:11:19.974 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:11:19 smithi191 ceph-mon[98311]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T19:11:19.975 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:11:19 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:11:19.975 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:11:19 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:11:19.975 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:11:19 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-07-14T19:11:19.975 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:11:19 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:11:19.975 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:11:19 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:11:19.975 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:11:19 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T19:11:19.975 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:11:19 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:11:20.001 INFO:teuthology.orchestra.run.smithi130.stderr:Inferring config /var/lib/ceph/adc52f04-2279-11ee-9b33-001a4aab830c/mon.b/config 2023-07-14T19:11:20.970 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:11:20 smithi130 ceph-mon[97832]: purged_snaps scrub starts 2023-07-14T19:11:20.971 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:11:20 smithi130 ceph-mon[97832]: purged_snaps scrub ok 2023-07-14T19:11:20.971 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:11:20 smithi130 ceph-mon[97832]: Detected new or changed devices on smithi040 2023-07-14T19:11:20.971 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:11:20 smithi130 ceph-mon[97832]: Adjusting osd_memory_target on smithi040 to 17173M 2023-07-14T19:11:20.971 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:11:20 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-07-14T19:11:20.971 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:11:20 smithi130 ceph-mon[97832]: osd.0 [v2:172.21.15.40:6802/3599703125,v1:172.21.15.40:6803/3599703125] boot 2023-07-14T19:11:20.971 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:11:20 smithi130 ceph-mon[97832]: osdmap e8: 1 total, 1 up, 1 in 2023-07-14T19:11:20.972 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:11:20 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-07-14T19:11:20.972 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:11:20 smithi191 ceph-mon[98311]: purged_snaps scrub starts 2023-07-14T19:11:20.972 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:11:20 smithi191 ceph-mon[98311]: purged_snaps scrub ok 2023-07-14T19:11:20.973 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:11:20 smithi191 ceph-mon[98311]: Detected new or changed devices on smithi040 2023-07-14T19:11:20.973 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:11:20 smithi191 ceph-mon[98311]: Adjusting osd_memory_target on smithi040 to 17173M 2023-07-14T19:11:20.973 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:11:20 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-07-14T19:11:20.973 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:11:20 smithi191 ceph-mon[98311]: osd.0 [v2:172.21.15.40:6802/3599703125,v1:172.21.15.40:6803/3599703125] boot 2023-07-14T19:11:20.973 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:11:20 smithi191 ceph-mon[98311]: osdmap e8: 1 total, 1 up, 1 in 2023-07-14T19:11:20.973 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:11:20 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-07-14T19:11:20.973 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:11:20 smithi040 ceph-mon[96518]: purged_snaps scrub starts 2023-07-14T19:11:20.974 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:11:20 smithi040 ceph-mon[96518]: purged_snaps scrub ok 2023-07-14T19:11:20.974 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:11:20 smithi040 ceph-mon[96518]: Detected new or changed devices on smithi040 2023-07-14T19:11:20.974 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:11:20 smithi040 ceph-mon[96518]: Adjusting osd_memory_target on smithi040 to 17173M 2023-07-14T19:11:20.974 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:11:20 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-07-14T19:11:20.974 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:11:20 smithi040 ceph-mon[96518]: osd.0 [v2:172.21.15.40:6802/3599703125,v1:172.21.15.40:6803/3599703125] boot 2023-07-14T19:11:20.974 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:11:20 smithi040 ceph-mon[96518]: osdmap e8: 1 total, 1 up, 1 in 2023-07-14T19:11:20.974 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:11:20 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-07-14T19:11:21.173 INFO:teuthology.orchestra.run.smithi130.stderr:2023-07-14T19:11:21.169+0000 7f501561d700 1 -- 172.21.15.130:0/1007339062 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f50100fe3f0 msgr2=0x7f5010108290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:11:21.174 INFO:teuthology.orchestra.run.smithi130.stderr:2023-07-14T19:11:21.169+0000 7f501561d700 1 --2- 172.21.15.130:0/1007339062 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f50100fe3f0 0x7f5010108290 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f500800b0a0 tx=0x7f500800b3b0 comp rx=0 tx=0).stop 2023-07-14T19:11:21.174 INFO:teuthology.orchestra.run.smithi130.stderr:2023-07-14T19:11:21.169+0000 7f501561d700 1 -- 172.21.15.130:0/1007339062 shutdown_connections 2023-07-14T19:11:21.174 INFO:teuthology.orchestra.run.smithi130.stderr:2023-07-14T19:11:21.169+0000 7f501561d700 1 --2- 172.21.15.130:0/1007339062 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f50100fe3f0 0x7f5010108290 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:11:21.174 INFO:teuthology.orchestra.run.smithi130.stderr:2023-07-14T19:11:21.169+0000 7f501561d700 1 --2- 172.21.15.130:0/1007339062 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f50101012a0 0x7f50101016b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:11:21.174 INFO:teuthology.orchestra.run.smithi130.stderr:2023-07-14T19:11:21.169+0000 7f501561d700 1 --2- 172.21.15.130:0/1007339062 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5010101c80 0x7f50100fdd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:11:21.174 INFO:teuthology.orchestra.run.smithi130.stderr:2023-07-14T19:11:21.169+0000 7f501561d700 1 -- 172.21.15.130:0/1007339062 >> 172.21.15.130:0/1007339062 conn(0x7f50100f9990 msgr2=0x7f50100fbdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:11:21.174 INFO:teuthology.orchestra.run.smithi130.stderr:2023-07-14T19:11:21.169+0000 7f501561d700 1 -- 172.21.15.130:0/1007339062 shutdown_connections 2023-07-14T19:11:21.175 INFO:teuthology.orchestra.run.smithi130.stderr:2023-07-14T19:11:21.169+0000 7f501561d700 1 -- 172.21.15.130:0/1007339062 wait complete. 2023-07-14T19:11:21.175 INFO:teuthology.orchestra.run.smithi130.stderr:2023-07-14T19:11:21.170+0000 7f501561d700 1 Processor -- start 2023-07-14T19:11:21.175 INFO:teuthology.orchestra.run.smithi130.stderr:2023-07-14T19:11:21.170+0000 7f501561d700 1 -- start start 2023-07-14T19:11:21.175 INFO:teuthology.orchestra.run.smithi130.stderr:2023-07-14T19:11:21.170+0000 7f501561d700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f50100fe3f0 0x7f5010195b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:11:21.175 INFO:teuthology.orchestra.run.smithi130.stderr:2023-07-14T19:11:21.170+0000 7f501561d700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f50101012a0 0x7f5010196090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:11:21.175 INFO:teuthology.orchestra.run.smithi130.stderr:2023-07-14T19:11:21.170+0000 7f501561d700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5010101c80 0x7f501019d110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:11:21.175 INFO:teuthology.orchestra.run.smithi130.stderr:2023-07-14T19:11:21.170+0000 7f501561d700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f501019d6e0 con 0x7f50101012a0 2023-07-14T19:11:21.175 INFO:teuthology.orchestra.run.smithi130.stderr:2023-07-14T19:11:21.170+0000 7f501561d700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f501019d820 con 0x7f5010101c80 2023-07-14T19:11:21.176 INFO:teuthology.orchestra.run.smithi130.stderr:2023-07-14T19:11:21.170+0000 7f501561d700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f501019d9b0 con 0x7f50100fe3f0 2023-07-14T19:11:21.176 INFO:teuthology.orchestra.run.smithi130.stderr:2023-07-14T19:11:21.170+0000 7f500f7fe700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5010101c80 0x7f501019d110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:11:21.176 INFO:teuthology.orchestra.run.smithi130.stderr:2023-07-14T19:11:21.170+0000 7f500f7fe700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5010101c80 0x7f501019d110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.130:39564/0 (socket says 172.21.15.130:39564) 2023-07-14T19:11:21.176 INFO:teuthology.orchestra.run.smithi130.stderr:2023-07-14T19:11:21.170+0000 7f500f7fe700 1 -- 172.21.15.130:0/2508417073 learned_addr learned my addr 172.21.15.130:0/2508417073 (peer_addr_for_me v2:172.21.15.130:0/0) 2023-07-14T19:11:21.176 INFO:teuthology.orchestra.run.smithi130.stderr:2023-07-14T19:11:21.170+0000 7f500e7fc700 1 --2- 172.21.15.130:0/2508417073 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f50101012a0 0x7f5010196090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:11:21.176 INFO:teuthology.orchestra.run.smithi130.stderr:2023-07-14T19:11:21.170+0000 7f500effd700 1 --2- 172.21.15.130:0/2508417073 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f50100fe3f0 0x7f5010195b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:11:21.176 INFO:teuthology.orchestra.run.smithi130.stderr:2023-07-14T19:11:21.170+0000 7f500f7fe700 1 -- 172.21.15.130:0/2508417073 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f50100fe3f0 msgr2=0x7f5010195b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:11:21.177 INFO:teuthology.orchestra.run.smithi130.stderr:2023-07-14T19:11:21.170+0000 7f500f7fe700 1 --2- 172.21.15.130:0/2508417073 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f50100fe3f0 0x7f5010195b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:11:21.177 INFO:teuthology.orchestra.run.smithi130.stderr:2023-07-14T19:11:21.170+0000 7f500f7fe700 1 -- 172.21.15.130:0/2508417073 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f50101012a0 msgr2=0x7f5010196090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:11:21.177 INFO:teuthology.orchestra.run.smithi130.stderr:2023-07-14T19:11:21.170+0000 7f500f7fe700 1 --2- 172.21.15.130:0/2508417073 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f50101012a0 0x7f5010196090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:11:21.177 INFO:teuthology.orchestra.run.smithi130.stderr:2023-07-14T19:11:21.170+0000 7f500f7fe700 1 -- 172.21.15.130:0/2508417073 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5008009d00 con 0x7f5010101c80 2023-07-14T19:11:21.177 INFO:teuthology.orchestra.run.smithi130.stderr:2023-07-14T19:11:21.170+0000 7f500effd700 1 --2- 172.21.15.130:0/2508417073 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f50100fe3f0 0x7f5010195b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-14T19:11:21.177 INFO:teuthology.orchestra.run.smithi130.stderr:2023-07-14T19:11:21.171+0000 7f500f7fe700 1 --2- 172.21.15.130:0/2508417073 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5010101c80 0x7f501019d110 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f50080062a0 tx=0x7f5008004db0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:11:21.177 INFO:teuthology.orchestra.run.smithi130.stderr:2023-07-14T19:11:21.171+0000 7f4ff7fff700 1 -- 172.21.15.130:0/2508417073 <== mon.2 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f500800e040 con 0x7f5010101c80 2023-07-14T19:11:21.177 INFO:teuthology.orchestra.run.smithi130.stderr:2023-07-14T19:11:21.171+0000 7f4ff7fff700 1 -- 172.21.15.130:0/2508417073 <== mon.2 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f500801db50 con 0x7f5010101c80 2023-07-14T19:11:21.178 INFO:teuthology.orchestra.run.smithi130.stderr:2023-07-14T19:11:21.171+0000 7f501561d700 1 -- 172.21.15.130:0/2508417073 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f501019dc30 con 0x7f5010101c80 2023-07-14T19:11:21.178 INFO:teuthology.orchestra.run.smithi130.stderr:2023-07-14T19:11:21.171+0000 7f4ff7fff700 1 -- 172.21.15.130:0/2508417073 <== mon.2 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f500801bc30 con 0x7f5010101c80 2023-07-14T19:11:21.178 INFO:teuthology.orchestra.run.smithi130.stderr:2023-07-14T19:11:21.171+0000 7f501561d700 1 -- 172.21.15.130:0/2508417073 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f501019e0f0 con 0x7f5010101c80 2023-07-14T19:11:21.179 INFO:teuthology.orchestra.run.smithi130.stderr:2023-07-14T19:11:21.172+0000 7f501561d700 1 -- 172.21.15.130:0/2508417073 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f501010b550 con 0x7f5010101c80 2023-07-14T19:11:21.179 INFO:teuthology.orchestra.run.smithi130.stderr:2023-07-14T19:11:21.173+0000 7f4ff7fff700 1 -- 172.21.15.130:0/2508417073 <== mon.2 v2:172.21.15.130:3300/0 4 ==== mgrmap(e 15) v1 ==== 135659+0+0 (secure 0 0 0) 0x7f500801d690 con 0x7f5010101c80 2023-07-14T19:11:21.179 INFO:teuthology.orchestra.run.smithi130.stderr:2023-07-14T19:11:21.173+0000 7f4ff7fff700 1 --2- 172.21.15.130:0/2508417073 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f4ff80a1140 0x7f4ff80a35f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:11:21.179 INFO:teuthology.orchestra.run.smithi130.stderr:2023-07-14T19:11:21.173+0000 7f4ff7fff700 1 -- 172.21.15.130:0/2508417073 <== mon.2 v2:172.21.15.130:3300/0 5 ==== osd_map(9..9 src has 1..9) v4 ==== 1767+0+0 (secure 0 0 0) 0x7f50080cca40 con 0x7f5010101c80 2023-07-14T19:11:21.179 INFO:teuthology.orchestra.run.smithi130.stderr:2023-07-14T19:11:21.174+0000 7f500effd700 1 --2- 172.21.15.130:0/2508417073 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f4ff80a1140 0x7f4ff80a35f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:11:21.180 INFO:teuthology.orchestra.run.smithi130.stderr:2023-07-14T19:11:21.174+0000 7f500effd700 1 --2- 172.21.15.130:0/2508417073 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f4ff80a1140 0x7f4ff80a35f0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f5010067e20 tx=0x7f5004008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:11:21.180 INFO:teuthology.orchestra.run.smithi130.stderr:2023-07-14T19:11:21.178+0000 7f4ff7fff700 1 -- 172.21.15.130:0/2508417073 <== mon.2 v2:172.21.15.130:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f5008097c90 con 0x7f5010101c80 2023-07-14T19:11:21.372 INFO:teuthology.orchestra.run.smithi130.stderr:2023-07-14T19:11:21.370+0000 7f501561d700 1 -- 172.21.15.130:0/2508417073 --> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi130:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) v1 -- 0x7f5010060a80 con 0x7f4ff80a1140 2023-07-14T19:11:21.971 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:11:21 smithi040 ceph-mon[96518]: osdmap e9: 1 total, 1 up, 1 in 2023-07-14T19:11:21.972 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:11:21 smithi040 ceph-mon[96518]: pgmap v44: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-14T19:11:21.972 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:11:21 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-14T19:11:21.972 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:11:21 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-14T19:11:21.972 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:11:21 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:11:21.973 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:11:21 smithi191 ceph-mon[98311]: osdmap e9: 1 total, 1 up, 1 in 2023-07-14T19:11:21.973 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:11:21 smithi191 ceph-mon[98311]: pgmap v44: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-14T19:11:21.973 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:11:21 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-14T19:11:21.973 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:11:21 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-14T19:11:21.973 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:11:21 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:11:21.992 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:11:21 smithi130 ceph-mon[97832]: osdmap e9: 1 total, 1 up, 1 in 2023-07-14T19:11:21.993 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:11:21 smithi130 ceph-mon[97832]: pgmap v44: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-14T19:11:21.993 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:11:21 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-14T19:11:21.993 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:11:21 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-14T19:11:21.993 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:11:21 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:11:22.742 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:11:22 smithi130 ceph-mon[97832]: from='client.24146 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi130:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T19:11:22.971 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:11:22 smithi040 ceph-mon[96518]: from='client.24146 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi130:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T19:11:22.972 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:11:22 smithi191 ceph-mon[98311]: from='client.24146 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi130:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T19:11:23.971 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:11:23 smithi040 ceph-mon[96518]: pgmap v45: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-14T19:11:23.972 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:11:23 smithi191 ceph-mon[98311]: pgmap v45: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-14T19:11:23.993 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:11:23 smithi130 ceph-mon[97832]: pgmap v45: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-14T19:11:24.743 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:11:24 smithi130 ceph-mon[97832]: from='client.? 172.21.15.130:0/1631931806' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "40f8a3d1-a5e4-473c-b9f2-7600ec22be0d"}]: dispatch 2023-07-14T19:11:24.743 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:11:24 smithi130 ceph-mon[97832]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "40f8a3d1-a5e4-473c-b9f2-7600ec22be0d"}]: dispatch 2023-07-14T19:11:24.744 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:11:24 smithi130 ceph-mon[97832]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "40f8a3d1-a5e4-473c-b9f2-7600ec22be0d"}]': finished 2023-07-14T19:11:24.744 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:11:24 smithi130 ceph-mon[97832]: osdmap e10: 2 total, 1 up, 2 in 2023-07-14T19:11:24.744 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:11:24 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-07-14T19:11:24.744 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:11:24 smithi130 ceph-mon[97832]: from='client.? 172.21.15.130:0/220509930' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-14T19:11:24.971 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:11:24 smithi040 ceph-mon[96518]: from='client.? 172.21.15.130:0/1631931806' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "40f8a3d1-a5e4-473c-b9f2-7600ec22be0d"}]: dispatch 2023-07-14T19:11:24.972 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:11:24 smithi040 ceph-mon[96518]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "40f8a3d1-a5e4-473c-b9f2-7600ec22be0d"}]: dispatch 2023-07-14T19:11:24.972 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:11:24 smithi040 ceph-mon[96518]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "40f8a3d1-a5e4-473c-b9f2-7600ec22be0d"}]': finished 2023-07-14T19:11:24.972 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:11:24 smithi040 ceph-mon[96518]: osdmap e10: 2 total, 1 up, 2 in 2023-07-14T19:11:24.972 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:11:24 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-07-14T19:11:24.972 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:11:24 smithi040 ceph-mon[96518]: from='client.? 172.21.15.130:0/220509930' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-14T19:11:24.973 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:11:24 smithi191 ceph-mon[98311]: from='client.? 172.21.15.130:0/1631931806' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "40f8a3d1-a5e4-473c-b9f2-7600ec22be0d"}]: dispatch 2023-07-14T19:11:24.973 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:11:24 smithi191 ceph-mon[98311]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "40f8a3d1-a5e4-473c-b9f2-7600ec22be0d"}]: dispatch 2023-07-14T19:11:24.973 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:11:24 smithi191 ceph-mon[98311]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "40f8a3d1-a5e4-473c-b9f2-7600ec22be0d"}]': finished 2023-07-14T19:11:24.973 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:11:24 smithi191 ceph-mon[98311]: osdmap e10: 2 total, 1 up, 2 in 2023-07-14T19:11:24.973 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:11:24 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-07-14T19:11:24.973 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:11:24 smithi191 ceph-mon[98311]: from='client.? 172.21.15.130:0/220509930' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-14T19:11:25.971 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:11:25 smithi040 ceph-mon[96518]: pgmap v47: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-14T19:11:25.972 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:11:25 smithi191 ceph-mon[98311]: pgmap v47: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-14T19:11:25.993 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:11:25 smithi130 ceph-mon[97832]: pgmap v47: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-14T19:11:27.720 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:11:27 smithi130 ceph-mon[97832]: pgmap v48: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-14T19:11:27.971 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:11:27 smithi040 ceph-mon[96518]: pgmap v48: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-14T19:11:27.972 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:11:27 smithi191 ceph-mon[98311]: pgmap v48: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-14T19:11:28.896 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:11:28 smithi130 ceph-mon[97832]: pgmap v49: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-14T19:11:29.221 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:11:28 smithi040 ceph-mon[96518]: pgmap v49: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-14T19:11:29.222 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:11:28 smithi191 ceph-mon[98311]: pgmap v49: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-14T19:11:30.492 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:11:30 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2023-07-14T19:11:30.493 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:11:30 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:11:30.721 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:11:30 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2023-07-14T19:11:30.721 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:11:30 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:11:30.722 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:11:30 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2023-07-14T19:11:30.722 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:11:30 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:11:31.493 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:11:31 smithi130 ceph-mon[97832]: Deploying daemon osd.1 on smithi130 2023-07-14T19:11:31.493 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:11:31 smithi130 ceph-mon[97832]: pgmap v50: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-14T19:11:31.721 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:11:31 smithi040 ceph-mon[96518]: Deploying daemon osd.1 on smithi130 2023-07-14T19:11:31.722 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:11:31 smithi040 ceph-mon[96518]: pgmap v50: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-14T19:11:31.722 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:11:31 smithi191 ceph-mon[98311]: Deploying daemon osd.1 on smithi130 2023-07-14T19:11:31.722 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:11:31 smithi191 ceph-mon[98311]: pgmap v50: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-14T19:11:33.221 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:11:32 smithi040 ceph-mon[96518]: pgmap v51: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-14T19:11:33.222 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:11:32 smithi191 ceph-mon[98311]: pgmap v51: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-14T19:11:33.243 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:11:32 smithi130 ceph-mon[97832]: pgmap v51: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-14T19:11:34.893 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:11:34 smithi130 ceph-mon[97832]: pgmap v52: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-14T19:11:35.221 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:11:34 smithi040 ceph-mon[96518]: pgmap v52: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-14T19:11:35.222 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:11:34 smithi191 ceph-mon[98311]: pgmap v52: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-14T19:11:36.061 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:11:35 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:11:36.062 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:11:35 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:11:36.062 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:11:35 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T19:11:36.221 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:11:35 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:11:36.222 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:11:35 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:11:36.222 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:11:35 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T19:11:36.222 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:11:35 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:11:36.222 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:11:35 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:11:36.222 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:11:35 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T19:11:37.688 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:11:37 smithi130 ceph-mon[97832]: pgmap v53: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-14T19:11:37.689 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:11:37 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:11:37.689 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:11:37 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:11:37.689 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:11:37 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:11:37.689 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:11:37 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T19:11:37.689 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:11:37 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:11:37.737 INFO:teuthology.orchestra.run.smithi130.stderr:2023-07-14T19:11:37.735+0000 7f4ff7fff700 1 -- 172.21.15.130:0/2508417073 <== mgr.14152 v2:172.21.15.40:6800/1693027118 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f5010060a80 con 0x7f4ff80a1140 2023-07-14T19:11:37.737 INFO:teuthology.orchestra.run.smithi130.stdout:Created osd(s) 1 on host 'smithi130' 2023-07-14T19:11:37.739 INFO:teuthology.orchestra.run.smithi130.stderr:2023-07-14T19:11:37.736+0000 7f501561d700 1 -- 172.21.15.130:0/2508417073 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f4ff80a1140 msgr2=0x7f4ff80a35f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:11:37.739 INFO:teuthology.orchestra.run.smithi130.stderr:2023-07-14T19:11:37.736+0000 7f501561d700 1 --2- 172.21.15.130:0/2508417073 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f4ff80a1140 0x7f4ff80a35f0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f5010067e20 tx=0x7f5004008040 comp rx=0 tx=0).stop 2023-07-14T19:11:37.739 INFO:teuthology.orchestra.run.smithi130.stderr:2023-07-14T19:11:37.736+0000 7f501561d700 1 -- 172.21.15.130:0/2508417073 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5010101c80 msgr2=0x7f501019d110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:11:37.739 INFO:teuthology.orchestra.run.smithi130.stderr:2023-07-14T19:11:37.736+0000 7f501561d700 1 --2- 172.21.15.130:0/2508417073 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5010101c80 0x7f501019d110 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f50080062a0 tx=0x7f5008004db0 comp rx=0 tx=0).stop 2023-07-14T19:11:37.740 INFO:teuthology.orchestra.run.smithi130.stderr:2023-07-14T19:11:37.736+0000 7f501561d700 1 -- 172.21.15.130:0/2508417073 shutdown_connections 2023-07-14T19:11:37.740 INFO:teuthology.orchestra.run.smithi130.stderr:2023-07-14T19:11:37.736+0000 7f501561d700 1 --2- 172.21.15.130:0/2508417073 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f4ff80a1140 0x7f4ff80a35f0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:11:37.740 INFO:teuthology.orchestra.run.smithi130.stderr:2023-07-14T19:11:37.736+0000 7f501561d700 1 --2- 172.21.15.130:0/2508417073 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f50100fe3f0 0x7f5010195b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:11:37.740 INFO:teuthology.orchestra.run.smithi130.stderr:2023-07-14T19:11:37.736+0000 7f501561d700 1 --2- 172.21.15.130:0/2508417073 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5010101c80 0x7f501019d110 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:11:37.740 INFO:teuthology.orchestra.run.smithi130.stderr:2023-07-14T19:11:37.736+0000 7f501561d700 1 --2- 172.21.15.130:0/2508417073 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f50101012a0 0x7f5010196090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:11:37.740 INFO:teuthology.orchestra.run.smithi130.stderr:2023-07-14T19:11:37.736+0000 7f501561d700 1 -- 172.21.15.130:0/2508417073 >> 172.21.15.130:0/2508417073 conn(0x7f50100f9990 msgr2=0x7f50100fbbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:11:37.740 INFO:teuthology.orchestra.run.smithi130.stderr:2023-07-14T19:11:37.737+0000 7f501561d700 1 -- 172.21.15.130:0/2508417073 shutdown_connections 2023-07-14T19:11:37.741 INFO:teuthology.orchestra.run.smithi130.stderr:2023-07-14T19:11:37.737+0000 7f501561d700 1 -- 172.21.15.130:0/2508417073 wait complete. 2023-07-14T19:11:37.971 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:11:37 smithi040 ceph-mon[96518]: pgmap v53: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-14T19:11:37.972 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:11:37 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:11:37.972 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:11:37 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:11:37.972 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:11:37 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:11:37.972 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:11:37 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T19:11:37.973 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:11:37 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:11:37.973 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:11:37 smithi191 ceph-mon[98311]: pgmap v53: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-14T19:11:37.973 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:11:37 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:11:37.973 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:11:37 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:11:37.974 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:11:37 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:11:37.974 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:11:37 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T19:11:37.974 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:11:37 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:11:38.180 DEBUG:teuthology.orchestra.run.smithi130:osd.1> sudo journalctl -f -n 0 -u ceph-adc52f04-2279-11ee-9b33-001a4aab830c@osd.1.service 2023-07-14T19:11:38.182 INFO:tasks.cephadm:Deploying osd.2 on smithi191 with /dev/vg_nvme/lv_4... 2023-07-14T19:11:38.183 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid adc52f04-2279-11ee-9b33-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2023-07-14T19:11:38.214 INFO:journalctl@ceph.osd.1.smithi130.stdout:-- Logs begin at Fri 2023-07-14 18:50:32 UTC. -- 2023-07-14T19:11:38.741 INFO:teuthology.orchestra.run.smithi191.stderr:Inferring config /var/lib/ceph/adc52f04-2279-11ee-9b33-001a4aab830c/mon.c/config 2023-07-14T19:11:38.953 INFO:journalctl@ceph.osd.1.smithi130.stdout:Jul 14 19:11:38 smithi130 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-1[102894]: 2023-07-14T19:11:38.849+0000 7f64de339640 -1 osd.1 0 log_to_monitors true 2023-07-14T19:11:38.953 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:11:38 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:11:38.954 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:11:38 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:11:38.954 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:11:38 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T19:11:38.971 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:11:38 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:11:38.972 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:11:38 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:11:38.972 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:11:38 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T19:11:38.972 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:11:38 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:11:38.972 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:11:38 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:11:38.972 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:11:38 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T19:11:39.993 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:11:39 smithi130 ceph-mon[97832]: pgmap v54: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-14T19:11:39.993 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:11:39 smithi130 ceph-mon[97832]: from='osd.1 [v2:172.21.15.130:6800/2512760702,v1:172.21.15.130:6801/2512760702]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2023-07-14T19:11:39.993 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:11:39 smithi130 ceph-mon[97832]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2023-07-14T19:11:40.221 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:11:39 smithi040 ceph-mon[96518]: pgmap v54: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-14T19:11:40.222 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:11:39 smithi040 ceph-mon[96518]: from='osd.1 [v2:172.21.15.130:6800/2512760702,v1:172.21.15.130:6801/2512760702]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2023-07-14T19:11:40.222 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:11:39 smithi040 ceph-mon[96518]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2023-07-14T19:11:40.223 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:11:39 smithi191 ceph-mon[98311]: pgmap v54: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-14T19:11:40.223 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:11:39 smithi191 ceph-mon[98311]: from='osd.1 [v2:172.21.15.130:6800/2512760702,v1:172.21.15.130:6801/2512760702]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2023-07-14T19:11:40.223 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:11:39 smithi191 ceph-mon[98311]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2023-07-14T19:11:40.322 INFO:teuthology.orchestra.run.smithi191.stderr:--> Zapping: /dev/vg_nvme/lv_4 2023-07-14T19:11:40.323 INFO:teuthology.orchestra.run.smithi191.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2023-07-14T19:11:40.323 INFO:teuthology.orchestra.run.smithi191.stderr: stderr: 10+0 records in 2023-07-14T19:11:40.323 INFO:teuthology.orchestra.run.smithi191.stderr:10+0 records out 2023-07-14T19:11:40.323 INFO:teuthology.orchestra.run.smithi191.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.012896 s, 813 MB/s 2023-07-14T19:11:40.323 INFO:teuthology.orchestra.run.smithi191.stderr:--> Zapping successful for: 2023-07-14T19:11:40.694 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid adc52f04-2279-11ee-9b33-001a4aab830c -- ceph orch daemon add osd smithi191:vg_nvme/lv_4 2023-07-14T19:11:40.972 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:11:40 smithi191 ceph-mon[98311]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2023-07-14T19:11:40.972 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:11:40 smithi191 ceph-mon[98311]: osdmap e11: 2 total, 1 up, 2 in 2023-07-14T19:11:40.973 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:11:40 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-07-14T19:11:40.973 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:11:40 smithi191 ceph-mon[98311]: from='osd.1 [v2:172.21.15.130:6800/2512760702,v1:172.21.15.130:6801/2512760702]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi130", "root=default"]}]: dispatch 2023-07-14T19:11:40.973 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:11:40 smithi191 ceph-mon[98311]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi130", "root=default"]}]: dispatch 2023-07-14T19:11:40.973 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:11:40 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:11:40.973 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:11:40 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:11:40.973 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:11:40 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-07-14T19:11:40.973 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:11:40 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:11:40.974 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:11:40 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:11:40.974 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:11:40 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T19:11:40.974 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:11:40 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:11:40.993 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:11:40 smithi130 ceph-mon[97832]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2023-07-14T19:11:40.993 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:11:40 smithi130 ceph-mon[97832]: osdmap e11: 2 total, 1 up, 2 in 2023-07-14T19:11:40.993 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:11:40 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-07-14T19:11:40.993 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:11:40 smithi130 ceph-mon[97832]: from='osd.1 [v2:172.21.15.130:6800/2512760702,v1:172.21.15.130:6801/2512760702]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi130", "root=default"]}]: dispatch 2023-07-14T19:11:40.993 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:11:40 smithi130 ceph-mon[97832]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi130", "root=default"]}]: dispatch 2023-07-14T19:11:40.994 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:11:40 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:11:40.994 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:11:40 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:11:40.994 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:11:40 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-07-14T19:11:40.994 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:11:40 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:11:40.994 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:11:40 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:11:40.994 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:11:40 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T19:11:40.994 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:11:40 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:11:40.995 INFO:journalctl@ceph.osd.1.smithi130.stdout:Jul 14 19:11:40 smithi130 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-1[102894]: 2023-07-14T19:11:40.743+0000 7f64d39b1700 -1 osd.1 0 waiting for initial osdmap 2023-07-14T19:11:40.995 INFO:journalctl@ceph.osd.1.smithi130.stdout:Jul 14 19:11:40 smithi130 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-1[102894]: 2023-07-14T19:11:40.746+0000 7f64cdfa4700 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-07-14T19:11:41.221 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:11:40 smithi040 ceph-mon[96518]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2023-07-14T19:11:41.222 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:11:40 smithi040 ceph-mon[96518]: osdmap e11: 2 total, 1 up, 2 in 2023-07-14T19:11:41.222 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:11:40 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-07-14T19:11:41.222 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:11:40 smithi040 ceph-mon[96518]: from='osd.1 [v2:172.21.15.130:6800/2512760702,v1:172.21.15.130:6801/2512760702]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi130", "root=default"]}]: dispatch 2023-07-14T19:11:41.222 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:11:40 smithi040 ceph-mon[96518]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi130", "root=default"]}]: dispatch 2023-07-14T19:11:41.222 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:11:40 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:11:41.222 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:11:40 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:11:41.223 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:11:40 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-07-14T19:11:41.223 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:11:40 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:11:41.223 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:11:40 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:11:41.223 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:11:40 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T19:11:41.223 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:11:40 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:11:41.242 INFO:teuthology.orchestra.run.smithi191.stderr:Inferring config /var/lib/ceph/adc52f04-2279-11ee-9b33-001a4aab830c/mon.c/config 2023-07-14T19:11:41.992 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:11:41 smithi130 ceph-mon[97832]: Detected new or changed devices on smithi130 2023-07-14T19:11:41.993 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:11:41 smithi130 ceph-mon[97832]: Adjusting osd_memory_target on smithi130 to 17168M 2023-07-14T19:11:41.993 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:11:41 smithi130 ceph-mon[97832]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi130", "root=default"]}]': finished 2023-07-14T19:11:41.993 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:11:41 smithi130 ceph-mon[97832]: osdmap e12: 2 total, 1 up, 2 in 2023-07-14T19:11:41.993 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:11:41 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-07-14T19:11:41.993 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:11:41 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-07-14T19:11:41.993 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:11:41 smithi130 ceph-mon[97832]: pgmap v57: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-14T19:11:42.035 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:11:41 smithi191 ceph-mon[98311]: Detected new or changed devices on smithi130 2023-07-14T19:11:42.035 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:11:41 smithi191 ceph-mon[98311]: Adjusting osd_memory_target on smithi130 to 17168M 2023-07-14T19:11:42.036 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:11:41 smithi191 ceph-mon[98311]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi130", "root=default"]}]': finished 2023-07-14T19:11:42.036 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:11:41 smithi191 ceph-mon[98311]: osdmap e12: 2 total, 1 up, 2 in 2023-07-14T19:11:42.036 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:11:41 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-07-14T19:11:42.036 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:11:41 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-07-14T19:11:42.036 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:11:41 smithi191 ceph-mon[98311]: pgmap v57: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-14T19:11:42.221 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:11:41 smithi040 ceph-mon[96518]: Detected new or changed devices on smithi130 2023-07-14T19:11:42.222 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:11:41 smithi040 ceph-mon[96518]: Adjusting osd_memory_target on smithi130 to 17168M 2023-07-14T19:11:42.222 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:11:41 smithi040 ceph-mon[96518]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi130", "root=default"]}]': finished 2023-07-14T19:11:42.222 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:11:41 smithi040 ceph-mon[96518]: osdmap e12: 2 total, 1 up, 2 in 2023-07-14T19:11:42.222 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:11:41 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-07-14T19:11:42.222 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:11:41 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-07-14T19:11:42.222 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:11:41 smithi040 ceph-mon[96518]: pgmap v57: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-14T19:11:42.529 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:11:42.526+0000 7fcc5188b700 1 -- 172.21.15.191:0/1931622640 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fcc4c0ff810 msgr2=0x7fcc4c1070c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:11:42.529 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:11:42.526+0000 7fcc5188b700 1 --2- 172.21.15.191:0/1931622640 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fcc4c0ff810 0x7fcc4c1070c0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fcc4400b0a0 tx=0x7fcc4400b3b0 comp rx=0 tx=0).stop 2023-07-14T19:11:42.529 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:11:42.526+0000 7fcc5188b700 1 -- 172.21.15.191:0/1931622640 shutdown_connections 2023-07-14T19:11:42.529 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:11:42.526+0000 7fcc5188b700 1 --2- 172.21.15.191:0/1931622640 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fcc4c0ff810 0x7fcc4c1070c0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:11:42.529 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:11:42.526+0000 7fcc5188b700 1 --2- 172.21.15.191:0/1931622640 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcc4c100290 0x7fcc4c1006a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:11:42.530 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:11:42.526+0000 7fcc5188b700 1 --2- 172.21.15.191:0/1931622640 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fcc4c0fee60 0x7fcc4c0ff2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:11:42.530 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:11:42.526+0000 7fcc5188b700 1 -- 172.21.15.191:0/1931622640 >> 172.21.15.191:0/1931622640 conn(0x7fcc4c0f99c0 msgr2=0x7fcc4c0fbe10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:11:42.530 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:11:42.526+0000 7fcc5188b700 1 -- 172.21.15.191:0/1931622640 shutdown_connections 2023-07-14T19:11:42.530 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:11:42.526+0000 7fcc5188b700 1 -- 172.21.15.191:0/1931622640 wait complete. 2023-07-14T19:11:42.532 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:11:42.527+0000 7fcc5188b700 1 Processor -- start 2023-07-14T19:11:42.532 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:11:42.527+0000 7fcc5188b700 1 -- start start 2023-07-14T19:11:42.532 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:11:42.527+0000 7fcc5188b700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fcc4c0fee60 0x7fcc4c194950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:11:42.532 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:11:42.527+0000 7fcc5188b700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fcc4c0ff810 0x7fcc4c194e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:11:42.532 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:11:42.527+0000 7fcc5188b700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcc4c100290 0x7fcc4c19bf10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:11:42.532 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:11:42.527+0000 7fcc5188b700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc4c19c4e0 con 0x7fcc4c100290 2023-07-14T19:11:42.533 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:11:42.527+0000 7fcc5188b700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc4c19c620 con 0x7fcc4c0fee60 2023-07-14T19:11:42.533 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:11:42.527+0000 7fcc5188b700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc4c19c7b0 con 0x7fcc4c0ff810 2023-07-14T19:11:42.533 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:11:42.527+0000 7fcc4a7fc700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fcc4c0ff810 0x7fcc4c194e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:11:42.533 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:11:42.528+0000 7fcc4a7fc700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fcc4c0ff810 0x7fcc4c194e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.191:59276/0 (socket says 172.21.15.191:59276) 2023-07-14T19:11:42.533 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:11:42.528+0000 7fcc4a7fc700 1 -- 172.21.15.191:0/1455878312 learned_addr learned my addr 172.21.15.191:0/1455878312 (peer_addr_for_me v2:172.21.15.191:0/0) 2023-07-14T19:11:42.533 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:11:42.528+0000 7fcc4affd700 1 --2- 172.21.15.191:0/1455878312 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fcc4c0fee60 0x7fcc4c194950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:11:42.533 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:11:42.528+0000 7fcc4b7fe700 1 --2- 172.21.15.191:0/1455878312 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcc4c100290 0x7fcc4c19bf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:11:42.534 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:11:42.528+0000 7fcc4a7fc700 1 -- 172.21.15.191:0/1455878312 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fcc4c0fee60 msgr2=0x7fcc4c194950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:11:42.534 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:11:42.528+0000 7fcc4a7fc700 1 --2- 172.21.15.191:0/1455878312 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fcc4c0fee60 0x7fcc4c194950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:11:42.534 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:11:42.528+0000 7fcc4a7fc700 1 -- 172.21.15.191:0/1455878312 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcc4c100290 msgr2=0x7fcc4c19bf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:11:42.534 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:11:42.528+0000 7fcc4a7fc700 1 --2- 172.21.15.191:0/1455878312 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcc4c100290 0x7fcc4c19bf10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:11:42.534 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:11:42.528+0000 7fcc4a7fc700 1 -- 172.21.15.191:0/1455878312 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc44009d00 con 0x7fcc4c0ff810 2023-07-14T19:11:42.534 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:11:42.528+0000 7fcc4affd700 1 --2- 172.21.15.191:0/1455878312 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fcc4c0fee60 0x7fcc4c194950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T19:11:42.534 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:11:42.528+0000 7fcc4b7fe700 1 --2- 172.21.15.191:0/1455878312 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcc4c100290 0x7fcc4c19bf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T19:11:42.534 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:11:42.528+0000 7fcc4a7fc700 1 --2- 172.21.15.191:0/1455878312 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fcc4c0ff810 0x7fcc4c194e90 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fcc4000c9a0 tx=0x7fcc4000cd60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:11:42.535 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:11:42.528+0000 7fcc50889700 1 -- 172.21.15.191:0/1455878312 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc40007a60 con 0x7fcc4c0ff810 2023-07-14T19:11:42.535 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:11:42.528+0000 7fcc5188b700 1 -- 172.21.15.191:0/1455878312 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcc4c19ca90 con 0x7fcc4c0ff810 2023-07-14T19:11:42.535 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:11:42.528+0000 7fcc50889700 1 -- 172.21.15.191:0/1455878312 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fcc40004cf0 con 0x7fcc4c0ff810 2023-07-14T19:11:42.535 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:11:42.528+0000 7fcc50889700 1 -- 172.21.15.191:0/1455878312 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc40005600 con 0x7fcc4c0ff810 2023-07-14T19:11:42.535 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:11:42.528+0000 7fcc5188b700 1 -- 172.21.15.191:0/1455878312 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcc4c19cf90 con 0x7fcc4c0ff810 2023-07-14T19:11:42.535 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:11:42.529+0000 7fcc5188b700 1 -- 172.21.15.191:0/1455878312 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcc4c10a450 con 0x7fcc4c0ff810 2023-07-14T19:11:42.536 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:11:42.530+0000 7fcc50889700 1 -- 172.21.15.191:0/1455878312 <== mon.1 v2:172.21.15.191:3300/0 4 ==== mgrmap(e 15) v1 ==== 135659+0+0 (secure 0 0 0) 0x7fcc40007bc0 con 0x7fcc4c0ff810 2023-07-14T19:11:42.536 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:11:42.530+0000 7fcc50889700 1 --2- 172.21.15.191:0/1455878312 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7fcc340a10c0 0x7fcc340a3570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:11:42.536 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:11:42.531+0000 7fcc50889700 1 -- 172.21.15.191:0/1455878312 <== mon.1 v2:172.21.15.191:3300/0 5 ==== osd_map(13..13 src has 1..13) v4 ==== 2309+0+0 (secure 0 0 0) 0x7fcc40013070 con 0x7fcc4c0ff810 2023-07-14T19:11:42.536 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:11:42.531+0000 7fcc4affd700 1 --2- 172.21.15.191:0/1455878312 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7fcc340a10c0 0x7fcc340a3570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:11:42.537 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:11:42.531+0000 7fcc4affd700 1 --2- 172.21.15.191:0/1455878312 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7fcc340a10c0 0x7fcc340a3570 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fcc3c005fd0 tx=0x7fcc3c005e30 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:11:42.537 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:11:42.535+0000 7fcc50889700 1 -- 172.21.15.191:0/1455878312 <== mon.1 v2:172.21.15.191:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fcc40095e50 con 0x7fcc4c0ff810 2023-07-14T19:11:42.729 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:11:42.727+0000 7fcc5188b700 1 -- 172.21.15.191:0/1455878312 --> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi191:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) v1 -- 0x7fcc4c060a80 con 0x7fcc340a10c0 2023-07-14T19:11:43.221 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:11:42 smithi040 ceph-mon[96518]: purged_snaps scrub starts 2023-07-14T19:11:43.222 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:11:42 smithi040 ceph-mon[96518]: purged_snaps scrub ok 2023-07-14T19:11:43.222 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:11:42 smithi040 ceph-mon[96518]: osd.1 [v2:172.21.15.130:6800/2512760702,v1:172.21.15.130:6801/2512760702] boot 2023-07-14T19:11:43.222 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:11:42 smithi040 ceph-mon[96518]: osdmap e13: 2 total, 2 up, 2 in 2023-07-14T19:11:43.222 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:11:42 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-07-14T19:11:43.223 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:11:42 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-14T19:11:43.223 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:11:42 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-14T19:11:43.223 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:11:42 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:11:43.223 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:11:42 smithi191 ceph-mon[98311]: purged_snaps scrub starts 2023-07-14T19:11:43.224 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:11:42 smithi191 ceph-mon[98311]: purged_snaps scrub ok 2023-07-14T19:11:43.224 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:11:42 smithi191 ceph-mon[98311]: osd.1 [v2:172.21.15.130:6800/2512760702,v1:172.21.15.130:6801/2512760702] boot 2023-07-14T19:11:43.224 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:11:42 smithi191 ceph-mon[98311]: osdmap e13: 2 total, 2 up, 2 in 2023-07-14T19:11:43.224 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:11:42 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-07-14T19:11:43.224 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:11:42 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-14T19:11:43.225 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:11:42 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-14T19:11:43.225 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:11:42 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:11:43.242 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:11:42 smithi130 ceph-mon[97832]: purged_snaps scrub starts 2023-07-14T19:11:43.243 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:11:42 smithi130 ceph-mon[97832]: purged_snaps scrub ok 2023-07-14T19:11:43.243 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:11:42 smithi130 ceph-mon[97832]: osd.1 [v2:172.21.15.130:6800/2512760702,v1:172.21.15.130:6801/2512760702] boot 2023-07-14T19:11:43.243 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:11:42 smithi130 ceph-mon[97832]: osdmap e13: 2 total, 2 up, 2 in 2023-07-14T19:11:43.243 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:11:42 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-07-14T19:11:43.243 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:11:42 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-14T19:11:43.243 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:11:42 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-14T19:11:43.244 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:11:42 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:11:43.859 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:11:43 smithi191 ceph-mon[98311]: from='client.24163 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi191:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T19:11:43.859 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:11:43 smithi191 ceph-mon[98311]: osdmap e14: 2 total, 2 up, 2 in 2023-07-14T19:11:43.859 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:11:43 smithi191 ceph-mon[98311]: pgmap v60: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-14T19:11:44.221 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:11:43 smithi040 ceph-mon[96518]: from='client.24163 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi191:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T19:11:44.222 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:11:43 smithi040 ceph-mon[96518]: osdmap e14: 2 total, 2 up, 2 in 2023-07-14T19:11:44.222 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:11:43 smithi040 ceph-mon[96518]: pgmap v60: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-14T19:11:44.243 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:11:43 smithi130 ceph-mon[97832]: from='client.24163 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi191:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T19:11:44.243 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:11:43 smithi130 ceph-mon[97832]: osdmap e14: 2 total, 2 up, 2 in 2023-07-14T19:11:44.243 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:11:43 smithi130 ceph-mon[97832]: pgmap v60: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-14T19:11:45.221 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:11:44 smithi040 ceph-mon[96518]: pgmap v61: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-14T19:11:45.222 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:11:44 smithi191 ceph-mon[98311]: pgmap v61: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-14T19:11:45.242 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:11:44 smithi130 ceph-mon[97832]: pgmap v61: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-14T19:11:46.221 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:11:45 smithi040 ceph-mon[96518]: from='client.? 172.21.15.191:0/3035027333' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5a157891-cb7a-4ea6-9239-f48a7c67c003"}]: dispatch 2023-07-14T19:11:46.222 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:11:45 smithi040 ceph-mon[96518]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5a157891-cb7a-4ea6-9239-f48a7c67c003"}]: dispatch 2023-07-14T19:11:46.222 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:11:45 smithi040 ceph-mon[96518]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "5a157891-cb7a-4ea6-9239-f48a7c67c003"}]': finished 2023-07-14T19:11:46.222 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:11:45 smithi040 ceph-mon[96518]: osdmap e15: 3 total, 2 up, 3 in 2023-07-14T19:11:46.222 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:11:45 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-07-14T19:11:46.222 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:11:45 smithi040 ceph-mon[96518]: from='client.? 172.21.15.191:0/2578858705' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-14T19:11:46.223 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:11:45 smithi191 ceph-mon[98311]: from='client.? 172.21.15.191:0/3035027333' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5a157891-cb7a-4ea6-9239-f48a7c67c003"}]: dispatch 2023-07-14T19:11:46.223 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:11:45 smithi191 ceph-mon[98311]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5a157891-cb7a-4ea6-9239-f48a7c67c003"}]: dispatch 2023-07-14T19:11:46.223 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:11:45 smithi191 ceph-mon[98311]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "5a157891-cb7a-4ea6-9239-f48a7c67c003"}]': finished 2023-07-14T19:11:46.223 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:11:45 smithi191 ceph-mon[98311]: osdmap e15: 3 total, 2 up, 3 in 2023-07-14T19:11:46.224 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:11:45 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-07-14T19:11:46.224 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:11:45 smithi191 ceph-mon[98311]: from='client.? 172.21.15.191:0/2578858705' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-14T19:11:46.242 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:11:45 smithi130 ceph-mon[97832]: from='client.? 172.21.15.191:0/3035027333' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5a157891-cb7a-4ea6-9239-f48a7c67c003"}]: dispatch 2023-07-14T19:11:46.243 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:11:45 smithi130 ceph-mon[97832]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5a157891-cb7a-4ea6-9239-f48a7c67c003"}]: dispatch 2023-07-14T19:11:46.243 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:11:45 smithi130 ceph-mon[97832]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "5a157891-cb7a-4ea6-9239-f48a7c67c003"}]': finished 2023-07-14T19:11:46.243 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:11:45 smithi130 ceph-mon[97832]: osdmap e15: 3 total, 2 up, 3 in 2023-07-14T19:11:46.243 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:11:45 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-07-14T19:11:46.243 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:11:45 smithi130 ceph-mon[97832]: from='client.? 172.21.15.191:0/2578858705' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-14T19:11:47.971 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:11:47 smithi040 ceph-mon[96518]: pgmap v63: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-14T19:11:47.972 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:11:47 smithi191 ceph-mon[98311]: pgmap v63: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-14T19:11:47.992 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:11:47 smithi130 ceph-mon[97832]: pgmap v63: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-14T19:11:48.973 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:11:48 smithi191 ceph-mon[98311]: pgmap v64: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-14T19:11:49.221 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:11:48 smithi040 ceph-mon[96518]: pgmap v64: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-14T19:11:49.243 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:11:48 smithi130 ceph-mon[97832]: pgmap v64: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-14T19:11:51.190 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:11:50 smithi191 ceph-mon[98311]: pgmap v65: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-14T19:11:51.221 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:11:50 smithi040 ceph-mon[96518]: pgmap v65: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-14T19:11:51.242 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:11:50 smithi130 ceph-mon[97832]: pgmap v65: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-14T19:11:52.716 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:11:52 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2023-07-14T19:11:52.716 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:11:52 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:11:52.716 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:11:52 smithi191 ceph-mon[98311]: Deploying daemon osd.2 on smithi191 2023-07-14T19:11:52.971 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:11:52 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2023-07-14T19:11:52.972 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:11:52 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:11:52.972 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:11:52 smithi040 ceph-mon[96518]: Deploying daemon osd.2 on smithi191 2023-07-14T19:11:52.993 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:11:52 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2023-07-14T19:11:52.993 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:11:52 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:11:52.993 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:11:52 smithi130 ceph-mon[97832]: Deploying daemon osd.2 on smithi191 2023-07-14T19:11:53.611 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:11:53 smithi191 ceph-mon[98311]: pgmap v66: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-14T19:11:53.971 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:11:53 smithi040 ceph-mon[96518]: pgmap v66: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-14T19:11:53.992 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:11:53 smithi130 ceph-mon[97832]: pgmap v66: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-14T19:11:55.221 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:11:54 smithi040 ceph-mon[96518]: pgmap v67: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-14T19:11:55.222 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:11:54 smithi191 ceph-mon[98311]: pgmap v67: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-14T19:11:55.242 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:11:54 smithi130 ceph-mon[97832]: pgmap v67: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-14T19:11:56.972 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:11:56 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:11:56.972 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:11:56 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:11:56.972 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:11:56 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T19:11:56.973 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:11:56 smithi191 ceph-mon[98311]: pgmap v68: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-14T19:11:57.221 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:11:56 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:11:57.222 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:11:56 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:11:57.222 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:11:56 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T19:11:57.222 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:11:56 smithi040 ceph-mon[96518]: pgmap v68: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-14T19:11:57.242 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:11:56 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:11:57.243 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:11:56 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:11:57.243 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:11:56 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T19:11:57.243 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:11:56 smithi130 ceph-mon[97832]: pgmap v68: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-14T19:11:58.592 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:11:58.589+0000 7fcc50889700 1 -- 172.21.15.191:0/1455878312 <== mgr.14152 v2:172.21.15.40:6800/1693027118 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7fcc4c060a80 con 0x7fcc340a10c0 2023-07-14T19:11:58.592 INFO:teuthology.orchestra.run.smithi191.stdout:Created osd(s) 2 on host 'smithi191' 2023-07-14T19:11:58.594 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:11:58.592+0000 7fcc5188b700 1 -- 172.21.15.191:0/1455878312 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7fcc340a10c0 msgr2=0x7fcc340a3570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:11:58.594 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:11:58.592+0000 7fcc5188b700 1 --2- 172.21.15.191:0/1455878312 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7fcc340a10c0 0x7fcc340a3570 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fcc3c005fd0 tx=0x7fcc3c005e30 comp rx=0 tx=0).stop 2023-07-14T19:11:58.595 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:11:58.592+0000 7fcc5188b700 1 -- 172.21.15.191:0/1455878312 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fcc4c0ff810 msgr2=0x7fcc4c194e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:11:58.595 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:11:58.592+0000 7fcc5188b700 1 --2- 172.21.15.191:0/1455878312 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fcc4c0ff810 0x7fcc4c194e90 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fcc4000c9a0 tx=0x7fcc4000cd60 comp rx=0 tx=0).stop 2023-07-14T19:11:58.595 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:11:58.592+0000 7fcc5188b700 1 -- 172.21.15.191:0/1455878312 shutdown_connections 2023-07-14T19:11:58.595 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:11:58.592+0000 7fcc5188b700 1 --2- 172.21.15.191:0/1455878312 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7fcc340a10c0 0x7fcc340a3570 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:11:58.595 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:11:58.592+0000 7fcc5188b700 1 --2- 172.21.15.191:0/1455878312 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcc4c100290 0x7fcc4c19bf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:11:58.595 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:11:58.592+0000 7fcc5188b700 1 --2- 172.21.15.191:0/1455878312 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fcc4c0fee60 0x7fcc4c194950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:11:58.595 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:11:58.592+0000 7fcc5188b700 1 --2- 172.21.15.191:0/1455878312 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fcc4c0ff810 0x7fcc4c194e90 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:11:58.596 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:11:58.592+0000 7fcc5188b700 1 -- 172.21.15.191:0/1455878312 >> 172.21.15.191:0/1455878312 conn(0x7fcc4c0f99c0 msgr2=0x7fcc4c0fbb00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:11:58.596 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:11:58.592+0000 7fcc5188b700 1 -- 172.21.15.191:0/1455878312 shutdown_connections 2023-07-14T19:11:58.596 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:11:58.592+0000 7fcc5188b700 1 -- 172.21.15.191:0/1455878312 wait complete. 2023-07-14T19:11:59.031 DEBUG:teuthology.orchestra.run.smithi191:osd.2> sudo journalctl -f -n 0 -u ceph-adc52f04-2279-11ee-9b33-001a4aab830c@osd.2.service 2023-07-14T19:11:59.033 INFO:tasks.cephadm:Deploying osd.3 on smithi040 with /dev/vg_nvme/lv_3... 2023-07-14T19:11:59.033 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid adc52f04-2279-11ee-9b33-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2023-07-14T19:11:59.066 INFO:journalctl@ceph.osd.2.smithi191.stdout:-- Logs begin at Fri 2023-07-14 18:50:54 UTC. -- 2023-07-14T19:11:59.471 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:11:59 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:11:59.472 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:11:59 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:11:59.472 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:11:59 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:11:59.472 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:11:59 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T19:11:59.472 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:11:59 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:11:59.472 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:11:59 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:11:59.472 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:11:59 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:11:59.472 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:11:59 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T19:11:59.473 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:11:59 smithi040 ceph-mon[96518]: pgmap v69: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-14T19:11:59.473 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:11:59 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:11:59.473 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:11:59 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:11:59.473 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:11:59 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:11:59.474 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:11:59 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T19:11:59.474 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:11:59 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:11:59.474 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:11:59 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:11:59.474 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:11:59 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:11:59.474 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:11:59 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T19:11:59.474 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:11:59 smithi191 ceph-mon[98311]: pgmap v69: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-14T19:11:59.492 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:11:59 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:11:59.493 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:11:59 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:11:59.493 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:11:59 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:11:59.493 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:11:59 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T19:11:59.493 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:11:59 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:11:59.493 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:11:59 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:11:59.493 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:11:59 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:11:59.493 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:11:59 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T19:11:59.494 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:11:59 smithi130 ceph-mon[97832]: pgmap v69: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-14T19:11:59.595 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/adc52f04-2279-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T19:12:00.201 INFO:journalctl@ceph.osd.2.smithi191.stdout:Jul 14 19:11:59 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-2[104126]: 2023-07-14T19:11:59.859+0000 7f58ee947640 -1 osd.2 0 log_to_monitors true 2023-07-14T19:12:00.722 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:12:00 smithi191 ceph-mon[98311]: from='osd.2 [v2:172.21.15.191:6800/589569135,v1:172.21.15.191:6801/589569135]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2023-07-14T19:12:00.723 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:12:00 smithi191 ceph-mon[98311]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2023-07-14T19:12:00.972 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:12:00 smithi040 ceph-mon[96518]: from='osd.2 [v2:172.21.15.191:6800/589569135,v1:172.21.15.191:6801/589569135]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2023-07-14T19:12:00.972 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:12:00 smithi040 ceph-mon[96518]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2023-07-14T19:12:00.992 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:12:00 smithi130 ceph-mon[97832]: from='osd.2 [v2:172.21.15.191:6800/589569135,v1:172.21.15.191:6801/589569135]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2023-07-14T19:12:00.993 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:12:00 smithi130 ceph-mon[97832]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2023-07-14T19:12:01.094 INFO:teuthology.orchestra.run.smithi040.stderr:--> Zapping: /dev/vg_nvme/lv_3 2023-07-14T19:12:01.094 INFO:teuthology.orchestra.run.smithi040.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2023-07-14T19:12:01.094 INFO:teuthology.orchestra.run.smithi040.stderr: stderr: 10+0 records in 2023-07-14T19:12:01.095 INFO:teuthology.orchestra.run.smithi040.stderr:10+0 records out 2023-07-14T19:12:01.095 INFO:teuthology.orchestra.run.smithi040.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0108126 s, 970 MB/s 2023-07-14T19:12:01.095 INFO:teuthology.orchestra.run.smithi040.stderr:--> Zapping successful for: 2023-07-14T19:12:01.642 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid adc52f04-2279-11ee-9b33-001a4aab830c -- ceph orch daemon add osd smithi040:vg_nvme/lv_3 2023-07-14T19:12:01.722 INFO:journalctl@ceph.osd.2.smithi191.stdout:Jul 14 19:12:01 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-2[104126]: 2023-07-14T19:12:01.583+0000 7f58e3fbf700 -1 osd.2 0 waiting for initial osdmap 2023-07-14T19:12:01.723 INFO:journalctl@ceph.osd.2.smithi191.stdout:Jul 14 19:12:01 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-2[104126]: 2023-07-14T19:12:01.587+0000 7f58dd5b0700 -1 osd.2 17 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-07-14T19:12:01.723 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:12:01 smithi191 ceph-mon[98311]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2023-07-14T19:12:01.723 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:12:01 smithi191 ceph-mon[98311]: osdmap e16: 3 total, 2 up, 3 in 2023-07-14T19:12:01.723 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:12:01 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-07-14T19:12:01.724 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:12:01 smithi191 ceph-mon[98311]: from='osd.2 [v2:172.21.15.191:6800/589569135,v1:172.21.15.191:6801/589569135]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi191", "root=default"]}]: dispatch 2023-07-14T19:12:01.724 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:12:01 smithi191 ceph-mon[98311]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi191", "root=default"]}]: dispatch 2023-07-14T19:12:01.724 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:12:01 smithi191 ceph-mon[98311]: pgmap v71: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-14T19:12:01.724 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:12:01 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:12:01.724 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:12:01 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:12:01.724 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:12:01 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-07-14T19:12:01.724 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:12:01 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:12:01.725 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:12:01 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:12:01.725 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:12:01 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T19:12:01.725 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:12:01 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:12:01.972 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:12:01 smithi040 ceph-mon[96518]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2023-07-14T19:12:01.972 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:12:01 smithi040 ceph-mon[96518]: osdmap e16: 3 total, 2 up, 3 in 2023-07-14T19:12:01.973 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:12:01 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-07-14T19:12:01.973 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:12:01 smithi040 ceph-mon[96518]: from='osd.2 [v2:172.21.15.191:6800/589569135,v1:172.21.15.191:6801/589569135]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi191", "root=default"]}]: dispatch 2023-07-14T19:12:01.973 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:12:01 smithi040 ceph-mon[96518]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi191", "root=default"]}]: dispatch 2023-07-14T19:12:01.974 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:12:01 smithi040 ceph-mon[96518]: pgmap v71: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-14T19:12:01.974 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:12:01 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:12:01.974 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:12:01 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:12:01.974 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:12:01 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-07-14T19:12:01.975 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:12:01 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:12:01.975 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:12:01 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:12:01.975 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:12:01 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T19:12:01.976 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:12:01 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:12:01.992 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:12:01 smithi130 ceph-mon[97832]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2023-07-14T19:12:01.993 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:12:01 smithi130 ceph-mon[97832]: osdmap e16: 3 total, 2 up, 3 in 2023-07-14T19:12:01.993 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:12:01 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-07-14T19:12:01.993 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:12:01 smithi130 ceph-mon[97832]: from='osd.2 [v2:172.21.15.191:6800/589569135,v1:172.21.15.191:6801/589569135]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi191", "root=default"]}]: dispatch 2023-07-14T19:12:01.993 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:12:01 smithi130 ceph-mon[97832]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi191", "root=default"]}]: dispatch 2023-07-14T19:12:01.994 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:12:01 smithi130 ceph-mon[97832]: pgmap v71: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-14T19:12:01.994 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:12:01 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:12:01.994 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:12:01 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:12:01.994 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:12:01 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-07-14T19:12:01.994 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:12:01 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:12:01.994 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:12:01 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:12:01.994 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:12:01 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T19:12:01.995 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:12:01 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:12:02.201 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/adc52f04-2279-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T19:12:02.722 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:12:02 smithi040 ceph-mon[96518]: Detected new or changed devices on smithi191 2023-07-14T19:12:02.722 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:12:02 smithi040 ceph-mon[96518]: Adjusting osd_memory_target on smithi191 to 17168M 2023-07-14T19:12:02.722 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:12:02 smithi040 ceph-mon[96518]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi191", "root=default"]}]': finished 2023-07-14T19:12:02.722 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:12:02 smithi040 ceph-mon[96518]: osdmap e17: 3 total, 2 up, 3 in 2023-07-14T19:12:02.723 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:12:02 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-07-14T19:12:02.723 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:12:02 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-07-14T19:12:02.972 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:12:02 smithi191 ceph-mon[98311]: Detected new or changed devices on smithi191 2023-07-14T19:12:02.973 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:12:02 smithi191 ceph-mon[98311]: Adjusting osd_memory_target on smithi191 to 17168M 2023-07-14T19:12:02.973 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:12:02 smithi191 ceph-mon[98311]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi191", "root=default"]}]': finished 2023-07-14T19:12:02.973 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:12:02 smithi191 ceph-mon[98311]: osdmap e17: 3 total, 2 up, 3 in 2023-07-14T19:12:02.973 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:12:02 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-07-14T19:12:02.973 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:12:02 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-07-14T19:12:02.992 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:12:02 smithi130 ceph-mon[97832]: Detected new or changed devices on smithi191 2023-07-14T19:12:02.993 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:12:02 smithi130 ceph-mon[97832]: Adjusting osd_memory_target on smithi191 to 17168M 2023-07-14T19:12:02.993 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:12:02 smithi130 ceph-mon[97832]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi191", "root=default"]}]': finished 2023-07-14T19:12:02.993 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:12:02 smithi130 ceph-mon[97832]: osdmap e17: 3 total, 2 up, 3 in 2023-07-14T19:12:02.993 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:12:02 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-07-14T19:12:02.993 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:12:02 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-07-14T19:12:03.554 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:12:03.552+0000 7faad3a86700 1 -- 172.21.15.40:0/2704865280 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7faacc0ffcf0 msgr2=0x7faacc100100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:12:03.555 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:12:03.552+0000 7faad3a86700 1 --2- 172.21.15.40:0/2704865280 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7faacc0ffcf0 0x7faacc100100 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7faac8009990 tx=0x7faac8009ca0 comp rx=0 tx=0).stop 2023-07-14T19:12:03.555 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:12:03.552+0000 7faad3a86700 1 -- 172.21.15.40:0/2704865280 shutdown_connections 2023-07-14T19:12:03.555 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:12:03.552+0000 7faad3a86700 1 --2- 172.21.15.40:0/2704865280 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7faacc101880 0x7faacc108140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:12:03.555 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:12:03.552+0000 7faad3a86700 1 --2- 172.21.15.40:0/2704865280 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7faacc0ffcf0 0x7faacc100100 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:12:03.555 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:12:03.552+0000 7faad3a86700 1 --2- 172.21.15.40:0/2704865280 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7faacc100ef0 0x7faacc101340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:12:03.555 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:12:03.552+0000 7faad3a86700 1 -- 172.21.15.40:0/2704865280 >> 172.21.15.40:0/2704865280 conn(0x7faacc0fb360 msgr2=0x7faacc0fd770 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:12:03.556 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:12:03.552+0000 7faad3a86700 1 -- 172.21.15.40:0/2704865280 shutdown_connections 2023-07-14T19:12:03.556 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:12:03.552+0000 7faad3a86700 1 -- 172.21.15.40:0/2704865280 wait complete. 2023-07-14T19:12:03.557 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:12:03.553+0000 7faad3a86700 1 Processor -- start 2023-07-14T19:12:03.557 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:12:03.553+0000 7faad3a86700 1 -- start start 2023-07-14T19:12:03.558 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:12:03.553+0000 7faad3a86700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7faacc0ffcf0 0x7faacc1937c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:12:03.558 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:12:03.553+0000 7faad3a86700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7faacc100ef0 0x7faacc193d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:12:03.558 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:12:03.553+0000 7faad3a86700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7faacc101880 0x7faacc19ad80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:12:03.558 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:12:03.553+0000 7faad3a86700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faacc19b350 con 0x7faacc100ef0 2023-07-14T19:12:03.558 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:12:03.553+0000 7faad3a86700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faacc19b490 con 0x7faacc101880 2023-07-14T19:12:03.558 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:12:03.553+0000 7faad3a86700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faacc19b620 con 0x7faacc0ffcf0 2023-07-14T19:12:03.558 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:12:03.553+0000 7faad1021700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7faacc100ef0 0x7faacc193d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:12:03.559 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:12:03.553+0000 7faad1021700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7faacc100ef0 0x7faacc193d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:60268/0 (socket says 172.21.15.40:60268) 2023-07-14T19:12:03.559 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:12:03.553+0000 7faad1021700 1 -- 172.21.15.40:0/1835941393 learned_addr learned my addr 172.21.15.40:0/1835941393 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-07-14T19:12:03.559 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:12:03.553+0000 7faad2023700 1 --2- 172.21.15.40:0/1835941393 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7faacc101880 0x7faacc19ad80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:12:03.559 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:12:03.553+0000 7faad1822700 1 --2- 172.21.15.40:0/1835941393 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7faacc0ffcf0 0x7faacc1937c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:12:03.559 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:12:03.553+0000 7faad1021700 1 -- 172.21.15.40:0/1835941393 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7faacc0ffcf0 msgr2=0x7faacc1937c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:12:03.559 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:12:03.553+0000 7faad1021700 1 --2- 172.21.15.40:0/1835941393 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7faacc0ffcf0 0x7faacc1937c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:12:03.559 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:12:03.553+0000 7faad1021700 1 -- 172.21.15.40:0/1835941393 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7faacc101880 msgr2=0x7faacc19ad80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:12:03.559 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:12:03.553+0000 7faad1021700 1 --2- 172.21.15.40:0/1835941393 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7faacc101880 0x7faacc19ad80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:12:03.560 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:12:03.553+0000 7faad1021700 1 -- 172.21.15.40:0/1835941393 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faac80095f0 con 0x7faacc100ef0 2023-07-14T19:12:03.560 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:12:03.553+0000 7faad2023700 1 --2- 172.21.15.40:0/1835941393 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7faacc101880 0x7faacc19ad80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T19:12:03.560 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:12:03.553+0000 7faad1822700 1 --2- 172.21.15.40:0/1835941393 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7faacc0ffcf0 0x7faacc1937c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T19:12:03.560 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:12:03.554+0000 7faad1021700 1 --2- 172.21.15.40:0/1835941393 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7faacc100ef0 0x7faacc193d00 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7faabc00d980 tx=0x7faabc00dd40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:12:03.560 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:12:03.554+0000 7faac2ffd700 1 -- 172.21.15.40:0/1835941393 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faabc009b40 con 0x7faacc100ef0 2023-07-14T19:12:03.560 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:12:03.554+0000 7faad3a86700 1 -- 172.21.15.40:0/1835941393 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faacc19b900 con 0x7faacc100ef0 2023-07-14T19:12:03.560 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:12:03.554+0000 7faac2ffd700 1 -- 172.21.15.40:0/1835941393 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7faabc00b6e0 con 0x7faacc100ef0 2023-07-14T19:12:03.561 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:12:03.554+0000 7faac2ffd700 1 -- 172.21.15.40:0/1835941393 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faabc00f620 con 0x7faacc100ef0 2023-07-14T19:12:03.561 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:12:03.554+0000 7faad3a86700 1 -- 172.21.15.40:0/1835941393 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faacc19be00 con 0x7faacc100ef0 2023-07-14T19:12:03.561 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:12:03.555+0000 7faad3a86700 1 -- 172.21.15.40:0/1835941393 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faacc1072f0 con 0x7faacc100ef0 2023-07-14T19:12:03.561 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:12:03.556+0000 7faac2ffd700 1 -- 172.21.15.40:0/1835941393 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 15) v1 ==== 135659+0+0 (secure 0 0 0) 0x7faabc010460 con 0x7faacc100ef0 2023-07-14T19:12:03.562 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:12:03.557+0000 7faac2ffd700 1 --2- 172.21.15.40:0/1835941393 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7faab80a54d0 0x7faab80a7980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:12:03.562 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:12:03.557+0000 7faad1822700 1 --2- 172.21.15.40:0/1835941393 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7faab80a54d0 0x7faab80a7980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:12:03.562 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:12:03.557+0000 7faac2ffd700 1 -- 172.21.15.40:0/1835941393 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 2851+0+0 (secure 0 0 0) 0x7faabc0cb1e0 con 0x7faacc100ef0 2023-07-14T19:12:03.562 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:12:03.557+0000 7faad1822700 1 --2- 172.21.15.40:0/1835941393 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7faab80a54d0 0x7faab80a7980 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7faac8006010 tx=0x7faac800b540 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:12:03.562 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:12:03.560+0000 7faac2ffd700 1 -- 172.21.15.40:0/1835941393 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7faabc095f60 con 0x7faacc100ef0 2023-07-14T19:12:03.721 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:12:03 smithi040 ceph-mon[96518]: purged_snaps scrub starts 2023-07-14T19:12:03.721 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:12:03 smithi040 ceph-mon[96518]: purged_snaps scrub ok 2023-07-14T19:12:03.722 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:12:03 smithi040 ceph-mon[96518]: osd.2 [v2:172.21.15.191:6800/589569135,v1:172.21.15.191:6801/589569135] boot 2023-07-14T19:12:03.722 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:12:03 smithi040 ceph-mon[96518]: osdmap e18: 3 total, 3 up, 3 in 2023-07-14T19:12:03.722 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:12:03 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-07-14T19:12:03.722 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:12:03 smithi040 ceph-mon[96518]: pgmap v74: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-14T19:12:03.722 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:12:03 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2023-07-14T19:12:03.774 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:12:03.772+0000 7faad3a86700 1 -- 172.21.15.40:0/1835941393 --> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi040:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) v1 -- 0x7faacc060a80 con 0x7faab80a54d0 2023-07-14T19:12:03.972 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:12:03 smithi191 ceph-mon[98311]: purged_snaps scrub starts 2023-07-14T19:12:03.972 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:12:03 smithi191 ceph-mon[98311]: purged_snaps scrub ok 2023-07-14T19:12:03.973 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:12:03 smithi191 ceph-mon[98311]: osd.2 [v2:172.21.15.191:6800/589569135,v1:172.21.15.191:6801/589569135] boot 2023-07-14T19:12:03.973 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:12:03 smithi191 ceph-mon[98311]: osdmap e18: 3 total, 3 up, 3 in 2023-07-14T19:12:03.973 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:12:03 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-07-14T19:12:03.973 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:12:03 smithi191 ceph-mon[98311]: pgmap v74: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-14T19:12:03.973 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:12:03 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2023-07-14T19:12:03.992 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:12:03 smithi130 ceph-mon[97832]: purged_snaps scrub starts 2023-07-14T19:12:03.993 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:12:03 smithi130 ceph-mon[97832]: purged_snaps scrub ok 2023-07-14T19:12:03.993 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:12:03 smithi130 ceph-mon[97832]: osd.2 [v2:172.21.15.191:6800/589569135,v1:172.21.15.191:6801/589569135] boot 2023-07-14T19:12:03.993 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:12:03 smithi130 ceph-mon[97832]: osdmap e18: 3 total, 3 up, 3 in 2023-07-14T19:12:03.993 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:12:03 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-07-14T19:12:03.993 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:12:03 smithi130 ceph-mon[97832]: pgmap v74: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-14T19:12:03.993 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:12:03 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2023-07-14T19:12:04.742 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:12:04 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2023-07-14T19:12:04.743 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:12:04 smithi130 ceph-mon[97832]: osdmap e19: 3 total, 3 up, 3 in 2023-07-14T19:12:04.743 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:12:04 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2023-07-14T19:12:04.743 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:12:04 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-14T19:12:04.743 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:12:04 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-14T19:12:04.743 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:12:04 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:12:04.896 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:12:04 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2023-07-14T19:12:04.896 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:12:04 smithi040 ceph-mon[96518]: osdmap e19: 3 total, 3 up, 3 in 2023-07-14T19:12:04.896 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:12:04 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2023-07-14T19:12:04.896 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:12:04 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-14T19:12:04.896 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:12:04 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-14T19:12:04.897 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:12:04 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:12:04.972 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:12:04 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2023-07-14T19:12:04.972 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:12:04 smithi191 ceph-mon[98311]: osdmap e19: 3 total, 3 up, 3 in 2023-07-14T19:12:04.972 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:12:04 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2023-07-14T19:12:04.973 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:12:04 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-14T19:12:04.973 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:12:04 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-14T19:12:04.973 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:12:04 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:12:05.422 INFO:journalctl@ceph.osd.0.smithi040.stdout:Jul 14 19:12:05 smithi040 sudo[112551]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2023-07-14T19:12:05.422 INFO:journalctl@ceph.osd.0.smithi040.stdout:Jul 14 19:12:05 smithi040 sudo[112551]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-07-14T19:12:05.422 INFO:journalctl@ceph.osd.0.smithi040.stdout:Jul 14 19:12:05 smithi040 sudo[112551]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-07-14T19:12:05.422 INFO:journalctl@ceph.osd.0.smithi040.stdout:Jul 14 19:12:05 smithi040 sudo[112551]: pam_unix(sudo:session): session closed for user root 2023-07-14T19:12:05.622 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:12:05.619+0000 7faac2ffd700 1 -- 172.21.15.40:0/1835941393 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7faabc096330 con 0x7faacc100ef0 2023-07-14T19:12:05.894 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:12:05 smithi130 ceph-mon[97832]: from='client.14280 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi040:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T19:12:05.894 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:12:05 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2023-07-14T19:12:05.895 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:12:05 smithi130 ceph-mon[97832]: osdmap e20: 3 total, 3 up, 3 in 2023-07-14T19:12:05.895 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:12:05 smithi130 ceph-mon[97832]: pgmap v77: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2023-07-14T19:12:05.971 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:12:05 smithi040 ceph-mon[96518]: from='client.14280 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi040:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T19:12:05.972 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:12:05 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2023-07-14T19:12:05.972 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:12:05 smithi040 ceph-mon[96518]: osdmap e20: 3 total, 3 up, 3 in 2023-07-14T19:12:05.972 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:12:05 smithi040 ceph-mon[96518]: pgmap v77: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2023-07-14T19:12:05.973 INFO:journalctl@ceph.osd.0.smithi040.stdout:Jul 14 19:12:05 smithi040 sudo[112601]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2023-07-14T19:12:05.973 INFO:journalctl@ceph.osd.0.smithi040.stdout:Jul 14 19:12:05 smithi040 sudo[112601]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-07-14T19:12:05.973 INFO:journalctl@ceph.osd.0.smithi040.stdout:Jul 14 19:12:05 smithi040 sudo[112601]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-07-14T19:12:05.973 INFO:journalctl@ceph.osd.0.smithi040.stdout:Jul 14 19:12:05 smithi040 sudo[112601]: pam_unix(sudo:session): session closed for user root 2023-07-14T19:12:05.973 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:12:05 smithi191 ceph-mon[98311]: from='client.14280 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi040:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T19:12:05.974 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:12:05 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2023-07-14T19:12:05.974 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:12:05 smithi191 ceph-mon[98311]: osdmap e20: 3 total, 3 up, 3 in 2023-07-14T19:12:05.974 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:12:05 smithi191 ceph-mon[98311]: pgmap v77: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2023-07-14T19:12:06.146 INFO:journalctl@ceph.osd.1.smithi130.stdout:Jul 14 19:12:05 smithi130 sudo[105094]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2023-07-14T19:12:06.147 INFO:journalctl@ceph.osd.1.smithi130.stdout:Jul 14 19:12:05 smithi130 sudo[105094]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-07-14T19:12:06.147 INFO:journalctl@ceph.osd.1.smithi130.stdout:Jul 14 19:12:05 smithi130 sudo[105094]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-07-14T19:12:06.147 INFO:journalctl@ceph.osd.1.smithi130.stdout:Jul 14 19:12:05 smithi130 sudo[105094]: pam_unix(sudo:session): session closed for user root 2023-07-14T19:12:06.492 INFO:journalctl@ceph.osd.1.smithi130.stdout:Jul 14 19:12:06 smithi130 sudo[105097]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2023-07-14T19:12:06.493 INFO:journalctl@ceph.osd.1.smithi130.stdout:Jul 14 19:12:06 smithi130 sudo[105097]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-07-14T19:12:06.493 INFO:journalctl@ceph.osd.1.smithi130.stdout:Jul 14 19:12:06 smithi130 sudo[105097]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-07-14T19:12:06.493 INFO:journalctl@ceph.osd.1.smithi130.stdout:Jul 14 19:12:06 smithi130 sudo[105097]: pam_unix(sudo:session): session closed for user root 2023-07-14T19:12:06.669 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:12:06 smithi191 ceph-mon[98311]: osdmap e21: 3 total, 3 up, 3 in 2023-07-14T19:12:06.669 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:12:06 smithi191 ceph-mon[98311]: mgrmap e16: x(active, since 2m), standbys: z, y 2023-07-14T19:12:06.670 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:12:06 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:12:06.670 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:12:06 smithi191 ceph-mon[98311]: from='client.? 172.21.15.40:0/2492748366' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8020e7dc-ac40-487f-857a-8da40bccb459"}]: dispatch 2023-07-14T19:12:06.670 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:12:06 smithi191 ceph-mon[98311]: from='client.? 172.21.15.40:0/2492748366' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "8020e7dc-ac40-487f-857a-8da40bccb459"}]': finished 2023-07-14T19:12:06.670 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:12:06 smithi191 ceph-mon[98311]: osdmap e22: 4 total, 3 up, 4 in 2023-07-14T19:12:06.670 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:12:06 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-07-14T19:12:06.670 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:12:06 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:12:06.671 INFO:journalctl@ceph.osd.2.smithi191.stdout:Jul 14 19:12:06 smithi191 sudo[106383]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2023-07-14T19:12:06.671 INFO:journalctl@ceph.osd.2.smithi191.stdout:Jul 14 19:12:06 smithi191 sudo[106383]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-07-14T19:12:06.671 INFO:journalctl@ceph.osd.2.smithi191.stdout:Jul 14 19:12:06 smithi191 sudo[106383]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-07-14T19:12:06.671 INFO:journalctl@ceph.osd.2.smithi191.stdout:Jul 14 19:12:06 smithi191 sudo[106383]: pam_unix(sudo:session): session closed for user root 2023-07-14T19:12:06.971 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:12:06 smithi040 ceph-mon[96518]: osdmap e21: 3 total, 3 up, 3 in 2023-07-14T19:12:06.972 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:12:06 smithi040 ceph-mon[96518]: mgrmap e16: x(active, since 2m), standbys: z, y 2023-07-14T19:12:06.972 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:12:06 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:12:06.972 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:12:06 smithi040 ceph-mon[96518]: from='client.? 172.21.15.40:0/2492748366' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8020e7dc-ac40-487f-857a-8da40bccb459"}]: dispatch 2023-07-14T19:12:06.972 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:12:06 smithi040 ceph-mon[96518]: from='client.? 172.21.15.40:0/2492748366' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "8020e7dc-ac40-487f-857a-8da40bccb459"}]': finished 2023-07-14T19:12:06.972 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:12:06 smithi040 ceph-mon[96518]: osdmap e22: 4 total, 3 up, 4 in 2023-07-14T19:12:06.973 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:12:06 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-07-14T19:12:06.973 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:12:06 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:12:06.973 INFO:journalctl@ceph.osd.2.smithi191.stdout:Jul 14 19:12:06 smithi191 sudo[106386]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2023-07-14T19:12:06.973 INFO:journalctl@ceph.osd.2.smithi191.stdout:Jul 14 19:12:06 smithi191 sudo[106386]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-07-14T19:12:06.974 INFO:journalctl@ceph.osd.2.smithi191.stdout:Jul 14 19:12:06 smithi191 sudo[106386]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-07-14T19:12:06.974 INFO:journalctl@ceph.osd.2.smithi191.stdout:Jul 14 19:12:06 smithi191 sudo[106386]: pam_unix(sudo:session): session closed for user root 2023-07-14T19:12:06.992 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:12:06 smithi130 ceph-mon[97832]: osdmap e21: 3 total, 3 up, 3 in 2023-07-14T19:12:06.993 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:12:06 smithi130 ceph-mon[97832]: mgrmap e16: x(active, since 2m), standbys: z, y 2023-07-14T19:12:06.993 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:12:06 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:12:06.993 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:12:06 smithi130 ceph-mon[97832]: from='client.? 172.21.15.40:0/2492748366' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8020e7dc-ac40-487f-857a-8da40bccb459"}]: dispatch 2023-07-14T19:12:06.993 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:12:06 smithi130 ceph-mon[97832]: from='client.? 172.21.15.40:0/2492748366' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "8020e7dc-ac40-487f-857a-8da40bccb459"}]': finished 2023-07-14T19:12:06.993 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:12:06 smithi130 ceph-mon[97832]: osdmap e22: 4 total, 3 up, 4 in 2023-07-14T19:12:06.993 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:12:06 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-07-14T19:12:06.993 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:12:06 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:12:07.471 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:12:07 smithi040 sudo[112674]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2023-07-14T19:12:07.472 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:12:07 smithi040 sudo[112674]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-07-14T19:12:07.472 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:12:07 smithi040 sudo[112674]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-07-14T19:12:07.472 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:12:07 smithi040 sudo[112674]: pam_unix(sudo:session): session closed for user root 2023-07-14T19:12:07.971 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:12:07 smithi040 sudo[112677]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2023-07-14T19:12:07.972 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:12:07 smithi040 sudo[112677]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-07-14T19:12:07.972 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:12:07 smithi040 sudo[112677]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-07-14T19:12:07.972 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:12:07 smithi040 sudo[112677]: pam_unix(sudo:session): session closed for user root 2023-07-14T19:12:08.077 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:12:07 smithi191 sudo[106389]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2023-07-14T19:12:08.078 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:12:07 smithi191 sudo[106389]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-07-14T19:12:08.078 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:12:07 smithi191 sudo[106389]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-07-14T19:12:08.078 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:12:07 smithi191 sudo[106389]: pam_unix(sudo:session): session closed for user root 2023-07-14T19:12:08.472 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:12:08 smithi191 sudo[106392]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2023-07-14T19:12:08.473 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:12:08 smithi191 sudo[106392]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-07-14T19:12:08.473 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:12:08 smithi191 sudo[106392]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-07-14T19:12:08.473 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:12:08 smithi191 sudo[106392]: pam_unix(sudo:session): session closed for user root 2023-07-14T19:12:08.572 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:12:08 smithi130 sudo[105100]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2023-07-14T19:12:08.573 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:12:08 smithi130 sudo[105100]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-07-14T19:12:08.573 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:12:08 smithi130 sudo[105100]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-07-14T19:12:08.573 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:12:08 smithi130 sudo[105100]: pam_unix(sudo:session): session closed for user root 2023-07-14T19:12:08.971 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:12:08 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:12:08.972 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:12:08 smithi040 ceph-mon[96518]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2023-07-14T19:12:08.972 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:12:08 smithi040 ceph-mon[96518]: pgmap v80: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail 2023-07-14T19:12:08.972 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:12:08 smithi040 ceph-mon[96518]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2023-07-14T19:12:08.973 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:12:08 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-07-14T19:12:08.973 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:12:08 smithi040 ceph-mon[96518]: from='client.? 172.21.15.40:0/3706606007' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-14T19:12:08.973 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:12:08 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-14T19:12:08.973 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:12:08 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-14T19:12:08.973 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:12:08 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-07-14T19:12:08.973 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:12:08 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-14T19:12:08.973 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:12:08 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-14T19:12:08.974 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:12:08 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:12:08.974 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:12:08 smithi191 ceph-mon[98311]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2023-07-14T19:12:08.974 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:12:08 smithi191 ceph-mon[98311]: pgmap v80: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail 2023-07-14T19:12:08.974 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:12:08 smithi191 ceph-mon[98311]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2023-07-14T19:12:08.975 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:12:08 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-07-14T19:12:08.975 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:12:08 smithi191 ceph-mon[98311]: from='client.? 172.21.15.40:0/3706606007' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-14T19:12:08.975 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:12:08 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-14T19:12:08.975 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:12:08 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-14T19:12:08.975 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:12:08 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-07-14T19:12:08.975 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:12:08 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-14T19:12:08.975 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:12:08 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-14T19:12:08.992 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:12:08 smithi130 sudo[105103]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2023-07-14T19:12:08.992 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:12:08 smithi130 sudo[105103]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-07-14T19:12:08.993 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:12:08 smithi130 sudo[105103]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-07-14T19:12:08.993 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:12:08 smithi130 sudo[105103]: pam_unix(sudo:session): session closed for user root 2023-07-14T19:12:08.993 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:12:08 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:12:08.993 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:12:08 smithi130 ceph-mon[97832]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2023-07-14T19:12:08.993 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:12:08 smithi130 ceph-mon[97832]: pgmap v80: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail 2023-07-14T19:12:08.993 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:12:08 smithi130 ceph-mon[97832]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2023-07-14T19:12:08.993 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:12:08 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-07-14T19:12:08.994 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:12:08 smithi130 ceph-mon[97832]: from='client.? 172.21.15.40:0/3706606007' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-14T19:12:08.994 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:12:08 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-14T19:12:08.994 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:12:08 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-14T19:12:08.994 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:12:08 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-07-14T19:12:08.994 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:12:08 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-14T19:12:08.994 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:12:08 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-14T19:12:09.971 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:12:09 smithi040 ceph-mon[96518]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2023-07-14T19:12:09.971 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:12:09 smithi040 ceph-mon[96518]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2023-07-14T19:12:09.972 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:12:09 smithi040 ceph-mon[96518]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2023-07-14T19:12:09.972 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:12:09 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-07-14T19:12:09.972 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:12:09 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-14T19:12:09.972 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:12:09 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-14T19:12:09.972 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:12:09 smithi040 ceph-mon[96518]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2023-07-14T19:12:09.973 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:12:09 smithi191 ceph-mon[98311]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2023-07-14T19:12:09.973 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:12:09 smithi191 ceph-mon[98311]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2023-07-14T19:12:09.973 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:12:09 smithi191 ceph-mon[98311]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2023-07-14T19:12:09.973 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:12:09 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-07-14T19:12:09.973 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:12:09 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-14T19:12:09.973 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:12:09 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-14T19:12:09.974 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:12:09 smithi191 ceph-mon[98311]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2023-07-14T19:12:09.992 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:12:09 smithi130 ceph-mon[97832]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2023-07-14T19:12:09.992 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:12:09 smithi130 ceph-mon[97832]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2023-07-14T19:12:09.993 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:12:09 smithi130 ceph-mon[97832]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2023-07-14T19:12:09.993 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:12:09 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-07-14T19:12:09.993 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:12:09 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-14T19:12:09.993 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:12:09 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-14T19:12:09.993 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:12:09 smithi130 ceph-mon[97832]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2023-07-14T19:12:10.971 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:12:10 smithi040 ceph-mon[96518]: pgmap v81: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail 2023-07-14T19:12:10.972 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:12:10 smithi191 ceph-mon[98311]: pgmap v81: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail 2023-07-14T19:12:10.992 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:12:10 smithi130 ceph-mon[97832]: pgmap v81: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail 2023-07-14T19:12:12.851 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:12:12 smithi040 ceph-mon[96518]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail 2023-07-14T19:12:12.972 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:12:12 smithi191 ceph-mon[98311]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail 2023-07-14T19:12:12.992 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:12:12 smithi130 ceph-mon[97832]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail 2023-07-14T19:12:14.721 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:12:14 smithi040 ceph-mon[96518]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail; 1.7 KiB/s rd, 44 KiB/s wr, 4 op/s 2023-07-14T19:12:14.722 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:12:14 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2023-07-14T19:12:14.722 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:12:14 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:12:14.972 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:12:14 smithi191 ceph-mon[98311]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail; 1.7 KiB/s rd, 44 KiB/s wr, 4 op/s 2023-07-14T19:12:14.972 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:12:14 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2023-07-14T19:12:14.973 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:12:14 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:12:14.992 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:12:14 smithi130 ceph-mon[97832]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail; 1.7 KiB/s rd, 44 KiB/s wr, 4 op/s 2023-07-14T19:12:14.993 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:12:14 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2023-07-14T19:12:14.993 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:12:14 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:12:15.714 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:12:15 smithi040 ceph-mon[96518]: Deploying daemon osd.3 on smithi040 2023-07-14T19:12:15.972 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:12:15 smithi191 ceph-mon[98311]: Deploying daemon osd.3 on smithi040 2023-07-14T19:12:15.992 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:12:15 smithi130 ceph-mon[97832]: Deploying daemon osd.3 on smithi040 2023-07-14T19:12:16.853 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:12:16 smithi040 ceph-mon[96518]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail; 1.2 KiB/s rd, 33 KiB/s wr, 3 op/s 2023-07-14T19:12:16.972 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:12:16 smithi191 ceph-mon[98311]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail; 1.2 KiB/s rd, 33 KiB/s wr, 3 op/s 2023-07-14T19:12:16.992 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:12:16 smithi130 ceph-mon[97832]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail; 1.2 KiB/s rd, 33 KiB/s wr, 3 op/s 2023-07-14T19:12:18.971 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:12:18 smithi040 ceph-mon[96518]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail; 1023 B/s rd, 26 KiB/s wr, 2 op/s 2023-07-14T19:12:18.971 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:12:18 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:12:18.972 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:12:18 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:12:18.972 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:12:18 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T19:12:18.972 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:12:18 smithi191 ceph-mon[98311]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail; 1023 B/s rd, 26 KiB/s wr, 2 op/s 2023-07-14T19:12:18.972 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:12:18 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:12:18.972 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:12:18 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:12:18.973 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:12:18 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T19:12:18.992 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:12:18 smithi130 ceph-mon[97832]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail; 1023 B/s rd, 26 KiB/s wr, 2 op/s 2023-07-14T19:12:18.993 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:12:18 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:12:18.993 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:12:18 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:12:18.993 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:12:18 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T19:12:20.860 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:12:20 smithi040 ceph-mon[96518]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail; 853 B/s rd, 22 KiB/s wr, 2 op/s 2023-07-14T19:12:20.972 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:12:20 smithi191 ceph-mon[98311]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail; 853 B/s rd, 22 KiB/s wr, 2 op/s 2023-07-14T19:12:20.992 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:12:20 smithi130 ceph-mon[97832]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail; 853 B/s rd, 22 KiB/s wr, 2 op/s 2023-07-14T19:12:21.179 INFO:teuthology.orchestra.run.smithi040.stdout:Created osd(s) 3 on host 'smithi040' 2023-07-14T19:12:21.179 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:12:21.176+0000 7faac2ffd700 1 -- 172.21.15.40:0/1835941393 <== mgr.14152 v2:172.21.15.40:6800/1693027118 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7faacc060a80 con 0x7faab80a54d0 2023-07-14T19:12:21.181 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:12:21.179+0000 7faad3a86700 1 -- 172.21.15.40:0/1835941393 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7faab80a54d0 msgr2=0x7faab80a7980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:12:21.181 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:12:21.179+0000 7faad3a86700 1 --2- 172.21.15.40:0/1835941393 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7faab80a54d0 0x7faab80a7980 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7faac8006010 tx=0x7faac800b540 comp rx=0 tx=0).stop 2023-07-14T19:12:21.182 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:12:21.179+0000 7faad3a86700 1 -- 172.21.15.40:0/1835941393 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7faacc100ef0 msgr2=0x7faacc193d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:12:21.182 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:12:21.179+0000 7faad3a86700 1 --2- 172.21.15.40:0/1835941393 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7faacc100ef0 0x7faacc193d00 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7faabc00d980 tx=0x7faabc00dd40 comp rx=0 tx=0).stop 2023-07-14T19:12:21.182 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:12:21.179+0000 7faad3a86700 1 -- 172.21.15.40:0/1835941393 shutdown_connections 2023-07-14T19:12:21.182 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:12:21.179+0000 7faad3a86700 1 --2- 172.21.15.40:0/1835941393 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7faab80a54d0 0x7faab80a7980 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:12:21.183 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:12:21.179+0000 7faad3a86700 1 --2- 172.21.15.40:0/1835941393 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7faacc0ffcf0 0x7faacc1937c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:12:21.183 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:12:21.179+0000 7faad3a86700 1 --2- 172.21.15.40:0/1835941393 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7faacc101880 0x7faacc19ad80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:12:21.183 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:12:21.179+0000 7faad3a86700 1 --2- 172.21.15.40:0/1835941393 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7faacc100ef0 0x7faacc193d00 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:12:21.183 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:12:21.179+0000 7faad3a86700 1 -- 172.21.15.40:0/1835941393 >> 172.21.15.40:0/1835941393 conn(0x7faacc0fb360 msgr2=0x7faacc104430 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:12:21.183 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:12:21.180+0000 7faad3a86700 1 -- 172.21.15.40:0/1835941393 shutdown_connections 2023-07-14T19:12:21.183 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:12:21.180+0000 7faad3a86700 1 -- 172.21.15.40:0/1835941393 wait complete. 2023-07-14T19:12:21.634 DEBUG:teuthology.orchestra.run.smithi040:osd.3> sudo journalctl -f -n 0 -u ceph-adc52f04-2279-11ee-9b33-001a4aab830c@osd.3.service 2023-07-14T19:12:21.636 INFO:tasks.cephadm:Deploying osd.4 on smithi130 with /dev/vg_nvme/lv_3... 2023-07-14T19:12:21.637 DEBUG:teuthology.orchestra.run.smithi130:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid adc52f04-2279-11ee-9b33-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2023-07-14T19:12:21.673 INFO:journalctl@ceph.osd.3.smithi040.stdout:-- Logs begin at Fri 2023-07-14 18:50:26 UTC. -- 2023-07-14T19:12:21.971 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:12:21 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:12:21.972 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:12:21 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:12:21.972 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:12:21 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:12:21.972 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:12:21 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T19:12:21.972 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:12:21 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:12:21.972 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:12:21 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T19:12:21.972 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:12:21 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:12:21.973 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:12:21 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:12:21.973 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:12:21 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:12:21.973 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:12:21 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:12:21.973 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:12:21 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:12:21.973 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:12:21 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T19:12:21.974 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:12:21 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:12:21.974 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:12:21 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T19:12:21.974 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:12:21 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:12:21.974 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:12:21 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:12:21.992 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:12:21 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:12:21.993 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:12:21 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:12:21.993 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:12:21 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:12:21.993 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:12:21 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T19:12:21.993 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:12:21 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:12:21.993 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:12:21 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T19:12:21.993 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:12:21 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:12:21.994 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:12:21 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:12:22.200 INFO:teuthology.orchestra.run.smithi130.stderr:Inferring config /var/lib/ceph/adc52f04-2279-11ee-9b33-001a4aab830c/mon.b/config 2023-07-14T19:12:22.436 INFO:journalctl@ceph.osd.3.smithi040.stdout:Jul 14 19:12:22 smithi040 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-3[114874]: 2023-07-14T19:12:22.335+0000 7fd248c05640 -1 osd.3 0 log_to_monitors true 2023-07-14T19:12:22.701 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:12:22 smithi040 ceph-mon[96518]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail; 853 B/s rd, 22 KiB/s wr, 2 op/s 2023-07-14T19:12:22.701 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:12:22 smithi040 ceph-mon[96518]: from='osd.3 [v2:172.21.15.40:6810/996287860,v1:172.21.15.40:6811/996287860]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2023-07-14T19:12:22.972 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:12:22 smithi191 ceph-mon[98311]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail; 853 B/s rd, 22 KiB/s wr, 2 op/s 2023-07-14T19:12:22.972 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:12:22 smithi191 ceph-mon[98311]: from='osd.3 [v2:172.21.15.40:6810/996287860,v1:172.21.15.40:6811/996287860]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2023-07-14T19:12:22.992 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:12:22 smithi130 ceph-mon[97832]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail; 853 B/s rd, 22 KiB/s wr, 2 op/s 2023-07-14T19:12:22.993 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:12:22 smithi130 ceph-mon[97832]: from='osd.3 [v2:172.21.15.40:6810/996287860,v1:172.21.15.40:6811/996287860]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2023-07-14T19:12:23.718 INFO:teuthology.orchestra.run.smithi130.stderr:--> Zapping: /dev/vg_nvme/lv_3 2023-07-14T19:12:23.718 INFO:teuthology.orchestra.run.smithi130.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2023-07-14T19:12:23.718 INFO:teuthology.orchestra.run.smithi130.stderr: stderr: 10+0 records in 2023-07-14T19:12:23.718 INFO:teuthology.orchestra.run.smithi130.stderr:10+0 records out 2023-07-14T19:12:23.718 INFO:teuthology.orchestra.run.smithi130.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.01257 s, 834 MB/s 2023-07-14T19:12:23.719 INFO:teuthology.orchestra.run.smithi130.stderr:--> Zapping successful for: 2023-07-14T19:12:23.942 INFO:journalctl@ceph.osd.3.smithi040.stdout:Jul 14 19:12:23 smithi040 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-3[114874]: 2023-07-14T19:12:23.685+0000 7fd23e27d700 -1 osd.3 0 waiting for initial osdmap 2023-07-14T19:12:23.942 INFO:journalctl@ceph.osd.3.smithi040.stdout:Jul 14 19:12:23 smithi040 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-3[114874]: 2023-07-14T19:12:23.691+0000 7fd23806f700 -1 osd.3 24 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-07-14T19:12:23.942 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:12:23 smithi040 ceph-mon[96518]: from='osd.3 [v2:172.21.15.40:6810/996287860,v1:172.21.15.40:6811/996287860]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2023-07-14T19:12:23.942 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:12:23 smithi040 ceph-mon[96518]: osdmap e23: 4 total, 3 up, 4 in 2023-07-14T19:12:23.943 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:12:23 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-07-14T19:12:23.943 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:12:23 smithi040 ceph-mon[96518]: from='osd.3 [v2:172.21.15.40:6810/996287860,v1:172.21.15.40:6811/996287860]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi040", "root=default"]}]: dispatch 2023-07-14T19:12:23.943 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:12:23 smithi040 ceph-mon[96518]: from='osd.3 [v2:172.21.15.40:6810/996287860,v1:172.21.15.40:6811/996287860]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi040", "root=default"]}]': finished 2023-07-14T19:12:23.943 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:12:23 smithi040 ceph-mon[96518]: osdmap e24: 4 total, 3 up, 4 in 2023-07-14T19:12:23.943 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:12:23 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-07-14T19:12:23.972 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:12:23 smithi191 ceph-mon[98311]: from='osd.3 [v2:172.21.15.40:6810/996287860,v1:172.21.15.40:6811/996287860]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2023-07-14T19:12:23.972 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:12:23 smithi191 ceph-mon[98311]: osdmap e23: 4 total, 3 up, 4 in 2023-07-14T19:12:23.972 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:12:23 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-07-14T19:12:23.973 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:12:23 smithi191 ceph-mon[98311]: from='osd.3 [v2:172.21.15.40:6810/996287860,v1:172.21.15.40:6811/996287860]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi040", "root=default"]}]: dispatch 2023-07-14T19:12:23.973 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:12:23 smithi191 ceph-mon[98311]: from='osd.3 [v2:172.21.15.40:6810/996287860,v1:172.21.15.40:6811/996287860]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi040", "root=default"]}]': finished 2023-07-14T19:12:23.973 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:12:23 smithi191 ceph-mon[98311]: osdmap e24: 4 total, 3 up, 4 in 2023-07-14T19:12:23.973 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:12:23 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-07-14T19:12:23.992 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:12:23 smithi130 ceph-mon[97832]: from='osd.3 [v2:172.21.15.40:6810/996287860,v1:172.21.15.40:6811/996287860]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2023-07-14T19:12:23.993 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:12:23 smithi130 ceph-mon[97832]: osdmap e23: 4 total, 3 up, 4 in 2023-07-14T19:12:23.993 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:12:23 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-07-14T19:12:23.993 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:12:23 smithi130 ceph-mon[97832]: from='osd.3 [v2:172.21.15.40:6810/996287860,v1:172.21.15.40:6811/996287860]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi040", "root=default"]}]: dispatch 2023-07-14T19:12:23.993 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:12:23 smithi130 ceph-mon[97832]: from='osd.3 [v2:172.21.15.40:6810/996287860,v1:172.21.15.40:6811/996287860]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi040", "root=default"]}]': finished 2023-07-14T19:12:23.993 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:12:23 smithi130 ceph-mon[97832]: osdmap e24: 4 total, 3 up, 4 in 2023-07-14T19:12:23.993 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:12:23 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-07-14T19:12:24.152 DEBUG:teuthology.orchestra.run.smithi130:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid adc52f04-2279-11ee-9b33-001a4aab830c -- ceph orch daemon add osd smithi130:vg_nvme/lv_3 2023-07-14T19:12:24.704 INFO:teuthology.orchestra.run.smithi130.stderr:Inferring config /var/lib/ceph/adc52f04-2279-11ee-9b33-001a4aab830c/mon.b/config 2023-07-14T19:12:24.950 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:12:24 smithi130 ceph-mon[97832]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail 2023-07-14T19:12:24.950 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:12:24 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-07-14T19:12:24.950 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:12:24 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:12:24.950 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:12:24 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:12:24.950 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:12:24 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-07-14T19:12:24.951 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:12:24 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-07-14T19:12:24.951 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:12:24 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:12:24.951 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:12:24 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:12:24.951 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:12:24 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T19:12:24.951 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:12:24 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:12:24.971 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:12:24 smithi040 ceph-mon[96518]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail 2023-07-14T19:12:24.971 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:12:24 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-07-14T19:12:24.972 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:12:24 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:12:24.972 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:12:24 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:12:24.972 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:12:24 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-07-14T19:12:24.972 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:12:24 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-07-14T19:12:24.972 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:12:24 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:12:24.972 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:12:24 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:12:24.973 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:12:24 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T19:12:24.973 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:12:24 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:12:24.973 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:12:24 smithi191 ceph-mon[98311]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail 2023-07-14T19:12:24.973 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:12:24 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-07-14T19:12:24.973 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:12:24 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:12:24.974 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:12:24 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:12:24.974 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:12:24 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-07-14T19:12:24.974 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:12:24 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-07-14T19:12:24.974 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:12:24 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:12:24.974 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:12:24 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:12:24.974 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:12:24 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T19:12:24.974 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:12:24 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:12:25.926 INFO:teuthology.orchestra.run.smithi130.stderr:2023-07-14T19:12:25.922+0000 7f737be06700 1 -- 172.21.15.130:0/3662683925 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f73741001a0 msgr2=0x7f73741005b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:12:25.926 INFO:teuthology.orchestra.run.smithi130.stderr:2023-07-14T19:12:25.922+0000 7f737be06700 1 --2- 172.21.15.130:0/3662683925 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f73741001a0 0x7f73741005b0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f7370009990 tx=0x7f7370009ca0 comp rx=0 tx=0).stop 2023-07-14T19:12:25.926 INFO:teuthology.orchestra.run.smithi130.stderr:2023-07-14T19:12:25.922+0000 7f737be06700 1 -- 172.21.15.130:0/3662683925 shutdown_connections 2023-07-14T19:12:25.926 INFO:teuthology.orchestra.run.smithi130.stderr:2023-07-14T19:12:25.922+0000 7f737be06700 1 --2- 172.21.15.130:0/3662683925 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f73740ff7e0 0x7f73741070b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:12:25.926 INFO:teuthology.orchestra.run.smithi130.stderr:2023-07-14T19:12:25.922+0000 7f737be06700 1 --2- 172.21.15.130:0/3662683925 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f73741001a0 0x7f73741005b0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:12:25.927 INFO:teuthology.orchestra.run.smithi130.stderr:2023-07-14T19:12:25.922+0000 7f737be06700 1 --2- 172.21.15.130:0/3662683925 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f73740fee50 0x7f73740ff2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:12:25.927 INFO:teuthology.orchestra.run.smithi130.stderr:2023-07-14T19:12:25.922+0000 7f737be06700 1 -- 172.21.15.130:0/3662683925 >> 172.21.15.130:0/3662683925 conn(0x7f73740f9990 msgr2=0x7f73740fbdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:12:25.927 INFO:teuthology.orchestra.run.smithi130.stderr:2023-07-14T19:12:25.922+0000 7f737be06700 1 -- 172.21.15.130:0/3662683925 shutdown_connections 2023-07-14T19:12:25.927 INFO:teuthology.orchestra.run.smithi130.stderr:2023-07-14T19:12:25.922+0000 7f737be06700 1 -- 172.21.15.130:0/3662683925 wait complete. 2023-07-14T19:12:25.927 INFO:teuthology.orchestra.run.smithi130.stderr:2023-07-14T19:12:25.922+0000 7f737be06700 1 Processor -- start 2023-07-14T19:12:25.927 INFO:teuthology.orchestra.run.smithi130.stderr:2023-07-14T19:12:25.922+0000 7f737be06700 1 -- start start 2023-07-14T19:12:25.928 INFO:teuthology.orchestra.run.smithi130.stderr:2023-07-14T19:12:25.923+0000 7f737be06700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f73740fee50 0x7f73741959c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:12:25.928 INFO:teuthology.orchestra.run.smithi130.stderr:2023-07-14T19:12:25.923+0000 7f737be06700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f73740ff7e0 0x7f7374195f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:12:25.928 INFO:teuthology.orchestra.run.smithi130.stderr:2023-07-14T19:12:25.923+0000 7f737be06700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f73741001a0 0x7f737419cf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:12:25.928 INFO:teuthology.orchestra.run.smithi130.stderr:2023-07-14T19:12:25.923+0000 7f737be06700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f737419d550 con 0x7f73741001a0 2023-07-14T19:12:25.928 INFO:teuthology.orchestra.run.smithi130.stderr:2023-07-14T19:12:25.923+0000 7f737be06700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f737419d690 con 0x7f73740fee50 2023-07-14T19:12:25.928 INFO:teuthology.orchestra.run.smithi130.stderr:2023-07-14T19:12:25.923+0000 7f737be06700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f737419d820 con 0x7f73740ff7e0 2023-07-14T19:12:25.929 INFO:teuthology.orchestra.run.smithi130.stderr:2023-07-14T19:12:25.923+0000 7f7379ba2700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f73740fee50 0x7f73741959c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:12:25.929 INFO:teuthology.orchestra.run.smithi130.stderr:2023-07-14T19:12:25.923+0000 7f7379ba2700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f73740fee50 0x7f73741959c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.130:39632/0 (socket says 172.21.15.130:39632) 2023-07-14T19:12:25.929 INFO:teuthology.orchestra.run.smithi130.stderr:2023-07-14T19:12:25.923+0000 7f7379ba2700 1 -- 172.21.15.130:0/3517383001 learned_addr learned my addr 172.21.15.130:0/3517383001 (peer_addr_for_me v2:172.21.15.130:0/0) 2023-07-14T19:12:25.929 INFO:teuthology.orchestra.run.smithi130.stderr:2023-07-14T19:12:25.923+0000 7f737a3a3700 1 --2- 172.21.15.130:0/3517383001 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f73741001a0 0x7f737419cf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:12:25.929 INFO:teuthology.orchestra.run.smithi130.stderr:2023-07-14T19:12:25.923+0000 7f73793a1700 1 --2- 172.21.15.130:0/3517383001 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f73740ff7e0 0x7f7374195f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:12:25.929 INFO:teuthology.orchestra.run.smithi130.stderr:2023-07-14T19:12:25.923+0000 7f7379ba2700 1 -- 172.21.15.130:0/3517383001 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f73740ff7e0 msgr2=0x7f7374195f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:12:25.930 INFO:teuthology.orchestra.run.smithi130.stderr:2023-07-14T19:12:25.923+0000 7f7379ba2700 1 --2- 172.21.15.130:0/3517383001 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f73740ff7e0 0x7f7374195f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:12:25.930 INFO:teuthology.orchestra.run.smithi130.stderr:2023-07-14T19:12:25.923+0000 7f7379ba2700 1 -- 172.21.15.130:0/3517383001 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f73741001a0 msgr2=0x7f737419cf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:12:25.930 INFO:teuthology.orchestra.run.smithi130.stderr:2023-07-14T19:12:25.923+0000 7f7379ba2700 1 --2- 172.21.15.130:0/3517383001 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f73741001a0 0x7f737419cf80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:12:25.930 INFO:teuthology.orchestra.run.smithi130.stderr:2023-07-14T19:12:25.923+0000 7f7379ba2700 1 -- 172.21.15.130:0/3517383001 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73700095f0 con 0x7f73740fee50 2023-07-14T19:12:25.930 INFO:teuthology.orchestra.run.smithi130.stderr:2023-07-14T19:12:25.923+0000 7f73793a1700 1 --2- 172.21.15.130:0/3517383001 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f73740ff7e0 0x7f7374195f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T19:12:25.930 INFO:teuthology.orchestra.run.smithi130.stderr:2023-07-14T19:12:25.923+0000 7f7379ba2700 1 --2- 172.21.15.130:0/3517383001 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f73740fee50 0x7f73741959c0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f7370000c00 tx=0x7f737000f700 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:12:25.931 INFO:teuthology.orchestra.run.smithi130.stderr:2023-07-14T19:12:25.923+0000 7f736affd700 1 -- 172.21.15.130:0/3517383001 <== mon.2 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f737001d070 con 0x7f73740fee50 2023-07-14T19:12:25.931 INFO:teuthology.orchestra.run.smithi130.stderr:2023-07-14T19:12:25.923+0000 7f736affd700 1 -- 172.21.15.130:0/3517383001 <== mon.2 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f737000fbf0 con 0x7f73740fee50 2023-07-14T19:12:25.931 INFO:teuthology.orchestra.run.smithi130.stderr:2023-07-14T19:12:25.923+0000 7f737be06700 1 -- 172.21.15.130:0/3517383001 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f737419daa0 con 0x7f73740fee50 2023-07-14T19:12:25.931 INFO:teuthology.orchestra.run.smithi130.stderr:2023-07-14T19:12:25.923+0000 7f736affd700 1 -- 172.21.15.130:0/3517383001 <== mon.2 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7370017650 con 0x7f73740fee50 2023-07-14T19:12:25.932 INFO:teuthology.orchestra.run.smithi130.stderr:2023-07-14T19:12:25.923+0000 7f737be06700 1 -- 172.21.15.130:0/3517383001 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7374101250 con 0x7f73740fee50 2023-07-14T19:12:25.932 INFO:teuthology.orchestra.run.smithi130.stderr:2023-07-14T19:12:25.924+0000 7f737be06700 1 -- 172.21.15.130:0/3517383001 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f737410a400 con 0x7f73740fee50 2023-07-14T19:12:25.932 INFO:teuthology.orchestra.run.smithi130.stderr:2023-07-14T19:12:25.925+0000 7f736affd700 1 -- 172.21.15.130:0/3517383001 <== mon.2 v2:172.21.15.130:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f73700177b0 con 0x7f73740fee50 2023-07-14T19:12:25.933 INFO:teuthology.orchestra.run.smithi130.stderr:2023-07-14T19:12:25.926+0000 7f736affd700 1 --2- 172.21.15.130:0/3517383001 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f73600a11d0 0x7f73600a3680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:12:25.933 INFO:teuthology.orchestra.run.smithi130.stderr:2023-07-14T19:12:25.926+0000 7f736affd700 1 -- 172.21.15.130:0/3517383001 <== mon.2 v2:172.21.15.130:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 3698+0+0 (secure 0 0 0) 0x7f73700cd0c0 con 0x7f73740fee50 2023-07-14T19:12:25.933 INFO:teuthology.orchestra.run.smithi130.stderr:2023-07-14T19:12:25.926+0000 7f73793a1700 1 --2- 172.21.15.130:0/3517383001 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f73600a11d0 0x7f73600a3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:12:25.933 INFO:teuthology.orchestra.run.smithi130.stderr:2023-07-14T19:12:25.927+0000 7f73793a1700 1 --2- 172.21.15.130:0/3517383001 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f73600a11d0 0x7f73600a3680 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f7364005fd0 tx=0x7f7364005e30 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:12:25.933 INFO:teuthology.orchestra.run.smithi130.stderr:2023-07-14T19:12:25.929+0000 7f736affd700 1 -- 172.21.15.130:0/3517383001 <== mon.2 v2:172.21.15.130:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f7370097a00 con 0x7f73740fee50 2023-07-14T19:12:25.971 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:12:25 smithi040 ceph-mon[96518]: purged_snaps scrub starts 2023-07-14T19:12:25.971 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:12:25 smithi040 ceph-mon[96518]: purged_snaps scrub ok 2023-07-14T19:12:25.972 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:12:25 smithi040 ceph-mon[96518]: Detected new or changed devices on smithi040 2023-07-14T19:12:25.972 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:12:25 smithi040 ceph-mon[96518]: Adjusting osd_memory_target on smithi040 to 8586M 2023-07-14T19:12:25.972 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:12:25 smithi040 ceph-mon[96518]: osd.3 [v2:172.21.15.40:6810/996287860,v1:172.21.15.40:6811/996287860] boot 2023-07-14T19:12:25.972 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:12:25 smithi040 ceph-mon[96518]: osdmap e25: 4 total, 4 up, 4 in 2023-07-14T19:12:25.972 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:12:25 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-07-14T19:12:25.973 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:12:25 smithi191 ceph-mon[98311]: purged_snaps scrub starts 2023-07-14T19:12:25.973 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:12:25 smithi191 ceph-mon[98311]: purged_snaps scrub ok 2023-07-14T19:12:25.973 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:12:25 smithi191 ceph-mon[98311]: Detected new or changed devices on smithi040 2023-07-14T19:12:25.973 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:12:25 smithi191 ceph-mon[98311]: Adjusting osd_memory_target on smithi040 to 8586M 2023-07-14T19:12:25.973 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:12:25 smithi191 ceph-mon[98311]: osd.3 [v2:172.21.15.40:6810/996287860,v1:172.21.15.40:6811/996287860] boot 2023-07-14T19:12:25.974 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:12:25 smithi191 ceph-mon[98311]: osdmap e25: 4 total, 4 up, 4 in 2023-07-14T19:12:25.974 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:12:25 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-07-14T19:12:25.992 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:12:25 smithi130 ceph-mon[97832]: purged_snaps scrub starts 2023-07-14T19:12:25.993 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:12:25 smithi130 ceph-mon[97832]: purged_snaps scrub ok 2023-07-14T19:12:25.993 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:12:25 smithi130 ceph-mon[97832]: Detected new or changed devices on smithi040 2023-07-14T19:12:25.993 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:12:25 smithi130 ceph-mon[97832]: Adjusting osd_memory_target on smithi040 to 8586M 2023-07-14T19:12:25.993 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:12:25 smithi130 ceph-mon[97832]: osd.3 [v2:172.21.15.40:6810/996287860,v1:172.21.15.40:6811/996287860] boot 2023-07-14T19:12:25.993 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:12:25 smithi130 ceph-mon[97832]: osdmap e25: 4 total, 4 up, 4 in 2023-07-14T19:12:25.993 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:12:25 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-07-14T19:12:26.124 INFO:teuthology.orchestra.run.smithi130.stderr:2023-07-14T19:12:26.122+0000 7f737be06700 1 -- 172.21.15.130:0/3517383001 --> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi130:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) v1 -- 0x7f73740fe470 con 0x7f73600a11d0 2023-07-14T19:12:26.968 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:12:26 smithi130 ceph-mon[97832]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail 2023-07-14T19:12:26.968 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:12:26 smithi130 ceph-mon[97832]: osdmap e26: 4 total, 4 up, 4 in 2023-07-14T19:12:26.969 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:12:26 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-14T19:12:26.969 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:12:26 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-14T19:12:26.969 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:12:26 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:12:26.969 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:12:26 smithi130 ceph-mon[97832]: osdmap e27: 4 total, 4 up, 4 in 2023-07-14T19:12:26.971 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:12:26 smithi040 ceph-mon[96518]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail 2023-07-14T19:12:26.971 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:12:26 smithi040 ceph-mon[96518]: osdmap e26: 4 total, 4 up, 4 in 2023-07-14T19:12:26.971 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:12:26 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-14T19:12:26.972 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:12:26 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-14T19:12:26.972 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:12:26 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:12:26.972 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:12:26 smithi040 ceph-mon[96518]: osdmap e27: 4 total, 4 up, 4 in 2023-07-14T19:12:26.972 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:12:26 smithi191 ceph-mon[98311]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail 2023-07-14T19:12:26.972 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:12:26 smithi191 ceph-mon[98311]: osdmap e26: 4 total, 4 up, 4 in 2023-07-14T19:12:26.973 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:12:26 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-14T19:12:26.973 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:12:26 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-14T19:12:26.973 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:12:26 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:12:26.973 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:12:26 smithi191 ceph-mon[98311]: osdmap e27: 4 total, 4 up, 4 in 2023-07-14T19:12:27.971 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:12:27 smithi040 ceph-mon[96518]: from='client.24221 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi130:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T19:12:27.972 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:12:27 smithi191 ceph-mon[98311]: from='client.24221 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi130:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T19:12:27.992 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:12:27 smithi130 ceph-mon[97832]: from='client.24221 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi130:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T19:12:28.971 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:12:28 smithi040 ceph-mon[96518]: pgmap v95: 1 pgs: 1 activating+undersized+degraded+remapped; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail; 2/6 objects degraded (33.333%) 2023-07-14T19:12:28.971 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:12:28 smithi040 ceph-mon[96518]: Health check failed: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded (PG_DEGRADED) 2023-07-14T19:12:28.972 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:12:28 smithi040 ceph-mon[96518]: osdmap e28: 4 total, 4 up, 4 in 2023-07-14T19:12:28.972 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:12:28 smithi040 ceph-mon[96518]: from='client.? 172.21.15.130:0/780844195' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c5501b35-f62d-482d-8d22-115c88f0298e"}]: dispatch 2023-07-14T19:12:28.972 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:12:28 smithi040 ceph-mon[96518]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c5501b35-f62d-482d-8d22-115c88f0298e"}]: dispatch 2023-07-14T19:12:28.972 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:12:28 smithi040 ceph-mon[96518]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c5501b35-f62d-482d-8d22-115c88f0298e"}]': finished 2023-07-14T19:12:28.972 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:12:28 smithi040 ceph-mon[96518]: osdmap e29: 5 total, 4 up, 5 in 2023-07-14T19:12:28.972 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:12:28 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-07-14T19:12:28.973 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:12:28 smithi191 ceph-mon[98311]: pgmap v95: 1 pgs: 1 activating+undersized+degraded+remapped; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail; 2/6 objects degraded (33.333%) 2023-07-14T19:12:28.973 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:12:28 smithi191 ceph-mon[98311]: Health check failed: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded (PG_DEGRADED) 2023-07-14T19:12:28.973 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:12:28 smithi191 ceph-mon[98311]: osdmap e28: 4 total, 4 up, 4 in 2023-07-14T19:12:28.973 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:12:28 smithi191 ceph-mon[98311]: from='client.? 172.21.15.130:0/780844195' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c5501b35-f62d-482d-8d22-115c88f0298e"}]: dispatch 2023-07-14T19:12:28.973 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:12:28 smithi191 ceph-mon[98311]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c5501b35-f62d-482d-8d22-115c88f0298e"}]: dispatch 2023-07-14T19:12:28.973 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:12:28 smithi191 ceph-mon[98311]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c5501b35-f62d-482d-8d22-115c88f0298e"}]': finished 2023-07-14T19:12:28.974 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:12:28 smithi191 ceph-mon[98311]: osdmap e29: 5 total, 4 up, 5 in 2023-07-14T19:12:28.974 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:12:28 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-07-14T19:12:28.992 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:12:28 smithi130 ceph-mon[97832]: pgmap v95: 1 pgs: 1 activating+undersized+degraded+remapped; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail; 2/6 objects degraded (33.333%) 2023-07-14T19:12:28.993 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:12:28 smithi130 ceph-mon[97832]: Health check failed: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded (PG_DEGRADED) 2023-07-14T19:12:28.993 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:12:28 smithi130 ceph-mon[97832]: osdmap e28: 4 total, 4 up, 4 in 2023-07-14T19:12:28.993 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:12:28 smithi130 ceph-mon[97832]: from='client.? 172.21.15.130:0/780844195' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c5501b35-f62d-482d-8d22-115c88f0298e"}]: dispatch 2023-07-14T19:12:28.993 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:12:28 smithi130 ceph-mon[97832]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c5501b35-f62d-482d-8d22-115c88f0298e"}]: dispatch 2023-07-14T19:12:28.993 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:12:28 smithi130 ceph-mon[97832]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c5501b35-f62d-482d-8d22-115c88f0298e"}]': finished 2023-07-14T19:12:28.993 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:12:28 smithi130 ceph-mon[97832]: osdmap e29: 5 total, 4 up, 5 in 2023-07-14T19:12:28.994 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:12:28 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-07-14T19:12:29.971 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:12:29 smithi040 ceph-mon[96518]: from='client.? 172.21.15.130:0/1022025972' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-14T19:12:29.972 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:12:29 smithi191 ceph-mon[98311]: from='client.? 172.21.15.130:0/1022025972' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-14T19:12:29.992 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:12:29 smithi130 ceph-mon[97832]: from='client.? 172.21.15.130:0/1022025972' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-14T19:12:30.971 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:12:30 smithi040 ceph-mon[96518]: pgmap v98: 1 pgs: 1 activating+undersized+degraded+remapped; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail; 2/6 objects degraded (33.333%) 2023-07-14T19:12:30.972 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:12:30 smithi191 ceph-mon[98311]: pgmap v98: 1 pgs: 1 activating+undersized+degraded+remapped; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail; 2/6 objects degraded (33.333%) 2023-07-14T19:12:30.992 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:12:30 smithi130 ceph-mon[97832]: pgmap v98: 1 pgs: 1 activating+undersized+degraded+remapped; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail; 2/6 objects degraded (33.333%) 2023-07-14T19:12:32.971 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:12:32 smithi040 ceph-mon[96518]: pgmap v99: 1 pgs: 1 activating+undersized+degraded+remapped; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail; 2/6 objects degraded (33.333%) 2023-07-14T19:12:32.972 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:12:32 smithi191 ceph-mon[98311]: pgmap v99: 1 pgs: 1 activating+undersized+degraded+remapped; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail; 2/6 objects degraded (33.333%) 2023-07-14T19:12:32.992 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:12:32 smithi130 ceph-mon[97832]: pgmap v99: 1 pgs: 1 activating+undersized+degraded+remapped; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail; 2/6 objects degraded (33.333%) 2023-07-14T19:12:34.221 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:12:33 smithi040 ceph-mon[96518]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail; 95 KiB/s, 0 objects/s recovering 2023-07-14T19:12:34.221 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:12:33 smithi040 ceph-mon[96518]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded) 2023-07-14T19:12:34.222 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:12:33 smithi040 ceph-mon[96518]: Cluster is now healthy 2023-07-14T19:12:34.222 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:12:33 smithi191 ceph-mon[98311]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail; 95 KiB/s, 0 objects/s recovering 2023-07-14T19:12:34.222 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:12:33 smithi191 ceph-mon[98311]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded) 2023-07-14T19:12:34.222 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:12:33 smithi191 ceph-mon[98311]: Cluster is now healthy 2023-07-14T19:12:34.242 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:12:33 smithi130 ceph-mon[97832]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail; 95 KiB/s, 0 objects/s recovering 2023-07-14T19:12:34.243 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:12:33 smithi130 ceph-mon[97832]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded) 2023-07-14T19:12:34.243 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:12:33 smithi130 ceph-mon[97832]: Cluster is now healthy 2023-07-14T19:12:36.086 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:12:35 smithi130 ceph-mon[97832]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail; 72 KiB/s, 0 objects/s recovering 2023-07-14T19:12:36.086 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:12:35 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2023-07-14T19:12:36.086 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:12:35 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:12:36.086 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:12:35 smithi130 ceph-mon[97832]: Deploying daemon osd.4 on smithi130 2023-07-14T19:12:36.221 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:12:35 smithi040 ceph-mon[96518]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail; 72 KiB/s, 0 objects/s recovering 2023-07-14T19:12:36.221 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:12:35 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2023-07-14T19:12:36.222 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:12:35 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:12:36.222 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:12:35 smithi040 ceph-mon[96518]: Deploying daemon osd.4 on smithi130 2023-07-14T19:12:36.222 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:12:35 smithi191 ceph-mon[98311]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail; 72 KiB/s, 0 objects/s recovering 2023-07-14T19:12:36.222 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:12:35 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2023-07-14T19:12:36.223 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:12:35 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:12:36.223 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:12:35 smithi191 ceph-mon[98311]: Deploying daemon osd.4 on smithi130 2023-07-14T19:12:38.869 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:12:38 smithi130 ceph-mon[97832]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail; 64 KiB/s, 0 objects/s recovering 2023-07-14T19:12:38.971 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:12:38 smithi040 ceph-mon[96518]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail; 64 KiB/s, 0 objects/s recovering 2023-07-14T19:12:38.972 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:12:38 smithi191 ceph-mon[98311]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail; 64 KiB/s, 0 objects/s recovering 2023-07-14T19:12:40.971 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:12:40 smithi040 ceph-mon[96518]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail; 56 KiB/s, 0 objects/s recovering 2023-07-14T19:12:40.972 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:12:40 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:12:40.972 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:12:40 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:12:40.972 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:12:40 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T19:12:40.972 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:12:40 smithi191 ceph-mon[98311]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail; 56 KiB/s, 0 objects/s recovering 2023-07-14T19:12:40.972 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:12:40 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:12:40.973 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:12:40 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:12:40.973 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:12:40 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T19:12:40.992 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:12:40 smithi130 ceph-mon[97832]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail; 56 KiB/s, 0 objects/s recovering 2023-07-14T19:12:40.992 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:12:40 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:12:40.993 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:12:40 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:12:40.993 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:12:40 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T19:12:42.891 INFO:teuthology.orchestra.run.smithi130.stderr:2023-07-14T19:12:42.889+0000 7f736affd700 1 -- 172.21.15.130:0/3517383001 <== mgr.14152 v2:172.21.15.40:6800/1693027118 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f73740fe470 con 0x7f73600a11d0 2023-07-14T19:12:42.891 INFO:teuthology.orchestra.run.smithi130.stdout:Created osd(s) 4 on host 'smithi130' 2023-07-14T19:12:42.893 INFO:teuthology.orchestra.run.smithi130.stderr:2023-07-14T19:12:42.892+0000 7f737be06700 1 -- 172.21.15.130:0/3517383001 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f73600a11d0 msgr2=0x7f73600a3680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:12:42.893 INFO:teuthology.orchestra.run.smithi130.stderr:2023-07-14T19:12:42.892+0000 7f737be06700 1 --2- 172.21.15.130:0/3517383001 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f73600a11d0 0x7f73600a3680 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f7364005fd0 tx=0x7f7364005e30 comp rx=0 tx=0).stop 2023-07-14T19:12:42.893 INFO:teuthology.orchestra.run.smithi130.stderr:2023-07-14T19:12:42.892+0000 7f737be06700 1 -- 172.21.15.130:0/3517383001 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f73740fee50 msgr2=0x7f73741959c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:12:42.893 INFO:teuthology.orchestra.run.smithi130.stderr:2023-07-14T19:12:42.892+0000 7f737be06700 1 --2- 172.21.15.130:0/3517383001 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f73740fee50 0x7f73741959c0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f7370000c00 tx=0x7f737000f700 comp rx=0 tx=0).stop 2023-07-14T19:12:42.894 INFO:teuthology.orchestra.run.smithi130.stderr:2023-07-14T19:12:42.892+0000 7f737be06700 1 -- 172.21.15.130:0/3517383001 shutdown_connections 2023-07-14T19:12:42.894 INFO:teuthology.orchestra.run.smithi130.stderr:2023-07-14T19:12:42.892+0000 7f737be06700 1 --2- 172.21.15.130:0/3517383001 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f73600a11d0 0x7f73600a3680 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:12:42.894 INFO:teuthology.orchestra.run.smithi130.stderr:2023-07-14T19:12:42.892+0000 7f737be06700 1 --2- 172.21.15.130:0/3517383001 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f73741001a0 0x7f737419cf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:12:42.894 INFO:teuthology.orchestra.run.smithi130.stderr:2023-07-14T19:12:42.892+0000 7f737be06700 1 --2- 172.21.15.130:0/3517383001 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f73740fee50 0x7f73741959c0 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:12:42.894 INFO:teuthology.orchestra.run.smithi130.stderr:2023-07-14T19:12:42.892+0000 7f737be06700 1 --2- 172.21.15.130:0/3517383001 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f73740ff7e0 0x7f7374195f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:12:42.894 INFO:teuthology.orchestra.run.smithi130.stderr:2023-07-14T19:12:42.892+0000 7f737be06700 1 -- 172.21.15.130:0/3517383001 >> 172.21.15.130:0/3517383001 conn(0x7f73740f9990 msgr2=0x7f73740fdd60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:12:42.894 INFO:teuthology.orchestra.run.smithi130.stderr:2023-07-14T19:12:42.892+0000 7f737be06700 1 -- 172.21.15.130:0/3517383001 shutdown_connections 2023-07-14T19:12:42.895 INFO:teuthology.orchestra.run.smithi130.stderr:2023-07-14T19:12:42.892+0000 7f737be06700 1 -- 172.21.15.130:0/3517383001 wait complete. 2023-07-14T19:12:42.971 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:12:42 smithi040 ceph-mon[96518]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail; 48 KiB/s, 0 objects/s recovering 2023-07-14T19:12:42.971 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:12:42 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:12:42.972 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:12:42 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:12:42.972 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:12:42 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:12:42.972 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:12:42 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T19:12:42.972 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:12:42 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:12:42.972 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:12:42 smithi191 ceph-mon[98311]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail; 48 KiB/s, 0 objects/s recovering 2023-07-14T19:12:42.973 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:12:42 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:12:42.973 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:12:42 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:12:42.973 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:12:42 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:12:42.973 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:12:42 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T19:12:42.973 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:12:42 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:12:42.992 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:12:42 smithi130 ceph-mon[97832]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail; 48 KiB/s, 0 objects/s recovering 2023-07-14T19:12:42.992 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:12:42 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:12:42.992 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:12:42 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:12:42.993 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:12:42 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:12:42.993 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:12:42 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T19:12:42.993 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:12:42 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:12:43.287 DEBUG:teuthology.orchestra.run.smithi130:osd.4> sudo journalctl -f -n 0 -u ceph-adc52f04-2279-11ee-9b33-001a4aab830c@osd.4.service 2023-07-14T19:12:43.289 INFO:tasks.cephadm:Deploying osd.5 on smithi191 with /dev/vg_nvme/lv_3... 2023-07-14T19:12:43.289 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid adc52f04-2279-11ee-9b33-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2023-07-14T19:12:43.330 INFO:journalctl@ceph.osd.4.smithi130.stdout:-- Logs begin at Fri 2023-07-14 18:50:32 UTC. -- 2023-07-14T19:12:43.742 INFO:journalctl@ceph.osd.4.smithi130.stdout:Jul 14 19:12:43 smithi130 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-4[108332]: 2023-07-14T19:12:43.568+0000 7fae2b8e8640 -1 osd.4 0 log_to_monitors true 2023-07-14T19:12:43.860 INFO:teuthology.orchestra.run.smithi191.stderr:Inferring config /var/lib/ceph/adc52f04-2279-11ee-9b33-001a4aab830c/mon.c/config 2023-07-14T19:12:44.109 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:12:43 smithi130 ceph-mon[97832]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail; 48 KiB/s, 0 objects/s recovering 2023-07-14T19:12:44.109 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:12:43 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:12:44.110 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:12:43 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T19:12:44.110 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:12:43 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:12:44.110 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:12:43 smithi130 ceph-mon[97832]: from='osd.4 [v2:172.21.15.130:6808/3507448720,v1:172.21.15.130:6809/3507448720]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2023-07-14T19:12:44.110 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:12:43 smithi130 ceph-mon[97832]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2023-07-14T19:12:44.133 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:12:43 smithi191 ceph-mon[98311]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail; 48 KiB/s, 0 objects/s recovering 2023-07-14T19:12:44.133 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:12:43 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:12:44.134 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:12:43 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T19:12:44.134 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:12:43 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:12:44.134 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:12:43 smithi191 ceph-mon[98311]: from='osd.4 [v2:172.21.15.130:6808/3507448720,v1:172.21.15.130:6809/3507448720]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2023-07-14T19:12:44.134 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:12:43 smithi191 ceph-mon[98311]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2023-07-14T19:12:44.221 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:12:43 smithi040 ceph-mon[96518]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail; 48 KiB/s, 0 objects/s recovering 2023-07-14T19:12:44.222 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:12:43 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:12:44.222 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:12:43 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T19:12:44.222 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:12:43 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:12:44.222 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:12:43 smithi040 ceph-mon[96518]: from='osd.4 [v2:172.21.15.130:6808/3507448720,v1:172.21.15.130:6809/3507448720]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2023-07-14T19:12:44.222 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:12:43 smithi040 ceph-mon[96518]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2023-07-14T19:12:44.972 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:12:44 smithi191 ceph-mon[98311]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2023-07-14T19:12:44.972 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:12:44 smithi191 ceph-mon[98311]: osdmap e30: 5 total, 4 up, 5 in 2023-07-14T19:12:44.973 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:12:44 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-07-14T19:12:44.973 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:12:44 smithi191 ceph-mon[98311]: from='osd.4 [v2:172.21.15.130:6808/3507448720,v1:172.21.15.130:6809/3507448720]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi130", "root=default"]}]: dispatch 2023-07-14T19:12:44.973 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:12:44 smithi191 ceph-mon[98311]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi130", "root=default"]}]: dispatch 2023-07-14T19:12:45.210 INFO:journalctl@ceph.osd.4.smithi130.stdout:Jul 14 19:12:44 smithi130 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-4[108332]: 2023-07-14T19:12:44.896+0000 7fae20f60700 -1 osd.4 0 waiting for initial osdmap 2023-07-14T19:12:45.210 INFO:journalctl@ceph.osd.4.smithi130.stdout:Jul 14 19:12:44 smithi130 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-4[108332]: 2023-07-14T19:12:44.902+0000 7fae1bd54700 -1 osd.4 31 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-07-14T19:12:45.210 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:12:44 smithi130 ceph-mon[97832]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2023-07-14T19:12:45.210 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:12:44 smithi130 ceph-mon[97832]: osdmap e30: 5 total, 4 up, 5 in 2023-07-14T19:12:45.211 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:12:44 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-07-14T19:12:45.211 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:12:44 smithi130 ceph-mon[97832]: from='osd.4 [v2:172.21.15.130:6808/3507448720,v1:172.21.15.130:6809/3507448720]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi130", "root=default"]}]: dispatch 2023-07-14T19:12:45.211 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:12:44 smithi130 ceph-mon[97832]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi130", "root=default"]}]: dispatch 2023-07-14T19:12:45.221 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:12:44 smithi040 ceph-mon[96518]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2023-07-14T19:12:45.221 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:12:44 smithi040 ceph-mon[96518]: osdmap e30: 5 total, 4 up, 5 in 2023-07-14T19:12:45.222 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:12:44 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-07-14T19:12:45.222 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:12:44 smithi040 ceph-mon[96518]: from='osd.4 [v2:172.21.15.130:6808/3507448720,v1:172.21.15.130:6809/3507448720]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi130", "root=default"]}]: dispatch 2023-07-14T19:12:45.222 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:12:44 smithi040 ceph-mon[96518]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi130", "root=default"]}]: dispatch 2023-07-14T19:12:45.498 INFO:teuthology.orchestra.run.smithi191.stderr:--> Zapping: /dev/vg_nvme/lv_3 2023-07-14T19:12:45.498 INFO:teuthology.orchestra.run.smithi191.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2023-07-14T19:12:45.498 INFO:teuthology.orchestra.run.smithi191.stderr: stderr: 10+0 records in 2023-07-14T19:12:45.498 INFO:teuthology.orchestra.run.smithi191.stderr:10+0 records out 2023-07-14T19:12:45.499 INFO:teuthology.orchestra.run.smithi191.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0110168 s, 952 MB/s 2023-07-14T19:12:45.499 INFO:teuthology.orchestra.run.smithi191.stderr:--> Zapping successful for: 2023-07-14T19:12:45.917 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid adc52f04-2279-11ee-9b33-001a4aab830c -- ceph orch daemon add osd smithi191:vg_nvme/lv_3 2023-07-14T19:12:46.221 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:12:45 smithi040 ceph-mon[96518]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail 2023-07-14T19:12:46.222 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:12:45 smithi040 ceph-mon[96518]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi130", "root=default"]}]': finished 2023-07-14T19:12:46.222 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:12:45 smithi040 ceph-mon[96518]: osdmap e31: 5 total, 4 up, 5 in 2023-07-14T19:12:46.222 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:12:45 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-07-14T19:12:46.222 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:12:45 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-07-14T19:12:46.222 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:12:45 smithi040 ceph-mon[96518]: Detected new or changed devices on smithi130 2023-07-14T19:12:46.222 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:12:45 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:12:46.223 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:12:45 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:12:46.223 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:12:45 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-07-14T19:12:46.223 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:12:45 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-07-14T19:12:46.223 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:12:45 smithi040 ceph-mon[96518]: Adjusting osd_memory_target on smithi130 to 8584M 2023-07-14T19:12:46.223 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:12:45 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:12:46.223 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:12:45 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:12:46.223 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:12:45 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T19:12:46.223 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:12:45 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:12:46.224 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:12:45 smithi191 ceph-mon[98311]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail 2023-07-14T19:12:46.224 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:12:45 smithi191 ceph-mon[98311]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi130", "root=default"]}]': finished 2023-07-14T19:12:46.224 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:12:45 smithi191 ceph-mon[98311]: osdmap e31: 5 total, 4 up, 5 in 2023-07-14T19:12:46.224 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:12:45 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-07-14T19:12:46.225 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:12:45 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-07-14T19:12:46.225 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:12:45 smithi191 ceph-mon[98311]: Detected new or changed devices on smithi130 2023-07-14T19:12:46.225 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:12:45 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:12:46.225 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:12:45 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:12:46.225 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:12:45 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-07-14T19:12:46.225 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:12:45 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-07-14T19:12:46.225 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:12:45 smithi191 ceph-mon[98311]: Adjusting osd_memory_target on smithi130 to 8584M 2023-07-14T19:12:46.225 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:12:45 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:12:46.226 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:12:45 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:12:46.226 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:12:45 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T19:12:46.226 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:12:45 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:12:46.242 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:12:45 smithi130 ceph-mon[97832]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail 2023-07-14T19:12:46.243 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:12:45 smithi130 ceph-mon[97832]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi130", "root=default"]}]': finished 2023-07-14T19:12:46.243 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:12:45 smithi130 ceph-mon[97832]: osdmap e31: 5 total, 4 up, 5 in 2023-07-14T19:12:46.243 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:12:45 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-07-14T19:12:46.243 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:12:45 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-07-14T19:12:46.243 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:12:45 smithi130 ceph-mon[97832]: Detected new or changed devices on smithi130 2023-07-14T19:12:46.243 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:12:45 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:12:46.243 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:12:45 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:12:46.244 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:12:45 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-07-14T19:12:46.244 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:12:45 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-07-14T19:12:46.244 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:12:45 smithi130 ceph-mon[97832]: Adjusting osd_memory_target on smithi130 to 8584M 2023-07-14T19:12:46.244 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:12:45 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:12:46.244 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:12:45 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:12:46.244 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:12:45 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T19:12:46.244 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:12:45 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:12:46.469 INFO:teuthology.orchestra.run.smithi191.stderr:Inferring config /var/lib/ceph/adc52f04-2279-11ee-9b33-001a4aab830c/mon.c/config 2023-07-14T19:12:46.972 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:12:46 smithi191 ceph-mon[98311]: purged_snaps scrub starts 2023-07-14T19:12:46.972 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:12:46 smithi191 ceph-mon[98311]: purged_snaps scrub ok 2023-07-14T19:12:46.973 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:12:46 smithi191 ceph-mon[98311]: osd.4 [v2:172.21.15.130:6808/3507448720,v1:172.21.15.130:6809/3507448720] boot 2023-07-14T19:12:46.973 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:12:46 smithi191 ceph-mon[98311]: osdmap e32: 5 total, 5 up, 5 in 2023-07-14T19:12:46.973 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:12:46 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-07-14T19:12:47.221 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:12:46 smithi040 ceph-mon[96518]: purged_snaps scrub starts 2023-07-14T19:12:47.221 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:12:46 smithi040 ceph-mon[96518]: purged_snaps scrub ok 2023-07-14T19:12:47.222 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:12:46 smithi040 ceph-mon[96518]: osd.4 [v2:172.21.15.130:6808/3507448720,v1:172.21.15.130:6809/3507448720] boot 2023-07-14T19:12:47.222 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:12:46 smithi040 ceph-mon[96518]: osdmap e32: 5 total, 5 up, 5 in 2023-07-14T19:12:47.222 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:12:46 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-07-14T19:12:47.242 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:12:46 smithi130 ceph-mon[97832]: purged_snaps scrub starts 2023-07-14T19:12:47.243 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:12:46 smithi130 ceph-mon[97832]: purged_snaps scrub ok 2023-07-14T19:12:47.243 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:12:46 smithi130 ceph-mon[97832]: osd.4 [v2:172.21.15.130:6808/3507448720,v1:172.21.15.130:6809/3507448720] boot 2023-07-14T19:12:47.243 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:12:46 smithi130 ceph-mon[97832]: osdmap e32: 5 total, 5 up, 5 in 2023-07-14T19:12:47.243 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:12:46 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-07-14T19:12:47.788 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:12:47.785+0000 7f20ddb28700 1 -- 172.21.15.191:0/2612996455 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f20d80ff7a0 msgr2=0x7f20d8107050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:12:47.788 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:12:47.785+0000 7f20ddb28700 1 --2- 172.21.15.191:0/2612996455 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f20d80ff7a0 0x7f20d8107050 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f20d000b0a0 tx=0x7f20d000b3b0 comp rx=0 tx=0).stop 2023-07-14T19:12:47.788 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:12:47.785+0000 7f20ddb28700 1 -- 172.21.15.191:0/2612996455 shutdown_connections 2023-07-14T19:12:47.788 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:12:47.785+0000 7f20ddb28700 1 --2- 172.21.15.191:0/2612996455 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f20d80ff7a0 0x7f20d8107050 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:12:47.789 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:12:47.785+0000 7f20ddb28700 1 --2- 172.21.15.191:0/2612996455 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f20d8100190 0x7f20d81005a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:12:47.789 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:12:47.785+0000 7f20ddb28700 1 --2- 172.21.15.191:0/2612996455 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f20d80fedf0 0x7f20d80ff260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:12:47.789 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:12:47.785+0000 7f20ddb28700 1 -- 172.21.15.191:0/2612996455 >> 172.21.15.191:0/2612996455 conn(0x7f20d80f9990 msgr2=0x7f20d80fbdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:12:47.789 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:12:47.786+0000 7f20ddb28700 1 -- 172.21.15.191:0/2612996455 shutdown_connections 2023-07-14T19:12:47.789 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:12:47.786+0000 7f20ddb28700 1 -- 172.21.15.191:0/2612996455 wait complete. 2023-07-14T19:12:47.791 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:12:47.786+0000 7f20ddb28700 1 Processor -- start 2023-07-14T19:12:47.791 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:12:47.786+0000 7f20ddb28700 1 -- start start 2023-07-14T19:12:47.791 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:12:47.786+0000 7f20ddb28700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f20d80fedf0 0x7f20d8102b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:12:47.791 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:12:47.786+0000 7f20ddb28700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f20d80ff7a0 0x7f20d81011e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:12:47.791 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:12:47.786+0000 7f20ddb28700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f20d8100190 0x7f20d8101720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:12:47.791 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:12:47.786+0000 7f20ddb28700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20d8101ea0 con 0x7f20d8100190 2023-07-14T19:12:47.792 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:12:47.786+0000 7f20ddb28700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20d8102010 con 0x7f20d80fedf0 2023-07-14T19:12:47.792 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:12:47.786+0000 7f20ddb28700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20d819fb30 con 0x7f20d80ff7a0 2023-07-14T19:12:47.792 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:12:47.786+0000 7f20d6ffd700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f20d80ff7a0 0x7f20d81011e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:12:47.792 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:12:47.787+0000 7f20d6ffd700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f20d80ff7a0 0x7f20d81011e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.191:59358/0 (socket says 172.21.15.191:59358) 2023-07-14T19:12:47.792 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:12:47.787+0000 7f20d6ffd700 1 -- 172.21.15.191:0/2020083770 learned_addr learned my addr 172.21.15.191:0/2020083770 (peer_addr_for_me v2:172.21.15.191:0/0) 2023-07-14T19:12:47.792 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:12:47.787+0000 7f20d77fe700 1 --2- 172.21.15.191:0/2020083770 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f20d80fedf0 0x7f20d8102b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:12:47.793 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:12:47.787+0000 7f20d7fff700 1 --2- 172.21.15.191:0/2020083770 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f20d8100190 0x7f20d8101720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:12:47.793 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:12:47.787+0000 7f20d6ffd700 1 -- 172.21.15.191:0/2020083770 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f20d80fedf0 msgr2=0x7f20d8102b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:12:47.793 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:12:47.787+0000 7f20d6ffd700 1 --2- 172.21.15.191:0/2020083770 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f20d80fedf0 0x7f20d8102b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:12:47.793 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:12:47.787+0000 7f20d6ffd700 1 -- 172.21.15.191:0/2020083770 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f20d8100190 msgr2=0x7f20d8101720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:12:47.793 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:12:47.787+0000 7f20d6ffd700 1 --2- 172.21.15.191:0/2020083770 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f20d8100190 0x7f20d8101720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:12:47.793 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:12:47.787+0000 7f20d6ffd700 1 -- 172.21.15.191:0/2020083770 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20d0009d00 con 0x7f20d80ff7a0 2023-07-14T19:12:47.793 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:12:47.787+0000 7f20d6ffd700 1 --2- 172.21.15.191:0/2020083770 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f20d80ff7a0 0x7f20d81011e0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f20cc00d980 tx=0x7f20cc00dc90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:12:47.794 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:12:47.787+0000 7f20d4ff9700 1 -- 172.21.15.191:0/2020083770 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20cc004020 con 0x7f20d80ff7a0 2023-07-14T19:12:47.794 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:12:47.787+0000 7f20d4ff9700 1 -- 172.21.15.191:0/2020083770 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f20cc00f690 con 0x7f20d80ff7a0 2023-07-14T19:12:47.794 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:12:47.787+0000 7f20ddb28700 1 -- 172.21.15.191:0/2020083770 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f20d819fe10 con 0x7f20d80ff7a0 2023-07-14T19:12:47.794 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:12:47.787+0000 7f20d4ff9700 1 -- 172.21.15.191:0/2020083770 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20cc003a60 con 0x7f20d80ff7a0 2023-07-14T19:12:47.794 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:12:47.787+0000 7f20ddb28700 1 -- 172.21.15.191:0/2020083770 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f20d81a0280 con 0x7f20d80ff7a0 2023-07-14T19:12:47.794 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:12:47.788+0000 7f20ddb28700 1 -- 172.21.15.191:0/2020083770 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f20d810a3b0 con 0x7f20d80ff7a0 2023-07-14T19:12:47.795 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:12:47.789+0000 7f20d4ff9700 1 -- 172.21.15.191:0/2020083770 <== mon.1 v2:172.21.15.191:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f20cc00fb00 con 0x7f20d80ff7a0 2023-07-14T19:12:47.795 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:12:47.790+0000 7f20d4ff9700 1 --2- 172.21.15.191:0/2020083770 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f20c00a1150 0x7f20c00a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:12:47.795 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:12:47.790+0000 7f20d4ff9700 1 -- 172.21.15.191:0/2020083770 <== mon.1 v2:172.21.15.191:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 4101+0+0 (secure 0 0 0) 0x7f20cc0cb770 con 0x7f20d80ff7a0 2023-07-14T19:12:47.795 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:12:47.790+0000 7f20d77fe700 1 --2- 172.21.15.191:0/2020083770 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f20c00a1150 0x7f20c00a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:12:47.796 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:12:47.790+0000 7f20d77fe700 1 --2- 172.21.15.191:0/2020083770 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f20c00a1150 0x7f20c00a3600 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f20d8067bd0 tx=0x7f20c8009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:12:47.796 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:12:47.793+0000 7f20d4ff9700 1 -- 172.21.15.191:0/2020083770 <== mon.1 v2:172.21.15.191:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f20cc095f30 con 0x7f20d80ff7a0 2023-07-14T19:12:47.989 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:12:47.987+0000 7f20ddb28700 1 -- 172.21.15.191:0/2020083770 --> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi191:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) v1 -- 0x7f20d80fe430 con 0x7f20c00a1150 2023-07-14T19:12:48.810 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:12:48 smithi191 ceph-mon[98311]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail 2023-07-14T19:12:48.811 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:12:48 smithi191 ceph-mon[98311]: osdmap e33: 5 total, 5 up, 5 in 2023-07-14T19:12:48.811 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:12:48 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-14T19:12:48.811 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:12:48 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-14T19:12:48.811 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:12:48 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:12:48.971 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:12:48 smithi040 ceph-mon[96518]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail 2023-07-14T19:12:48.972 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:12:48 smithi040 ceph-mon[96518]: osdmap e33: 5 total, 5 up, 5 in 2023-07-14T19:12:48.972 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:12:48 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-14T19:12:48.972 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:12:48 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-14T19:12:48.972 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:12:48 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:12:48.992 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:12:48 smithi130 ceph-mon[97832]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail 2023-07-14T19:12:48.993 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:12:48 smithi130 ceph-mon[97832]: osdmap e33: 5 total, 5 up, 5 in 2023-07-14T19:12:48.993 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:12:48 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-14T19:12:48.993 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:12:48 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-14T19:12:48.993 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:12:48 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:12:49.971 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:12:49 smithi040 ceph-mon[96518]: from='client.24238 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi191:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T19:12:49.972 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:12:49 smithi191 ceph-mon[98311]: from='client.24238 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi191:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T19:12:49.992 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:12:49 smithi130 ceph-mon[97832]: from='client.24238 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi191:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T19:12:50.971 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:12:50 smithi040 ceph-mon[96518]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 133 MiB used, 447 GiB / 447 GiB avail 2023-07-14T19:12:50.972 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:12:50 smithi040 ceph-mon[96518]: from='client.? 172.21.15.191:0/3046248289' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7770b6d6-5874-493a-986a-84db482cee42"}]: dispatch 2023-07-14T19:12:50.972 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:12:50 smithi040 ceph-mon[96518]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7770b6d6-5874-493a-986a-84db482cee42"}]: dispatch 2023-07-14T19:12:50.972 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:12:50 smithi040 ceph-mon[96518]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7770b6d6-5874-493a-986a-84db482cee42"}]': finished 2023-07-14T19:12:50.972 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:12:50 smithi040 ceph-mon[96518]: osdmap e34: 6 total, 5 up, 6 in 2023-07-14T19:12:50.972 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:12:50 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-07-14T19:12:50.973 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:12:50 smithi191 ceph-mon[98311]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 133 MiB used, 447 GiB / 447 GiB avail 2023-07-14T19:12:50.973 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:12:50 smithi191 ceph-mon[98311]: from='client.? 172.21.15.191:0/3046248289' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7770b6d6-5874-493a-986a-84db482cee42"}]: dispatch 2023-07-14T19:12:50.973 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:12:50 smithi191 ceph-mon[98311]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7770b6d6-5874-493a-986a-84db482cee42"}]: dispatch 2023-07-14T19:12:50.973 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:12:50 smithi191 ceph-mon[98311]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7770b6d6-5874-493a-986a-84db482cee42"}]': finished 2023-07-14T19:12:50.973 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:12:50 smithi191 ceph-mon[98311]: osdmap e34: 6 total, 5 up, 6 in 2023-07-14T19:12:50.973 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:12:50 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-07-14T19:12:50.992 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:12:50 smithi130 ceph-mon[97832]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 133 MiB used, 447 GiB / 447 GiB avail 2023-07-14T19:12:50.992 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:12:50 smithi130 ceph-mon[97832]: from='client.? 172.21.15.191:0/3046248289' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7770b6d6-5874-493a-986a-84db482cee42"}]: dispatch 2023-07-14T19:12:50.993 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:12:50 smithi130 ceph-mon[97832]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7770b6d6-5874-493a-986a-84db482cee42"}]: dispatch 2023-07-14T19:12:50.993 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:12:50 smithi130 ceph-mon[97832]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7770b6d6-5874-493a-986a-84db482cee42"}]': finished 2023-07-14T19:12:50.993 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:12:50 smithi130 ceph-mon[97832]: osdmap e34: 6 total, 5 up, 6 in 2023-07-14T19:12:50.993 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:12:50 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-07-14T19:12:51.971 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:12:51 smithi040 ceph-mon[96518]: from='client.? 172.21.15.191:0/3288473131' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-14T19:12:51.972 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:12:51 smithi191 ceph-mon[98311]: from='client.? 172.21.15.191:0/3288473131' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-14T19:12:51.992 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:12:51 smithi130 ceph-mon[97832]: from='client.? 172.21.15.191:0/3288473131' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-14T19:12:52.971 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:12:52 smithi040 ceph-mon[96518]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 133 MiB used, 447 GiB / 447 GiB avail 2023-07-14T19:12:52.972 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:12:52 smithi191 ceph-mon[98311]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 133 MiB used, 447 GiB / 447 GiB avail 2023-07-14T19:12:52.992 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:12:52 smithi130 ceph-mon[97832]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 133 MiB used, 447 GiB / 447 GiB avail 2023-07-14T19:12:54.722 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:12:54 smithi191 ceph-mon[98311]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 133 MiB used, 447 GiB / 447 GiB avail 2023-07-14T19:12:54.971 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:12:54 smithi040 ceph-mon[96518]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 133 MiB used, 447 GiB / 447 GiB avail 2023-07-14T19:12:54.992 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:12:54 smithi130 ceph-mon[97832]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 133 MiB used, 447 GiB / 447 GiB avail 2023-07-14T19:12:56.907 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:12:56 smithi191 ceph-mon[98311]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 133 MiB used, 447 GiB / 447 GiB avail 2023-07-14T19:12:56.971 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:12:56 smithi040 ceph-mon[96518]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 133 MiB used, 447 GiB / 447 GiB avail 2023-07-14T19:12:56.992 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:12:56 smithi130 ceph-mon[97832]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 133 MiB used, 447 GiB / 447 GiB avail 2023-07-14T19:12:57.798 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:12:57 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2023-07-14T19:12:57.798 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:12:57 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:12:57.971 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:12:57 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2023-07-14T19:12:57.971 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:12:57 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:12:57.992 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:12:57 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2023-07-14T19:12:57.993 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:12:57 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:12:58.882 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:12:58 smithi191 ceph-mon[98311]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 133 MiB used, 447 GiB / 447 GiB avail 2023-07-14T19:12:58.882 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:12:58 smithi191 ceph-mon[98311]: Deploying daemon osd.5 on smithi191 2023-07-14T19:12:58.971 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:12:58 smithi040 ceph-mon[96518]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 133 MiB used, 447 GiB / 447 GiB avail 2023-07-14T19:12:58.971 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:12:58 smithi040 ceph-mon[96518]: Deploying daemon osd.5 on smithi191 2023-07-14T19:12:58.992 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:12:58 smithi130 ceph-mon[97832]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 133 MiB used, 447 GiB / 447 GiB avail 2023-07-14T19:12:58.992 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:12:58 smithi130 ceph-mon[97832]: Deploying daemon osd.5 on smithi191 2023-07-14T19:13:00.971 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:13:00 smithi040 ceph-mon[96518]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 133 MiB used, 447 GiB / 447 GiB avail 2023-07-14T19:13:00.972 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:13:00 smithi191 ceph-mon[98311]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 133 MiB used, 447 GiB / 447 GiB avail 2023-07-14T19:13:00.992 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:13:00 smithi130 ceph-mon[97832]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 133 MiB used, 447 GiB / 447 GiB avail 2023-07-14T19:13:02.719 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:13:02 smithi191 ceph-mon[98311]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 133 MiB used, 447 GiB / 447 GiB avail 2023-07-14T19:13:02.719 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:13:02 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:13:02.719 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:13:02 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:13:02.719 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:13:02 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T19:13:02.971 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:13:02 smithi040 ceph-mon[96518]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 133 MiB used, 447 GiB / 447 GiB avail 2023-07-14T19:13:02.972 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:13:02 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:13:02.972 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:13:02 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:13:02.972 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:13:02 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T19:13:02.992 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:13:02 smithi130 ceph-mon[97832]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 133 MiB used, 447 GiB / 447 GiB avail 2023-07-14T19:13:02.992 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:13:02 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:13:02.993 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:13:02 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:13:02.993 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:13:02 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T19:13:04.409 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:13:04.408+0000 7f20d4ff9700 1 -- 172.21.15.191:0/2020083770 <== mgr.14152 v2:172.21.15.40:6800/1693027118 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f20d80fe430 con 0x7f20c00a1150 2023-07-14T19:13:04.410 INFO:teuthology.orchestra.run.smithi191.stdout:Created osd(s) 5 on host 'smithi191' 2023-07-14T19:13:04.411 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:13:04.410+0000 7f20ddb28700 1 -- 172.21.15.191:0/2020083770 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f20c00a1150 msgr2=0x7f20c00a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:13:04.411 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:13:04.410+0000 7f20ddb28700 1 --2- 172.21.15.191:0/2020083770 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f20c00a1150 0x7f20c00a3600 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f20d8067bd0 tx=0x7f20c8009260 comp rx=0 tx=0).stop 2023-07-14T19:13:04.412 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:13:04.410+0000 7f20ddb28700 1 -- 172.21.15.191:0/2020083770 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f20d80ff7a0 msgr2=0x7f20d81011e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:13:04.412 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:13:04.410+0000 7f20ddb28700 1 --2- 172.21.15.191:0/2020083770 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f20d80ff7a0 0x7f20d81011e0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f20cc00d980 tx=0x7f20cc00dc90 comp rx=0 tx=0).stop 2023-07-14T19:13:04.412 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:13:04.410+0000 7f20ddb28700 1 -- 172.21.15.191:0/2020083770 shutdown_connections 2023-07-14T19:13:04.412 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:13:04.410+0000 7f20ddb28700 1 --2- 172.21.15.191:0/2020083770 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f20c00a1150 0x7f20c00a3600 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:13:04.412 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:13:04.410+0000 7f20ddb28700 1 --2- 172.21.15.191:0/2020083770 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f20d8100190 0x7f20d8101720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:13:04.412 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:13:04.410+0000 7f20ddb28700 1 --2- 172.21.15.191:0/2020083770 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f20d80fedf0 0x7f20d8102b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:13:04.412 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:13:04.410+0000 7f20ddb28700 1 --2- 172.21.15.191:0/2020083770 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f20d80ff7a0 0x7f20d81011e0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:13:04.413 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:13:04.410+0000 7f20ddb28700 1 -- 172.21.15.191:0/2020083770 >> 172.21.15.191:0/2020083770 conn(0x7f20d80f9990 msgr2=0x7f20d80fdd20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:13:04.413 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:13:04.410+0000 7f20ddb28700 1 -- 172.21.15.191:0/2020083770 shutdown_connections 2023-07-14T19:13:04.413 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:13:04.410+0000 7f20ddb28700 1 -- 172.21.15.191:0/2020083770 wait complete. 2023-07-14T19:13:04.722 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:13:04 smithi191 ceph-mon[98311]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 133 MiB used, 447 GiB / 447 GiB avail 2023-07-14T19:13:04.722 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:13:04 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:13:04.722 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:13:04 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:13:04.723 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:13:04 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:13:04.723 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:13:04 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T19:13:04.723 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:13:04 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:13:04.723 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:13:04 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:13:04.723 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:13:04 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:13:04.723 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:13:04 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T19:13:04.862 DEBUG:teuthology.orchestra.run.smithi191:osd.5> sudo journalctl -f -n 0 -u ceph-adc52f04-2279-11ee-9b33-001a4aab830c@osd.5.service 2023-07-14T19:13:04.864 INFO:tasks.cephadm:Deploying osd.6 on smithi040 with /dev/vg_nvme/lv_2... 2023-07-14T19:13:04.864 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid adc52f04-2279-11ee-9b33-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2023-07-14T19:13:04.895 INFO:journalctl@ceph.osd.5.smithi191.stdout:-- Logs begin at Fri 2023-07-14 18:50:54 UTC. -- 2023-07-14T19:13:04.912 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:13:04 smithi040 ceph-mon[96518]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 133 MiB used, 447 GiB / 447 GiB avail 2023-07-14T19:13:04.912 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:13:04 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:13:04.912 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:13:04 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:13:04.913 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:13:04 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:13:04.913 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:13:04 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T19:13:04.913 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:13:04 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:13:04.913 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:13:04 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:13:04.913 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:13:04 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:13:04.913 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:13:04 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T19:13:04.992 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:13:04 smithi130 ceph-mon[97832]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 133 MiB used, 447 GiB / 447 GiB avail 2023-07-14T19:13:04.992 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:13:04 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:13:04.993 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:13:04 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:13:04.993 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:13:04 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:13:04.993 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:13:04 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T19:13:04.993 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:13:04 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:13:04.993 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:13:04 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:13:04.993 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:13:04 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:13:04.994 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:13:04 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T19:13:05.452 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/adc52f04-2279-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T19:13:05.643 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:13:05 smithi191 ceph-mon[98311]: from='osd.5 [v2:172.21.15.191:6808/1781944663,v1:172.21.15.191:6809/1781944663]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2023-07-14T19:13:05.644 INFO:journalctl@ceph.osd.5.smithi191.stdout:Jul 14 19:13:05 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-5[109616]: 2023-07-14T19:13:05.491+0000 7fa8b38e7640 -1 osd.5 0 log_to_monitors true 2023-07-14T19:13:05.899 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:13:05 smithi040 ceph-mon[96518]: from='osd.5 [v2:172.21.15.191:6808/1781944663,v1:172.21.15.191:6809/1781944663]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2023-07-14T19:13:05.992 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:13:05 smithi130 ceph-mon[97832]: from='osd.5 [v2:172.21.15.191:6808/1781944663,v1:172.21.15.191:6809/1781944663]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2023-07-14T19:13:06.971 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:13:06 smithi040 ceph-mon[96518]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 133 MiB used, 447 GiB / 447 GiB avail 2023-07-14T19:13:06.972 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:13:06 smithi040 ceph-mon[96518]: from='osd.5 [v2:172.21.15.191:6808/1781944663,v1:172.21.15.191:6809/1781944663]' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2023-07-14T19:13:06.972 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:13:06 smithi040 ceph-mon[96518]: osdmap e35: 6 total, 5 up, 6 in 2023-07-14T19:13:06.972 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:13:06 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-07-14T19:13:06.972 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:13:06 smithi040 ceph-mon[96518]: from='osd.5 [v2:172.21.15.191:6808/1781944663,v1:172.21.15.191:6809/1781944663]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi191", "root=default"]}]: dispatch 2023-07-14T19:13:06.973 INFO:journalctl@ceph.osd.5.smithi191.stdout:Jul 14 19:13:06 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-5[109616]: 2023-07-14T19:13:06.577+0000 7fa8a8f5f700 -1 osd.5 0 waiting for initial osdmap 2023-07-14T19:13:06.973 INFO:journalctl@ceph.osd.5.smithi191.stdout:Jul 14 19:13:06 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-5[109616]: 2023-07-14T19:13:06.584+0000 7fa8a3d53700 -1 osd.5 36 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-07-14T19:13:06.973 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:13:06 smithi191 ceph-mon[98311]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 133 MiB used, 447 GiB / 447 GiB avail 2023-07-14T19:13:06.973 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:13:06 smithi191 ceph-mon[98311]: from='osd.5 [v2:172.21.15.191:6808/1781944663,v1:172.21.15.191:6809/1781944663]' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2023-07-14T19:13:06.974 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:13:06 smithi191 ceph-mon[98311]: osdmap e35: 6 total, 5 up, 6 in 2023-07-14T19:13:06.974 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:13:06 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-07-14T19:13:06.974 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:13:06 smithi191 ceph-mon[98311]: from='osd.5 [v2:172.21.15.191:6808/1781944663,v1:172.21.15.191:6809/1781944663]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi191", "root=default"]}]: dispatch 2023-07-14T19:13:06.992 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:13:06 smithi130 ceph-mon[97832]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 133 MiB used, 447 GiB / 447 GiB avail 2023-07-14T19:13:06.993 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:13:06 smithi130 ceph-mon[97832]: from='osd.5 [v2:172.21.15.191:6808/1781944663,v1:172.21.15.191:6809/1781944663]' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2023-07-14T19:13:06.993 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:13:06 smithi130 ceph-mon[97832]: osdmap e35: 6 total, 5 up, 6 in 2023-07-14T19:13:06.993 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:13:06 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-07-14T19:13:06.993 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:13:06 smithi130 ceph-mon[97832]: from='osd.5 [v2:172.21.15.191:6808/1781944663,v1:172.21.15.191:6809/1781944663]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi191", "root=default"]}]: dispatch 2023-07-14T19:13:07.095 INFO:teuthology.orchestra.run.smithi040.stderr:--> Zapping: /dev/vg_nvme/lv_2 2023-07-14T19:13:07.096 INFO:teuthology.orchestra.run.smithi040.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2023-07-14T19:13:07.096 INFO:teuthology.orchestra.run.smithi040.stderr: stderr: 10+0 records in 2023-07-14T19:13:07.096 INFO:teuthology.orchestra.run.smithi040.stderr:10+0 records out 2023-07-14T19:13:07.096 INFO:teuthology.orchestra.run.smithi040.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0107646 s, 974 MB/s 2023-07-14T19:13:07.096 INFO:teuthology.orchestra.run.smithi040.stderr:--> Zapping successful for: 2023-07-14T19:13:07.513 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid adc52f04-2279-11ee-9b33-001a4aab830c -- ceph orch daemon add osd smithi040:vg_nvme/lv_2 2023-07-14T19:13:07.971 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:13:07 smithi040 ceph-mon[96518]: from='osd.5 [v2:172.21.15.191:6808/1781944663,v1:172.21.15.191:6809/1781944663]' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi191", "root=default"]}]': finished 2023-07-14T19:13:07.971 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:13:07 smithi040 ceph-mon[96518]: osdmap e36: 6 total, 5 up, 6 in 2023-07-14T19:13:07.972 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:13:07 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-07-14T19:13:07.972 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:13:07 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-07-14T19:13:07.972 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:13:07 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:13:07.972 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:13:07 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:13:07.972 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:13:07 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-07-14T19:13:07.972 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:13:07 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-07-14T19:13:07.973 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:13:07 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:13:07.973 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:13:07 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:13:07.973 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:13:07 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T19:13:07.973 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:13:07 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:13:07.973 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:13:07 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-07-14T19:13:07.974 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:13:07 smithi191 ceph-mon[98311]: from='osd.5 [v2:172.21.15.191:6808/1781944663,v1:172.21.15.191:6809/1781944663]' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi191", "root=default"]}]': finished 2023-07-14T19:13:07.974 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:13:07 smithi191 ceph-mon[98311]: osdmap e36: 6 total, 5 up, 6 in 2023-07-14T19:13:07.974 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:13:07 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-07-14T19:13:07.974 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:13:07 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-07-14T19:13:07.974 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:13:07 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:13:07.974 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:13:07 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:13:07.975 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:13:07 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-07-14T19:13:07.975 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:13:07 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-07-14T19:13:07.975 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:13:07 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:13:07.975 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:13:07 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:13:07.975 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:13:07 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T19:13:07.975 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:13:07 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:13:07.975 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:13:07 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-07-14T19:13:07.992 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:13:07 smithi130 ceph-mon[97832]: from='osd.5 [v2:172.21.15.191:6808/1781944663,v1:172.21.15.191:6809/1781944663]' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi191", "root=default"]}]': finished 2023-07-14T19:13:07.993 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:13:07 smithi130 ceph-mon[97832]: osdmap e36: 6 total, 5 up, 6 in 2023-07-14T19:13:07.993 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:13:07 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-07-14T19:13:07.993 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:13:07 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-07-14T19:13:07.993 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:13:07 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:13:07.993 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:13:07 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:13:07.993 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:13:07 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-07-14T19:13:07.993 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:13:07 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-07-14T19:13:07.994 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:13:07 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:13:07.994 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:13:07 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:13:07.994 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:13:07 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T19:13:07.994 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:13:07 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:13:07.994 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:13:07 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-07-14T19:13:08.077 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/adc52f04-2279-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T19:13:08.971 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:13:08 smithi040 ceph-mon[96518]: purged_snaps scrub starts 2023-07-14T19:13:08.972 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:13:08 smithi040 ceph-mon[96518]: purged_snaps scrub ok 2023-07-14T19:13:08.972 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:13:08 smithi040 ceph-mon[96518]: pgmap v124: 1 pgs: 1 peering; 577 KiB data, 134 MiB used, 447 GiB / 447 GiB avail 2023-07-14T19:13:08.972 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:13:08 smithi040 ceph-mon[96518]: Detected new or changed devices on smithi191 2023-07-14T19:13:08.972 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:13:08 smithi040 ceph-mon[96518]: Adjusting osd_memory_target on smithi191 to 8584M 2023-07-14T19:13:08.972 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:13:08 smithi040 ceph-mon[96518]: osd.5 [v2:172.21.15.191:6808/1781944663,v1:172.21.15.191:6809/1781944663] boot 2023-07-14T19:13:08.973 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:13:08 smithi040 ceph-mon[96518]: osdmap e37: 6 total, 6 up, 6 in 2023-07-14T19:13:08.973 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:13:08 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-07-14T19:13:08.973 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:13:08 smithi191 ceph-mon[98311]: purged_snaps scrub starts 2023-07-14T19:13:08.973 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:13:08 smithi191 ceph-mon[98311]: purged_snaps scrub ok 2023-07-14T19:13:08.973 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:13:08 smithi191 ceph-mon[98311]: pgmap v124: 1 pgs: 1 peering; 577 KiB data, 134 MiB used, 447 GiB / 447 GiB avail 2023-07-14T19:13:08.974 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:13:08 smithi191 ceph-mon[98311]: Detected new or changed devices on smithi191 2023-07-14T19:13:08.974 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:13:08 smithi191 ceph-mon[98311]: Adjusting osd_memory_target on smithi191 to 8584M 2023-07-14T19:13:08.974 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:13:08 smithi191 ceph-mon[98311]: osd.5 [v2:172.21.15.191:6808/1781944663,v1:172.21.15.191:6809/1781944663] boot 2023-07-14T19:13:08.974 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:13:08 smithi191 ceph-mon[98311]: osdmap e37: 6 total, 6 up, 6 in 2023-07-14T19:13:08.974 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:13:08 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-07-14T19:13:08.992 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:13:08 smithi130 ceph-mon[97832]: purged_snaps scrub starts 2023-07-14T19:13:08.992 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:13:08 smithi130 ceph-mon[97832]: purged_snaps scrub ok 2023-07-14T19:13:08.993 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:13:08 smithi130 ceph-mon[97832]: pgmap v124: 1 pgs: 1 peering; 577 KiB data, 134 MiB used, 447 GiB / 447 GiB avail 2023-07-14T19:13:08.993 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:13:08 smithi130 ceph-mon[97832]: Detected new or changed devices on smithi191 2023-07-14T19:13:08.993 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:13:08 smithi130 ceph-mon[97832]: Adjusting osd_memory_target on smithi191 to 8584M 2023-07-14T19:13:08.993 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:13:08 smithi130 ceph-mon[97832]: osd.5 [v2:172.21.15.191:6808/1781944663,v1:172.21.15.191:6809/1781944663] boot 2023-07-14T19:13:08.993 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:13:08 smithi130 ceph-mon[97832]: osdmap e37: 6 total, 6 up, 6 in 2023-07-14T19:13:08.993 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:13:08 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-07-14T19:13:09.213 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:13:09.210+0000 7fe3a251f700 1 -- 172.21.15.40:0/3755662178 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe39c0ffdb0 msgr2=0x7fe39c100200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:13:09.213 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:13:09.210+0000 7fe3a251f700 1 --2- 172.21.15.40:0/3755662178 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe39c0ffdb0 0x7fe39c100200 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fe390009990 tx=0x7fe390009ca0 comp rx=0 tx=0).stop 2023-07-14T19:13:09.213 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:13:09.210+0000 7fe3a251f700 1 -- 172.21.15.40:0/3755662178 shutdown_connections 2023-07-14T19:13:09.213 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:13:09.210+0000 7fe3a251f700 1 --2- 172.21.15.40:0/3755662178 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe39c0febb0 0x7fe39c0fefc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:13:09.214 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:13:09.210+0000 7fe3a251f700 1 --2- 172.21.15.40:0/3755662178 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe39c100740 0x7fe39c107000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:13:09.214 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:13:09.210+0000 7fe3a251f700 1 --2- 172.21.15.40:0/3755662178 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe39c0ffdb0 0x7fe39c100200 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:13:09.214 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:13:09.210+0000 7fe3a251f700 1 -- 172.21.15.40:0/3755662178 >> 172.21.15.40:0/3755662178 conn(0x7fe39c0fa1e0 msgr2=0x7fe39c0fc630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:13:09.214 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:13:09.211+0000 7fe3a251f700 1 -- 172.21.15.40:0/3755662178 shutdown_connections 2023-07-14T19:13:09.214 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:13:09.211+0000 7fe3a251f700 1 -- 172.21.15.40:0/3755662178 wait complete. 2023-07-14T19:13:09.214 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:13:09.211+0000 7fe3a251f700 1 Processor -- start 2023-07-14T19:13:09.214 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:13:09.211+0000 7fe3a251f700 1 -- start start 2023-07-14T19:13:09.216 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:13:09.211+0000 7fe3a251f700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe39c0febb0 0x7fe39c195960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:13:09.216 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:13:09.211+0000 7fe3a251f700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe39c0ffdb0 0x7fe39c195ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:13:09.216 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:13:09.211+0000 7fe3a251f700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe39c100740 0x7fe39c19cf20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:13:09.216 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:13:09.211+0000 7fe3a251f700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe39c19d4f0 con 0x7fe39c0febb0 2023-07-14T19:13:09.216 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:13:09.211+0000 7fe3a251f700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe39c19d630 con 0x7fe39c100740 2023-07-14T19:13:09.217 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:13:09.211+0000 7fe3a251f700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe39c19d7c0 con 0x7fe39c0ffdb0 2023-07-14T19:13:09.217 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:13:09.212+0000 7fe39bfff700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe39c0febb0 0x7fe39c195960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:13:09.217 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:13:09.212+0000 7fe39bfff700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe39c0febb0 0x7fe39c195960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:60404/0 (socket says 172.21.15.40:60404) 2023-07-14T19:13:09.217 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:13:09.212+0000 7fe39bfff700 1 -- 172.21.15.40:0/1662480857 learned_addr learned my addr 172.21.15.40:0/1662480857 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-07-14T19:13:09.217 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:13:09.212+0000 7fe3a0abc700 1 --2- 172.21.15.40:0/1662480857 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe39c100740 0x7fe39c19cf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:13:09.217 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:13:09.212+0000 7fe39b7fe700 1 --2- 172.21.15.40:0/1662480857 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe39c0ffdb0 0x7fe39c195ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:13:09.217 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:13:09.212+0000 7fe39bfff700 1 -- 172.21.15.40:0/1662480857 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe39c0ffdb0 msgr2=0x7fe39c195ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:13:09.218 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:13:09.212+0000 7fe39bfff700 1 --2- 172.21.15.40:0/1662480857 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe39c0ffdb0 0x7fe39c195ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:13:09.218 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:13:09.212+0000 7fe39bfff700 1 -- 172.21.15.40:0/1662480857 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe39c100740 msgr2=0x7fe39c19cf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:13:09.218 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:13:09.212+0000 7fe39bfff700 1 --2- 172.21.15.40:0/1662480857 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe39c100740 0x7fe39c19cf20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:13:09.218 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:13:09.212+0000 7fe39bfff700 1 -- 172.21.15.40:0/1662480857 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3900095f0 con 0x7fe39c0febb0 2023-07-14T19:13:09.218 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:13:09.212+0000 7fe39bfff700 1 --2- 172.21.15.40:0/1662480857 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe39c0febb0 0x7fe39c195960 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fe38c00c9c0 tx=0x7fe38c00ccd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:13:09.218 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:13:09.212+0000 7fe3997fa700 1 -- 172.21.15.40:0/1662480857 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe38c007ac0 con 0x7fe39c0febb0 2023-07-14T19:13:09.218 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:13:09.212+0000 7fe3a251f700 1 -- 172.21.15.40:0/1662480857 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe39c19daa0 con 0x7fe39c0febb0 2023-07-14T19:13:09.218 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:13:09.212+0000 7fe3997fa700 1 -- 172.21.15.40:0/1662480857 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe38c00ce60 con 0x7fe39c0febb0 2023-07-14T19:13:09.219 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:13:09.212+0000 7fe3997fa700 1 -- 172.21.15.40:0/1662480857 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe38c018400 con 0x7fe39c0febb0 2023-07-14T19:13:09.219 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:13:09.212+0000 7fe3a251f700 1 -- 172.21.15.40:0/1662480857 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe39c19dfa0 con 0x7fe39c0febb0 2023-07-14T19:13:09.219 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:13:09.213+0000 7fe3a251f700 1 -- 172.21.15.40:0/1662480857 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe39c106ca0 con 0x7fe39c0febb0 2023-07-14T19:13:09.219 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:13:09.215+0000 7fe3997fa700 1 -- 172.21.15.40:0/1662480857 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7fe38c00f450 con 0x7fe39c0febb0 2023-07-14T19:13:09.220 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:13:09.215+0000 7fe3997fa700 1 --2- 172.21.15.40:0/1662480857 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7fe3840a1150 0x7fe3840a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:13:09.220 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:13:09.215+0000 7fe3997fa700 1 -- 172.21.15.40:0/1662480857 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 4562+0+0 (secure 0 0 0) 0x7fe38c0cba20 con 0x7fe39c0febb0 2023-07-14T19:13:09.220 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:13:09.215+0000 7fe39b7fe700 1 --2- 172.21.15.40:0/1662480857 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7fe3840a1150 0x7fe3840a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:13:09.220 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:13:09.216+0000 7fe39b7fe700 1 --2- 172.21.15.40:0/1662480857 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7fe3840a1150 0x7fe3840a3600 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fe39000b5c0 tx=0x7fe390005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:13:09.221 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:13:09.220+0000 7fe3997fa700 1 -- 172.21.15.40:0/1662480857 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fe38c0960a0 con 0x7fe39c0febb0 2023-07-14T19:13:09.426 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:13:09.424+0000 7fe3a251f700 1 -- 172.21.15.40:0/1662480857 --> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi040:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) v1 -- 0x7fe39c060a80 con 0x7fe3840a1150 2023-07-14T19:13:09.971 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:13:09 smithi040 ceph-mon[96518]: osdmap e38: 6 total, 6 up, 6 in 2023-07-14T19:13:09.972 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:13:09 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-14T19:13:09.972 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:13:09 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-14T19:13:09.972 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:13:09 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:13:09.972 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:13:09 smithi191 ceph-mon[98311]: osdmap e38: 6 total, 6 up, 6 in 2023-07-14T19:13:09.973 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:13:09 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-14T19:13:09.973 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:13:09 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-14T19:13:09.973 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:13:09 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:13:09.992 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:13:09 smithi130 ceph-mon[97832]: osdmap e38: 6 total, 6 up, 6 in 2023-07-14T19:13:09.993 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:13:09 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-14T19:13:09.993 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:13:09 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-14T19:13:09.993 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:13:09 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:13:10.971 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:13:10 smithi040 ceph-mon[96518]: pgmap v127: 1 pgs: 1 peering; 577 KiB data, 160 MiB used, 536 GiB / 536 GiB avail 2023-07-14T19:13:10.972 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:13:10 smithi040 ceph-mon[96518]: from='client.14358 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi040:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T19:13:10.972 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:13:10 smithi040 ceph-mon[96518]: osdmap e39: 6 total, 6 up, 6 in 2023-07-14T19:13:10.972 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:13:10 smithi191 ceph-mon[98311]: pgmap v127: 1 pgs: 1 peering; 577 KiB data, 160 MiB used, 536 GiB / 536 GiB avail 2023-07-14T19:13:10.972 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:13:10 smithi191 ceph-mon[98311]: from='client.14358 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi040:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T19:13:10.973 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:13:10 smithi191 ceph-mon[98311]: osdmap e39: 6 total, 6 up, 6 in 2023-07-14T19:13:10.992 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:13:10 smithi130 ceph-mon[97832]: pgmap v127: 1 pgs: 1 peering; 577 KiB data, 160 MiB used, 536 GiB / 536 GiB avail 2023-07-14T19:13:10.993 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:13:10 smithi130 ceph-mon[97832]: from='client.14358 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi040:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T19:13:10.993 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:13:10 smithi130 ceph-mon[97832]: osdmap e39: 6 total, 6 up, 6 in 2023-07-14T19:13:11.972 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:13:11 smithi040 ceph-mon[96518]: osdmap e40: 6 total, 6 up, 6 in 2023-07-14T19:13:11.972 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:13:11 smithi191 ceph-mon[98311]: osdmap e40: 6 total, 6 up, 6 in 2023-07-14T19:13:11.992 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:13:11 smithi130 ceph-mon[97832]: osdmap e40: 6 total, 6 up, 6 in 2023-07-14T19:13:12.971 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:13:12 smithi040 ceph-mon[96518]: pgmap v130: 1 pgs: 1 peering; 577 KiB data, 160 MiB used, 536 GiB / 536 GiB avail 2023-07-14T19:13:12.972 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:13:12 smithi040 ceph-mon[96518]: osdmap e41: 6 total, 6 up, 6 in 2023-07-14T19:13:12.972 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:13:12 smithi040 ceph-mon[96518]: from='client.? 172.21.15.40:0/1574041033' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ef65ad00-dfee-4300-9961-dfc8573aab06"}]: dispatch 2023-07-14T19:13:12.972 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:13:12 smithi040 ceph-mon[96518]: from='client.? 172.21.15.40:0/1574041033' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ef65ad00-dfee-4300-9961-dfc8573aab06"}]': finished 2023-07-14T19:13:12.972 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:13:12 smithi040 ceph-mon[96518]: osdmap e42: 7 total, 6 up, 7 in 2023-07-14T19:13:12.972 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:13:12 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-07-14T19:13:12.972 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:13:12 smithi040 ceph-mon[96518]: from='client.? 172.21.15.40:0/311669039' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-14T19:13:12.973 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:13:12 smithi191 ceph-mon[98311]: pgmap v130: 1 pgs: 1 peering; 577 KiB data, 160 MiB used, 536 GiB / 536 GiB avail 2023-07-14T19:13:12.973 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:13:12 smithi191 ceph-mon[98311]: osdmap e41: 6 total, 6 up, 6 in 2023-07-14T19:13:12.973 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:13:12 smithi191 ceph-mon[98311]: from='client.? 172.21.15.40:0/1574041033' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ef65ad00-dfee-4300-9961-dfc8573aab06"}]: dispatch 2023-07-14T19:13:12.973 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:13:12 smithi191 ceph-mon[98311]: from='client.? 172.21.15.40:0/1574041033' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ef65ad00-dfee-4300-9961-dfc8573aab06"}]': finished 2023-07-14T19:13:12.974 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:13:12 smithi191 ceph-mon[98311]: osdmap e42: 7 total, 6 up, 7 in 2023-07-14T19:13:12.974 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:13:12 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-07-14T19:13:12.974 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:13:12 smithi191 ceph-mon[98311]: from='client.? 172.21.15.40:0/311669039' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-14T19:13:12.992 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:13:12 smithi130 ceph-mon[97832]: pgmap v130: 1 pgs: 1 peering; 577 KiB data, 160 MiB used, 536 GiB / 536 GiB avail 2023-07-14T19:13:12.992 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:13:12 smithi130 ceph-mon[97832]: osdmap e41: 6 total, 6 up, 6 in 2023-07-14T19:13:12.993 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:13:12 smithi130 ceph-mon[97832]: from='client.? 172.21.15.40:0/1574041033' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ef65ad00-dfee-4300-9961-dfc8573aab06"}]: dispatch 2023-07-14T19:13:12.993 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:13:12 smithi130 ceph-mon[97832]: from='client.? 172.21.15.40:0/1574041033' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ef65ad00-dfee-4300-9961-dfc8573aab06"}]': finished 2023-07-14T19:13:12.993 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:13:12 smithi130 ceph-mon[97832]: osdmap e42: 7 total, 6 up, 7 in 2023-07-14T19:13:12.993 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:13:12 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-07-14T19:13:12.993 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:13:12 smithi130 ceph-mon[97832]: from='client.? 172.21.15.40:0/311669039' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-14T19:13:14.971 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:13:14 smithi040 ceph-mon[96518]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail; 144 KiB/s, 0 objects/s recovering 2023-07-14T19:13:14.972 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:13:14 smithi191 ceph-mon[98311]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail; 144 KiB/s, 0 objects/s recovering 2023-07-14T19:13:14.992 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:13:14 smithi130 ceph-mon[97832]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail; 144 KiB/s, 0 objects/s recovering 2023-07-14T19:13:16.875 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:13:16 smithi040 ceph-mon[96518]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail; 111 KiB/s, 0 objects/s recovering 2023-07-14T19:13:16.972 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:13:16 smithi191 ceph-mon[98311]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail; 111 KiB/s, 0 objects/s recovering 2023-07-14T19:13:16.992 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:13:16 smithi130 ceph-mon[97832]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail; 111 KiB/s, 0 objects/s recovering 2023-07-14T19:13:18.971 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:13:18 smithi040 ceph-mon[96518]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail; 93 KiB/s, 0 objects/s recovering 2023-07-14T19:13:18.971 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:13:18 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2023-07-14T19:13:18.971 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:13:18 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:13:18.972 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:13:18 smithi191 ceph-mon[98311]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail; 93 KiB/s, 0 objects/s recovering 2023-07-14T19:13:18.972 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:13:18 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2023-07-14T19:13:18.972 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:13:18 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:13:18.992 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:13:18 smithi130 ceph-mon[97832]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail; 93 KiB/s, 0 objects/s recovering 2023-07-14T19:13:18.993 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:13:18 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2023-07-14T19:13:18.993 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:13:18 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:13:19.971 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:13:19 smithi040 ceph-mon[96518]: Deploying daemon osd.6 on smithi040 2023-07-14T19:13:19.972 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:13:19 smithi191 ceph-mon[98311]: Deploying daemon osd.6 on smithi040 2023-07-14T19:13:19.992 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:13:19 smithi130 ceph-mon[97832]: Deploying daemon osd.6 on smithi040 2023-07-14T19:13:20.895 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:13:20 smithi040 ceph-mon[96518]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail; 72 KiB/s, 0 objects/s recovering 2023-07-14T19:13:20.972 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:13:20 smithi191 ceph-mon[98311]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail; 72 KiB/s, 0 objects/s recovering 2023-07-14T19:13:20.992 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:13:20 smithi130 ceph-mon[97832]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail; 72 KiB/s, 0 objects/s recovering 2023-07-14T19:13:22.721 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:13:22 smithi040 ceph-mon[96518]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail; 63 KiB/s, 0 objects/s recovering 2023-07-14T19:13:22.972 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:13:22 smithi191 ceph-mon[98311]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail; 63 KiB/s, 0 objects/s recovering 2023-07-14T19:13:22.992 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:13:22 smithi130 ceph-mon[97832]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail; 63 KiB/s, 0 objects/s recovering 2023-07-14T19:13:24.721 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:13:24 smithi040 ceph-mon[96518]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail 2023-07-14T19:13:24.722 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:13:24 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:13:24.722 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:13:24 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:13:24.722 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:13:24 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T19:13:24.742 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:13:24 smithi130 ceph-mon[97832]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail 2023-07-14T19:13:24.743 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:13:24 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:13:24.743 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:13:24 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:13:24.743 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:13:24 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T19:13:24.972 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:13:24 smithi191 ceph-mon[98311]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail 2023-07-14T19:13:24.972 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:13:24 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:13:24.973 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:13:24 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:13:24.973 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:13:24 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T19:13:26.585 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:13:26.582+0000 7fe3997fa700 1 -- 172.21.15.40:0/1662480857 <== mgr.14152 v2:172.21.15.40:6800/1693027118 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7fe39c060a80 con 0x7fe3840a1150 2023-07-14T19:13:26.585 INFO:teuthology.orchestra.run.smithi040.stdout:Created osd(s) 6 on host 'smithi040' 2023-07-14T19:13:26.587 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:13:26.585+0000 7fe3a251f700 1 -- 172.21.15.40:0/1662480857 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7fe3840a1150 msgr2=0x7fe3840a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:13:26.587 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:13:26.585+0000 7fe3a251f700 1 --2- 172.21.15.40:0/1662480857 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7fe3840a1150 0x7fe3840a3600 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fe39000b5c0 tx=0x7fe390005fd0 comp rx=0 tx=0).stop 2023-07-14T19:13:26.587 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:13:26.585+0000 7fe3a251f700 1 -- 172.21.15.40:0/1662480857 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe39c0febb0 msgr2=0x7fe39c195960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:13:26.588 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:13:26.585+0000 7fe3a251f700 1 --2- 172.21.15.40:0/1662480857 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe39c0febb0 0x7fe39c195960 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fe38c00c9c0 tx=0x7fe38c00ccd0 comp rx=0 tx=0).stop 2023-07-14T19:13:26.588 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:13:26.585+0000 7fe3a251f700 1 -- 172.21.15.40:0/1662480857 shutdown_connections 2023-07-14T19:13:26.588 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:13:26.585+0000 7fe3a251f700 1 --2- 172.21.15.40:0/1662480857 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7fe3840a1150 0x7fe3840a3600 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:13:26.588 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:13:26.585+0000 7fe3a251f700 1 --2- 172.21.15.40:0/1662480857 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe39c100740 0x7fe39c19cf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:13:26.588 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:13:26.585+0000 7fe3a251f700 1 --2- 172.21.15.40:0/1662480857 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe39c0febb0 0x7fe39c195960 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:13:26.588 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:13:26.585+0000 7fe3a251f700 1 --2- 172.21.15.40:0/1662480857 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe39c0ffdb0 0x7fe39c195ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:13:26.589 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:13:26.585+0000 7fe3a251f700 1 -- 172.21.15.40:0/1662480857 >> 172.21.15.40:0/1662480857 conn(0x7fe39c0fa1e0 msgr2=0x7fe39c0fc3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:13:26.589 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:13:26.585+0000 7fe3a251f700 1 -- 172.21.15.40:0/1662480857 shutdown_connections 2023-07-14T19:13:26.589 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:13:26.585+0000 7fe3a251f700 1 -- 172.21.15.40:0/1662480857 wait complete. 2023-07-14T19:13:26.721 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:13:26 smithi040 ceph-mon[96518]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail 2023-07-14T19:13:26.721 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:13:26 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:13:26.722 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:13:26 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:13:26.722 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:13:26 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:13:26.722 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:13:26 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T19:13:26.722 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:13:26 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:13:26.742 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:13:26 smithi130 ceph-mon[97832]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail 2023-07-14T19:13:26.742 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:13:26 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:13:26.743 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:13:26 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:13:26.743 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:13:26 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:13:26.743 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:13:26 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T19:13:26.743 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:13:26 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:13:26.972 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:13:26 smithi191 ceph-mon[98311]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail 2023-07-14T19:13:26.972 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:13:26 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:13:26.972 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:13:26 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:13:26.973 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:13:26 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:13:26.973 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:13:26 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T19:13:26.973 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:13:26 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:13:27.169 DEBUG:teuthology.orchestra.run.smithi040:osd.6> sudo journalctl -f -n 0 -u ceph-adc52f04-2279-11ee-9b33-001a4aab830c@osd.6.service 2023-07-14T19:13:27.171 INFO:tasks.cephadm:Deploying osd.7 on smithi130 with /dev/vg_nvme/lv_2... 2023-07-14T19:13:27.171 DEBUG:teuthology.orchestra.run.smithi130:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid adc52f04-2279-11ee-9b33-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2023-07-14T19:13:27.200 INFO:journalctl@ceph.osd.6.smithi040.stdout:-- Logs begin at Fri 2023-07-14 18:50:26 UTC. -- 2023-07-14T19:13:27.673 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:13:27 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:13:27.673 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:13:27 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:13:27.674 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:13:27 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T19:13:27.674 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:13:27 smithi040 ceph-mon[96518]: from='osd.6 [v2:172.21.15.40:6818/693186773,v1:172.21.15.40:6819/693186773]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2023-07-14T19:13:27.674 INFO:journalctl@ceph.osd.6.smithi040.stdout:Jul 14 19:13:27 smithi040 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-6[120375]: 2023-07-14T19:13:27.416+0000 7f5d0ba67640 -1 osd.6 0 log_to_monitors true 2023-07-14T19:13:27.734 INFO:teuthology.orchestra.run.smithi130.stderr:Inferring config /var/lib/ceph/adc52f04-2279-11ee-9b33-001a4aab830c/mon.b/config 2023-07-14T19:13:27.950 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:13:27 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:13:27.950 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:13:27 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:13:27.951 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:13:27 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T19:13:27.951 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:13:27 smithi130 ceph-mon[97832]: from='osd.6 [v2:172.21.15.40:6818/693186773,v1:172.21.15.40:6819/693186773]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2023-07-14T19:13:27.972 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:13:27 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:13:27.972 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:13:27 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:13:27.972 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:13:27 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T19:13:27.972 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:13:27 smithi191 ceph-mon[98311]: from='osd.6 [v2:172.21.15.40:6818/693186773,v1:172.21.15.40:6819/693186773]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2023-07-14T19:13:28.566 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:13:28 smithi040 ceph-mon[96518]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail 2023-07-14T19:13:28.566 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:13:28 smithi040 ceph-mon[96518]: from='osd.6 [v2:172.21.15.40:6818/693186773,v1:172.21.15.40:6819/693186773]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2023-07-14T19:13:28.566 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:13:28 smithi040 ceph-mon[96518]: osdmap e43: 7 total, 6 up, 7 in 2023-07-14T19:13:28.567 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:13:28 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-07-14T19:13:28.567 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:13:28 smithi040 ceph-mon[96518]: from='osd.6 [v2:172.21.15.40:6818/693186773,v1:172.21.15.40:6819/693186773]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi040", "root=default"]}]: dispatch 2023-07-14T19:13:28.851 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:13:28 smithi130 ceph-mon[97832]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail 2023-07-14T19:13:28.851 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:13:28 smithi130 ceph-mon[97832]: from='osd.6 [v2:172.21.15.40:6818/693186773,v1:172.21.15.40:6819/693186773]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2023-07-14T19:13:28.851 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:13:28 smithi130 ceph-mon[97832]: osdmap e43: 7 total, 6 up, 7 in 2023-07-14T19:13:28.852 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:13:28 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-07-14T19:13:28.852 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:13:28 smithi130 ceph-mon[97832]: from='osd.6 [v2:172.21.15.40:6818/693186773,v1:172.21.15.40:6819/693186773]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi040", "root=default"]}]: dispatch 2023-07-14T19:13:28.884 INFO:journalctl@ceph.osd.6.smithi040.stdout:Jul 14 19:13:28 smithi040 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-6[120375]: 2023-07-14T19:13:28.568+0000 7f5d010df700 -1 osd.6 0 waiting for initial osdmap 2023-07-14T19:13:28.884 INFO:journalctl@ceph.osd.6.smithi040.stdout:Jul 14 19:13:28 smithi040 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-6[120375]: 2023-07-14T19:13:28.576+0000 7f5cfbed3700 -1 osd.6 44 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-07-14T19:13:28.972 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:13:28 smithi191 ceph-mon[98311]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail 2023-07-14T19:13:28.972 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:13:28 smithi191 ceph-mon[98311]: from='osd.6 [v2:172.21.15.40:6818/693186773,v1:172.21.15.40:6819/693186773]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2023-07-14T19:13:28.972 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:13:28 smithi191 ceph-mon[98311]: osdmap e43: 7 total, 6 up, 7 in 2023-07-14T19:13:28.972 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:13:28 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-07-14T19:13:28.973 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:13:28 smithi191 ceph-mon[98311]: from='osd.6 [v2:172.21.15.40:6818/693186773,v1:172.21.15.40:6819/693186773]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi040", "root=default"]}]: dispatch 2023-07-14T19:13:29.288 INFO:teuthology.orchestra.run.smithi130.stderr:--> Zapping: /dev/vg_nvme/lv_2 2023-07-14T19:13:29.288 INFO:teuthology.orchestra.run.smithi130.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2023-07-14T19:13:29.288 INFO:teuthology.orchestra.run.smithi130.stderr: stderr: 10+0 records in 2023-07-14T19:13:29.288 INFO:teuthology.orchestra.run.smithi130.stderr:10+0 records out 2023-07-14T19:13:29.289 INFO:teuthology.orchestra.run.smithi130.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0108785 s, 964 MB/s 2023-07-14T19:13:29.289 INFO:teuthology.orchestra.run.smithi130.stderr:--> Zapping successful for: 2023-07-14T19:13:29.651 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:13:29 smithi130 ceph-mon[97832]: from='osd.6 [v2:172.21.15.40:6818/693186773,v1:172.21.15.40:6819/693186773]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi040", "root=default"]}]': finished 2023-07-14T19:13:29.651 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:13:29 smithi130 ceph-mon[97832]: osdmap e44: 7 total, 6 up, 7 in 2023-07-14T19:13:29.651 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:13:29 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-07-14T19:13:29.651 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:13:29 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-07-14T19:13:29.651 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:13:29 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:13:29.652 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:13:29 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:13:29.652 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:13:29 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-07-14T19:13:29.652 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:13:29 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-07-14T19:13:29.652 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:13:29 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-07-14T19:13:29.652 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:13:29 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:13:29.652 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:13:29 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:13:29.652 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:13:29 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T19:13:29.653 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:13:29 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:13:29.673 DEBUG:teuthology.orchestra.run.smithi130:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid adc52f04-2279-11ee-9b33-001a4aab830c -- ceph orch daemon add osd smithi130:vg_nvme/lv_2 2023-07-14T19:13:29.971 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:13:29 smithi040 ceph-mon[96518]: from='osd.6 [v2:172.21.15.40:6818/693186773,v1:172.21.15.40:6819/693186773]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi040", "root=default"]}]': finished 2023-07-14T19:13:29.972 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:13:29 smithi040 ceph-mon[96518]: osdmap e44: 7 total, 6 up, 7 in 2023-07-14T19:13:29.972 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:13:29 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-07-14T19:13:29.972 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:13:29 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-07-14T19:13:29.972 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:13:29 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:13:29.972 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:13:29 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:13:29.973 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:13:29 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-07-14T19:13:29.973 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:13:29 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-07-14T19:13:29.973 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:13:29 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-07-14T19:13:29.973 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:13:29 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:13:29.973 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:13:29 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:13:29.973 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:13:29 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T19:13:29.973 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:13:29 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:13:29.974 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:13:29 smithi191 ceph-mon[98311]: from='osd.6 [v2:172.21.15.40:6818/693186773,v1:172.21.15.40:6819/693186773]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi040", "root=default"]}]': finished 2023-07-14T19:13:29.974 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:13:29 smithi191 ceph-mon[98311]: osdmap e44: 7 total, 6 up, 7 in 2023-07-14T19:13:29.974 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:13:29 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-07-14T19:13:29.974 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:13:29 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-07-14T19:13:29.975 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:13:29 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:13:29.975 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:13:29 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:13:29.975 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:13:29 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-07-14T19:13:29.975 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:13:29 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-07-14T19:13:29.975 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:13:29 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-07-14T19:13:29.975 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:13:29 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:13:29.975 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:13:29 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:13:29.976 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:13:29 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T19:13:29.976 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:13:29 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:13:30.225 INFO:teuthology.orchestra.run.smithi130.stderr:Inferring config /var/lib/ceph/adc52f04-2279-11ee-9b33-001a4aab830c/mon.b/config 2023-07-14T19:13:30.736 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:13:30 smithi130 ceph-mon[97832]: purged_snaps scrub starts 2023-07-14T19:13:30.736 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:13:30 smithi130 ceph-mon[97832]: purged_snaps scrub ok 2023-07-14T19:13:30.736 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:13:30 smithi130 ceph-mon[97832]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail 2023-07-14T19:13:30.736 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:13:30 smithi130 ceph-mon[97832]: Detected new or changed devices on smithi040 2023-07-14T19:13:30.736 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:13:30 smithi130 ceph-mon[97832]: Adjusting osd_memory_target on smithi040 to 5724M 2023-07-14T19:13:30.737 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:13:30 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-07-14T19:13:30.737 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:13:30 smithi130 ceph-mon[97832]: osd.6 [v2:172.21.15.40:6818/693186773,v1:172.21.15.40:6819/693186773] boot 2023-07-14T19:13:30.737 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:13:30 smithi130 ceph-mon[97832]: osdmap e45: 7 total, 7 up, 7 in 2023-07-14T19:13:30.737 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:13:30 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-07-14T19:13:30.971 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:13:30 smithi040 ceph-mon[96518]: purged_snaps scrub starts 2023-07-14T19:13:30.971 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:13:30 smithi040 ceph-mon[96518]: purged_snaps scrub ok 2023-07-14T19:13:30.972 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:13:30 smithi040 ceph-mon[96518]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail 2023-07-14T19:13:30.972 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:13:30 smithi040 ceph-mon[96518]: Detected new or changed devices on smithi040 2023-07-14T19:13:30.972 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:13:30 smithi040 ceph-mon[96518]: Adjusting osd_memory_target on smithi040 to 5724M 2023-07-14T19:13:30.972 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:13:30 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-07-14T19:13:30.972 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:13:30 smithi040 ceph-mon[96518]: osd.6 [v2:172.21.15.40:6818/693186773,v1:172.21.15.40:6819/693186773] boot 2023-07-14T19:13:30.972 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:13:30 smithi040 ceph-mon[96518]: osdmap e45: 7 total, 7 up, 7 in 2023-07-14T19:13:30.973 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:13:30 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-07-14T19:13:30.973 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:13:30 smithi191 ceph-mon[98311]: purged_snaps scrub starts 2023-07-14T19:13:30.973 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:13:30 smithi191 ceph-mon[98311]: purged_snaps scrub ok 2023-07-14T19:13:30.973 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:13:30 smithi191 ceph-mon[98311]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail 2023-07-14T19:13:30.974 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:13:30 smithi191 ceph-mon[98311]: Detected new or changed devices on smithi040 2023-07-14T19:13:30.974 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:13:30 smithi191 ceph-mon[98311]: Adjusting osd_memory_target on smithi040 to 5724M 2023-07-14T19:13:30.974 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:13:30 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-07-14T19:13:30.974 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:13:30 smithi191 ceph-mon[98311]: osd.6 [v2:172.21.15.40:6818/693186773,v1:172.21.15.40:6819/693186773] boot 2023-07-14T19:13:30.974 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:13:30 smithi191 ceph-mon[98311]: osdmap e45: 7 total, 7 up, 7 in 2023-07-14T19:13:30.974 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:13:30 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-07-14T19:13:31.451 INFO:teuthology.orchestra.run.smithi130.stderr:2023-07-14T19:13:31.450+0000 7f7c8a806700 1 -- 172.21.15.130:0/871046996 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7c840febb0 msgr2=0x7f7c840fefc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:13:31.451 INFO:teuthology.orchestra.run.smithi130.stderr:2023-07-14T19:13:31.450+0000 7f7c8a806700 1 --2- 172.21.15.130:0/871046996 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7c840febb0 0x7f7c840fefc0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f7c78009990 tx=0x7f7c78009ca0 comp rx=0 tx=0).stop 2023-07-14T19:13:31.451 INFO:teuthology.orchestra.run.smithi130.stderr:2023-07-14T19:13:31.450+0000 7f7c8a806700 1 -- 172.21.15.130:0/871046996 shutdown_connections 2023-07-14T19:13:31.452 INFO:teuthology.orchestra.run.smithi130.stderr:2023-07-14T19:13:31.450+0000 7f7c8a806700 1 --2- 172.21.15.130:0/871046996 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7c84100740 0x7f7c84107000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:13:31.452 INFO:teuthology.orchestra.run.smithi130.stderr:2023-07-14T19:13:31.450+0000 7f7c8a806700 1 --2- 172.21.15.130:0/871046996 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7c840febb0 0x7f7c840fefc0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:13:31.452 INFO:teuthology.orchestra.run.smithi130.stderr:2023-07-14T19:13:31.450+0000 7f7c8a806700 1 --2- 172.21.15.130:0/871046996 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7c840ffdb0 0x7f7c84100200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:13:31.452 INFO:teuthology.orchestra.run.smithi130.stderr:2023-07-14T19:13:31.450+0000 7f7c8a806700 1 -- 172.21.15.130:0/871046996 >> 172.21.15.130:0/871046996 conn(0x7f7c840fa1e0 msgr2=0x7f7c840fc630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:13:31.452 INFO:teuthology.orchestra.run.smithi130.stderr:2023-07-14T19:13:31.450+0000 7f7c8a806700 1 -- 172.21.15.130:0/871046996 shutdown_connections 2023-07-14T19:13:31.452 INFO:teuthology.orchestra.run.smithi130.stderr:2023-07-14T19:13:31.450+0000 7f7c8a806700 1 -- 172.21.15.130:0/871046996 wait complete. 2023-07-14T19:13:31.454 INFO:teuthology.orchestra.run.smithi130.stderr:2023-07-14T19:13:31.451+0000 7f7c8a806700 1 Processor -- start 2023-07-14T19:13:31.454 INFO:teuthology.orchestra.run.smithi130.stderr:2023-07-14T19:13:31.451+0000 7f7c8a806700 1 -- start start 2023-07-14T19:13:31.454 INFO:teuthology.orchestra.run.smithi130.stderr:2023-07-14T19:13:31.451+0000 7f7c8a806700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7c840febb0 0x7f7c84195ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:13:31.454 INFO:teuthology.orchestra.run.smithi130.stderr:2023-07-14T19:13:31.451+0000 7f7c8a806700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7c840ffdb0 0x7f7c84196010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:13:31.455 INFO:teuthology.orchestra.run.smithi130.stderr:2023-07-14T19:13:31.451+0000 7f7c8a806700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7c84100740 0x7f7c8419d090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:13:31.455 INFO:teuthology.orchestra.run.smithi130.stderr:2023-07-14T19:13:31.451+0000 7f7c8a806700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c8419d660 con 0x7f7c840ffdb0 2023-07-14T19:13:31.455 INFO:teuthology.orchestra.run.smithi130.stderr:2023-07-14T19:13:31.451+0000 7f7c8a806700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c8419d7a0 con 0x7f7c84100740 2023-07-14T19:13:31.455 INFO:teuthology.orchestra.run.smithi130.stderr:2023-07-14T19:13:31.451+0000 7f7c8a806700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c8419d930 con 0x7f7c840febb0 2023-07-14T19:13:31.455 INFO:teuthology.orchestra.run.smithi130.stderr:2023-07-14T19:13:31.451+0000 7f7c88da3700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7c84100740 0x7f7c8419d090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:13:31.455 INFO:teuthology.orchestra.run.smithi130.stderr:2023-07-14T19:13:31.451+0000 7f7c88da3700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7c84100740 0x7f7c8419d090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.130:39748/0 (socket says 172.21.15.130:39748) 2023-07-14T19:13:31.455 INFO:teuthology.orchestra.run.smithi130.stderr:2023-07-14T19:13:31.451+0000 7f7c88da3700 1 -- 172.21.15.130:0/110064191 learned_addr learned my addr 172.21.15.130:0/110064191 (peer_addr_for_me v2:172.21.15.130:0/0) 2023-07-14T19:13:31.456 INFO:teuthology.orchestra.run.smithi130.stderr:2023-07-14T19:13:31.451+0000 7f7c83fff700 1 --2- 172.21.15.130:0/110064191 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7c840febb0 0x7f7c84195ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:13:31.456 INFO:teuthology.orchestra.run.smithi130.stderr:2023-07-14T19:13:31.451+0000 7f7c837fe700 1 --2- 172.21.15.130:0/110064191 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7c840ffdb0 0x7f7c84196010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:13:31.456 INFO:teuthology.orchestra.run.smithi130.stderr:2023-07-14T19:13:31.451+0000 7f7c88da3700 1 -- 172.21.15.130:0/110064191 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7c840febb0 msgr2=0x7f7c84195ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:13:31.456 INFO:teuthology.orchestra.run.smithi130.stderr:2023-07-14T19:13:31.451+0000 7f7c88da3700 1 --2- 172.21.15.130:0/110064191 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7c840febb0 0x7f7c84195ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:13:31.456 INFO:teuthology.orchestra.run.smithi130.stderr:2023-07-14T19:13:31.451+0000 7f7c88da3700 1 -- 172.21.15.130:0/110064191 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7c840ffdb0 msgr2=0x7f7c84196010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:13:31.456 INFO:teuthology.orchestra.run.smithi130.stderr:2023-07-14T19:13:31.451+0000 7f7c88da3700 1 --2- 172.21.15.130:0/110064191 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7c840ffdb0 0x7f7c84196010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:13:31.456 INFO:teuthology.orchestra.run.smithi130.stderr:2023-07-14T19:13:31.451+0000 7f7c88da3700 1 -- 172.21.15.130:0/110064191 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c780095f0 con 0x7f7c84100740 2023-07-14T19:13:31.457 INFO:teuthology.orchestra.run.smithi130.stderr:2023-07-14T19:13:31.451+0000 7f7c837fe700 1 --2- 172.21.15.130:0/110064191 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7c840ffdb0 0x7f7c84196010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-14T19:13:31.457 INFO:teuthology.orchestra.run.smithi130.stderr:2023-07-14T19:13:31.452+0000 7f7c88da3700 1 --2- 172.21.15.130:0/110064191 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7c84100740 0x7f7c8419d090 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f7c7c00b970 tx=0x7f7c7c00bc80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:13:31.457 INFO:teuthology.orchestra.run.smithi130.stderr:2023-07-14T19:13:31.452+0000 7f7c817fa700 1 -- 172.21.15.130:0/110064191 <== mon.2 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c7c00c930 con 0x7f7c84100740 2023-07-14T19:13:31.457 INFO:teuthology.orchestra.run.smithi130.stderr:2023-07-14T19:13:31.452+0000 7f7c817fa700 1 -- 172.21.15.130:0/110064191 <== mon.2 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f7c7c014440 con 0x7f7c84100740 2023-07-14T19:13:31.457 INFO:teuthology.orchestra.run.smithi130.stderr:2023-07-14T19:13:31.452+0000 7f7c8a806700 1 -- 172.21.15.130:0/110064191 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7c8419dc10 con 0x7f7c84100740 2023-07-14T19:13:31.457 INFO:teuthology.orchestra.run.smithi130.stderr:2023-07-14T19:13:31.452+0000 7f7c817fa700 1 -- 172.21.15.130:0/110064191 <== mon.2 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c7c0075d0 con 0x7f7c84100740 2023-07-14T19:13:31.457 INFO:teuthology.orchestra.run.smithi130.stderr:2023-07-14T19:13:31.452+0000 7f7c8a806700 1 -- 172.21.15.130:0/110064191 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7c8419e160 con 0x7f7c84100740 2023-07-14T19:13:31.457 INFO:teuthology.orchestra.run.smithi130.stderr:2023-07-14T19:13:31.453+0000 7f7c8a806700 1 -- 172.21.15.130:0/110064191 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7c8410a470 con 0x7f7c84100740 2023-07-14T19:13:31.458 INFO:teuthology.orchestra.run.smithi130.stderr:2023-07-14T19:13:31.454+0000 7f7c817fa700 1 -- 172.21.15.130:0/110064191 <== mon.2 v2:172.21.15.130:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f7c7c0148d0 con 0x7f7c84100740 2023-07-14T19:13:31.458 INFO:teuthology.orchestra.run.smithi130.stderr:2023-07-14T19:13:31.455+0000 7f7c817fa700 1 --2- 172.21.15.130:0/110064191 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f7c6c0a11d0 0x7f7c6c0a3680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:13:31.458 INFO:teuthology.orchestra.run.smithi130.stderr:2023-07-14T19:13:31.455+0000 7f7c817fa700 1 -- 172.21.15.130:0/110064191 <== mon.2 v2:172.21.15.130:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 4965+0+0 (secure 0 0 0) 0x7f7c7c006e40 con 0x7f7c84100740 2023-07-14T19:13:31.459 INFO:teuthology.orchestra.run.smithi130.stderr:2023-07-14T19:13:31.455+0000 7f7c83fff700 1 --2- 172.21.15.130:0/110064191 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f7c6c0a11d0 0x7f7c6c0a3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:13:31.459 INFO:teuthology.orchestra.run.smithi130.stderr:2023-07-14T19:13:31.456+0000 7f7c83fff700 1 --2- 172.21.15.130:0/110064191 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f7c6c0a11d0 0x7f7c6c0a3680 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f7c7800b5c0 tx=0x7f7c78005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:13:31.459 INFO:teuthology.orchestra.run.smithi130.stderr:2023-07-14T19:13:31.458+0000 7f7c817fa700 1 -- 172.21.15.130:0/110064191 <== mon.2 v2:172.21.15.130:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f7c7c095a70 con 0x7f7c84100740 2023-07-14T19:13:31.657 INFO:teuthology.orchestra.run.smithi130.stderr:2023-07-14T19:13:31.656+0000 7f7c8a806700 1 -- 172.21.15.130:0/110064191 --> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi130:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) v1 -- 0x7f7c840690c0 con 0x7f7c6c0a11d0 2023-07-14T19:13:31.971 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:13:31 smithi040 ceph-mon[96518]: osdmap e46: 7 total, 7 up, 7 in 2023-07-14T19:13:31.972 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:13:31 smithi191 ceph-mon[98311]: osdmap e46: 7 total, 7 up, 7 in 2023-07-14T19:13:31.992 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:13:31 smithi130 ceph-mon[97832]: osdmap e46: 7 total, 7 up, 7 in 2023-07-14T19:13:32.742 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:13:32 smithi130 ceph-mon[97832]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 188 MiB used, 626 GiB / 626 GiB avail 2023-07-14T19:13:32.742 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:13:32 smithi130 ceph-mon[97832]: from='client.24287 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi130:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T19:13:32.743 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:13:32 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-14T19:13:32.743 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:13:32 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-14T19:13:32.743 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:13:32 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:13:32.971 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:13:32 smithi040 ceph-mon[96518]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 188 MiB used, 626 GiB / 626 GiB avail 2023-07-14T19:13:32.971 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:13:32 smithi040 ceph-mon[96518]: from='client.24287 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi130:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T19:13:32.971 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:13:32 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-14T19:13:32.972 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:13:32 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-14T19:13:32.972 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:13:32 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:13:32.972 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:13:32 smithi191 ceph-mon[98311]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 188 MiB used, 626 GiB / 626 GiB avail 2023-07-14T19:13:32.972 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:13:32 smithi191 ceph-mon[98311]: from='client.24287 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi130:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T19:13:32.973 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:13:32 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-14T19:13:32.973 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:13:32 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-14T19:13:32.973 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:13:32 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:13:34.971 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:13:34 smithi040 ceph-mon[96518]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 188 MiB used, 626 GiB / 626 GiB avail 2023-07-14T19:13:34.972 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:13:34 smithi040 ceph-mon[96518]: from='client.? 172.21.15.130:0/1482338220' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1700fbfa-cbfc-4f76-97be-727a3b100740"}]: dispatch 2023-07-14T19:13:34.972 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:13:34 smithi040 ceph-mon[96518]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1700fbfa-cbfc-4f76-97be-727a3b100740"}]: dispatch 2023-07-14T19:13:34.972 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:13:34 smithi040 ceph-mon[96518]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "1700fbfa-cbfc-4f76-97be-727a3b100740"}]': finished 2023-07-14T19:13:34.972 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:13:34 smithi040 ceph-mon[96518]: osdmap e47: 8 total, 7 up, 8 in 2023-07-14T19:13:34.972 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:13:34 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-07-14T19:13:34.973 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:13:34 smithi191 ceph-mon[98311]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 188 MiB used, 626 GiB / 626 GiB avail 2023-07-14T19:13:34.973 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:13:34 smithi191 ceph-mon[98311]: from='client.? 172.21.15.130:0/1482338220' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1700fbfa-cbfc-4f76-97be-727a3b100740"}]: dispatch 2023-07-14T19:13:34.973 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:13:34 smithi191 ceph-mon[98311]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1700fbfa-cbfc-4f76-97be-727a3b100740"}]: dispatch 2023-07-14T19:13:34.973 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:13:34 smithi191 ceph-mon[98311]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "1700fbfa-cbfc-4f76-97be-727a3b100740"}]': finished 2023-07-14T19:13:34.973 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:13:34 smithi191 ceph-mon[98311]: osdmap e47: 8 total, 7 up, 8 in 2023-07-14T19:13:34.974 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:13:34 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-07-14T19:13:34.993 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:13:34 smithi130 ceph-mon[97832]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 188 MiB used, 626 GiB / 626 GiB avail 2023-07-14T19:13:34.993 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:13:34 smithi130 ceph-mon[97832]: from='client.? 172.21.15.130:0/1482338220' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1700fbfa-cbfc-4f76-97be-727a3b100740"}]: dispatch 2023-07-14T19:13:34.993 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:13:34 smithi130 ceph-mon[97832]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1700fbfa-cbfc-4f76-97be-727a3b100740"}]: dispatch 2023-07-14T19:13:34.993 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:13:34 smithi130 ceph-mon[97832]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "1700fbfa-cbfc-4f76-97be-727a3b100740"}]': finished 2023-07-14T19:13:34.993 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:13:34 smithi130 ceph-mon[97832]: osdmap e47: 8 total, 7 up, 8 in 2023-07-14T19:13:34.994 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:13:34 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-07-14T19:13:35.971 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:13:35 smithi040 ceph-mon[96518]: from='client.? 172.21.15.130:0/2063235719' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-14T19:13:35.972 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:13:35 smithi191 ceph-mon[98311]: from='client.? 172.21.15.130:0/2063235719' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-14T19:13:35.992 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:13:35 smithi130 ceph-mon[97832]: from='client.? 172.21.15.130:0/2063235719' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-14T19:13:36.971 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:13:36 smithi040 ceph-mon[96518]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 188 MiB used, 626 GiB / 626 GiB avail 2023-07-14T19:13:36.972 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:13:36 smithi191 ceph-mon[98311]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 188 MiB used, 626 GiB / 626 GiB avail 2023-07-14T19:13:36.992 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:13:36 smithi130 ceph-mon[97832]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 188 MiB used, 626 GiB / 626 GiB avail 2023-07-14T19:13:38.854 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:13:38 smithi130 ceph-mon[97832]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 188 MiB used, 626 GiB / 626 GiB avail 2023-07-14T19:13:38.971 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:13:38 smithi040 ceph-mon[96518]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 188 MiB used, 626 GiB / 626 GiB avail 2023-07-14T19:13:38.972 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:13:38 smithi191 ceph-mon[98311]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 188 MiB used, 626 GiB / 626 GiB avail 2023-07-14T19:13:40.742 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:13:40 smithi130 ceph-mon[97832]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 188 MiB used, 626 GiB / 626 GiB avail 2023-07-14T19:13:40.743 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:13:40 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2023-07-14T19:13:40.743 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:13:40 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:13:40.971 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:13:40 smithi040 ceph-mon[96518]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 188 MiB used, 626 GiB / 626 GiB avail 2023-07-14T19:13:40.972 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:13:40 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2023-07-14T19:13:40.972 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:13:40 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:13:40.972 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:13:40 smithi191 ceph-mon[98311]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 188 MiB used, 626 GiB / 626 GiB avail 2023-07-14T19:13:40.973 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:13:40 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2023-07-14T19:13:40.973 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:13:40 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:13:41.742 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:13:41 smithi130 ceph-mon[97832]: Deploying daemon osd.7 on smithi130 2023-07-14T19:13:41.971 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:13:41 smithi040 ceph-mon[96518]: Deploying daemon osd.7 on smithi130 2023-07-14T19:13:41.973 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:13:41 smithi191 ceph-mon[98311]: Deploying daemon osd.7 on smithi130 2023-07-14T19:13:42.966 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:13:42 smithi130 ceph-mon[97832]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 188 MiB used, 626 GiB / 626 GiB avail 2023-07-14T19:13:42.971 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:13:42 smithi040 ceph-mon[96518]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 188 MiB used, 626 GiB / 626 GiB avail 2023-07-14T19:13:42.972 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:13:42 smithi191 ceph-mon[98311]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 188 MiB used, 626 GiB / 626 GiB avail 2023-07-14T19:13:44.886 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:13:44 smithi130 ceph-mon[97832]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 188 MiB used, 626 GiB / 626 GiB avail 2023-07-14T19:13:44.971 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:13:44 smithi040 ceph-mon[96518]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 188 MiB used, 626 GiB / 626 GiB avail 2023-07-14T19:13:44.972 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:13:44 smithi191 ceph-mon[98311]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 188 MiB used, 626 GiB / 626 GiB avail 2023-07-14T19:13:46.208 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:13:46 smithi130 ceph-mon[97832]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 188 MiB used, 626 GiB / 626 GiB avail 2023-07-14T19:13:46.208 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:13:46 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:13:46.208 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:13:46 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:13:46.209 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:13:46 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T19:13:46.471 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:13:46 smithi040 ceph-mon[96518]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 188 MiB used, 626 GiB / 626 GiB avail 2023-07-14T19:13:46.471 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:13:46 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:13:46.472 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:13:46 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:13:46.472 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:13:46 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T19:13:46.472 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:13:46 smithi191 ceph-mon[98311]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 188 MiB used, 626 GiB / 626 GiB avail 2023-07-14T19:13:46.472 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:13:46 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:13:46.472 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:13:46 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:13:46.473 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:13:46 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T19:13:47.673 INFO:teuthology.orchestra.run.smithi130.stderr:2023-07-14T19:13:47.670+0000 7f7c817fa700 1 -- 172.21.15.130:0/110064191 <== mgr.14152 v2:172.21.15.40:6800/1693027118 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f7c840690c0 con 0x7f7c6c0a11d0 2023-07-14T19:13:47.673 INFO:teuthology.orchestra.run.smithi130.stdout:Created osd(s) 7 on host 'smithi130' 2023-07-14T19:13:47.675 INFO:teuthology.orchestra.run.smithi130.stderr:2023-07-14T19:13:47.673+0000 7f7c8a806700 1 -- 172.21.15.130:0/110064191 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f7c6c0a11d0 msgr2=0x7f7c6c0a3680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:13:47.675 INFO:teuthology.orchestra.run.smithi130.stderr:2023-07-14T19:13:47.673+0000 7f7c8a806700 1 --2- 172.21.15.130:0/110064191 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f7c6c0a11d0 0x7f7c6c0a3680 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f7c7800b5c0 tx=0x7f7c78005fd0 comp rx=0 tx=0).stop 2023-07-14T19:13:47.675 INFO:teuthology.orchestra.run.smithi130.stderr:2023-07-14T19:13:47.673+0000 7f7c8a806700 1 -- 172.21.15.130:0/110064191 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7c84100740 msgr2=0x7f7c8419d090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:13:47.675 INFO:teuthology.orchestra.run.smithi130.stderr:2023-07-14T19:13:47.673+0000 7f7c8a806700 1 --2- 172.21.15.130:0/110064191 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7c84100740 0x7f7c8419d090 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f7c7c00b970 tx=0x7f7c7c00bc80 comp rx=0 tx=0).stop 2023-07-14T19:13:47.675 INFO:teuthology.orchestra.run.smithi130.stderr:2023-07-14T19:13:47.673+0000 7f7c8a806700 1 -- 172.21.15.130:0/110064191 shutdown_connections 2023-07-14T19:13:47.676 INFO:teuthology.orchestra.run.smithi130.stderr:2023-07-14T19:13:47.673+0000 7f7c8a806700 1 --2- 172.21.15.130:0/110064191 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f7c6c0a11d0 0x7f7c6c0a3680 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:13:47.676 INFO:teuthology.orchestra.run.smithi130.stderr:2023-07-14T19:13:47.673+0000 7f7c8a806700 1 --2- 172.21.15.130:0/110064191 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7c840febb0 0x7f7c84195ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:13:47.676 INFO:teuthology.orchestra.run.smithi130.stderr:2023-07-14T19:13:47.673+0000 7f7c8a806700 1 --2- 172.21.15.130:0/110064191 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7c84100740 0x7f7c8419d090 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:13:47.676 INFO:teuthology.orchestra.run.smithi130.stderr:2023-07-14T19:13:47.673+0000 7f7c8a806700 1 --2- 172.21.15.130:0/110064191 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7c840ffdb0 0x7f7c84196010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:13:47.676 INFO:teuthology.orchestra.run.smithi130.stderr:2023-07-14T19:13:47.673+0000 7f7c8a806700 1 -- 172.21.15.130:0/110064191 >> 172.21.15.130:0/110064191 conn(0x7f7c840fa1e0 msgr2=0x7f7c840689b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:13:47.676 INFO:teuthology.orchestra.run.smithi130.stderr:2023-07-14T19:13:47.673+0000 7f7c8a806700 1 -- 172.21.15.130:0/110064191 shutdown_connections 2023-07-14T19:13:47.677 INFO:teuthology.orchestra.run.smithi130.stderr:2023-07-14T19:13:47.673+0000 7f7c8a806700 1 -- 172.21.15.130:0/110064191 wait complete. 2023-07-14T19:13:48.120 DEBUG:teuthology.orchestra.run.smithi130:osd.7> sudo journalctl -f -n 0 -u ceph-adc52f04-2279-11ee-9b33-001a4aab830c@osd.7.service 2023-07-14T19:13:48.123 INFO:tasks.cephadm:Deploying osd.8 on smithi191 with /dev/vg_nvme/lv_2... 2023-07-14T19:13:48.123 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid adc52f04-2279-11ee-9b33-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2023-07-14T19:13:48.153 INFO:journalctl@ceph.osd.7.smithi130.stdout:-- Logs begin at Fri 2023-07-14 18:50:32 UTC. -- 2023-07-14T19:13:48.686 INFO:teuthology.orchestra.run.smithi191.stderr:Inferring config /var/lib/ceph/adc52f04-2279-11ee-9b33-001a4aab830c/mon.c/config 2023-07-14T19:13:48.891 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:13:48 smithi130 ceph-mon[97832]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 188 MiB used, 626 GiB / 626 GiB avail 2023-07-14T19:13:48.891 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:13:48 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:13:48.891 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:13:48 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:13:48.891 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:13:48 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:13:48.891 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:13:48 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T19:13:48.892 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:13:48 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:13:48.892 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:13:48 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:13:48.892 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:13:48 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:13:48.892 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:13:48 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T19:13:48.892 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:13:48 smithi191 ceph-mon[98311]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 188 MiB used, 626 GiB / 626 GiB avail 2023-07-14T19:13:48.893 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:13:48 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:13:48.893 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:13:48 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:13:48.893 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:13:48 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:13:48.893 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:13:48 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T19:13:48.893 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:13:48 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:13:48.893 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:13:48 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:13:48.893 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:13:48 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:13:48.894 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:13:48 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T19:13:48.971 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:13:48 smithi040 ceph-mon[96518]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 188 MiB used, 626 GiB / 626 GiB avail 2023-07-14T19:13:48.972 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:13:48 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:13:48.972 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:13:48 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:13:48.972 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:13:48 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:13:48.972 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:13:48 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T19:13:48.972 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:13:48 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:13:48.972 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:13:48 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:13:48.972 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:13:48 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:13:48.973 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:13:48 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T19:13:49.160 INFO:journalctl@ceph.osd.7.smithi130.stdout:Jul 14 19:13:49 smithi130 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-7[113739]: 2023-07-14T19:13:49.032+0000 7f62239bd640 -1 osd.7 0 log_to_monitors true 2023-07-14T19:13:49.897 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:13:49 smithi130 ceph-mon[97832]: from='osd.7 [v2:172.21.15.130:6816/2384019528,v1:172.21.15.130:6817/2384019528]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2023-07-14T19:13:49.897 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:13:49 smithi130 ceph-mon[97832]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2023-07-14T19:13:49.971 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:13:49 smithi040 ceph-mon[96518]: from='osd.7 [v2:172.21.15.130:6816/2384019528,v1:172.21.15.130:6817/2384019528]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2023-07-14T19:13:49.971 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:13:49 smithi040 ceph-mon[96518]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2023-07-14T19:13:49.972 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:13:49 smithi191 ceph-mon[98311]: from='osd.7 [v2:172.21.15.130:6816/2384019528,v1:172.21.15.130:6817/2384019528]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2023-07-14T19:13:49.972 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:13:49 smithi191 ceph-mon[98311]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2023-07-14T19:13:50.152 INFO:teuthology.orchestra.run.smithi191.stderr:--> Zapping: /dev/vg_nvme/lv_2 2023-07-14T19:13:50.152 INFO:teuthology.orchestra.run.smithi191.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2023-07-14T19:13:50.152 INFO:teuthology.orchestra.run.smithi191.stderr: stderr: 10+0 records in 2023-07-14T19:13:50.152 INFO:teuthology.orchestra.run.smithi191.stderr:10+0 records out 2023-07-14T19:13:50.152 INFO:teuthology.orchestra.run.smithi191.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.011079 s, 946 MB/s 2023-07-14T19:13:50.152 INFO:teuthology.orchestra.run.smithi191.stderr:--> Zapping successful for: 2023-07-14T19:13:50.626 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid adc52f04-2279-11ee-9b33-001a4aab830c -- ceph orch daemon add osd smithi191:vg_nvme/lv_2 2023-07-14T19:13:50.971 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:13:50 smithi040 ceph-mon[96518]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 188 MiB used, 626 GiB / 626 GiB avail 2023-07-14T19:13:50.972 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:13:50 smithi040 ceph-mon[96518]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2023-07-14T19:13:50.972 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:13:50 smithi040 ceph-mon[96518]: osdmap e48: 8 total, 7 up, 8 in 2023-07-14T19:13:50.972 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:13:50 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-07-14T19:13:50.973 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:13:50 smithi040 ceph-mon[96518]: from='osd.7 [v2:172.21.15.130:6816/2384019528,v1:172.21.15.130:6817/2384019528]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi130", "root=default"]}]: dispatch 2023-07-14T19:13:50.973 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:13:50 smithi040 ceph-mon[96518]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi130", "root=default"]}]: dispatch 2023-07-14T19:13:50.973 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:13:50 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:13:50.974 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:13:50 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:13:50.974 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:13:50 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-07-14T19:13:50.974 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:13:50 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-07-14T19:13:50.975 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:13:50 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-07-14T19:13:50.975 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:13:50 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:13:50.975 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:13:50 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:13:50.976 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:13:50 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T19:13:50.976 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:13:50 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:13:50.977 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:13:50 smithi191 ceph-mon[98311]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 188 MiB used, 626 GiB / 626 GiB avail 2023-07-14T19:13:50.977 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:13:50 smithi191 ceph-mon[98311]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2023-07-14T19:13:50.977 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:13:50 smithi191 ceph-mon[98311]: osdmap e48: 8 total, 7 up, 8 in 2023-07-14T19:13:50.977 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:13:50 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-07-14T19:13:50.978 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:13:50 smithi191 ceph-mon[98311]: from='osd.7 [v2:172.21.15.130:6816/2384019528,v1:172.21.15.130:6817/2384019528]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi130", "root=default"]}]: dispatch 2023-07-14T19:13:50.978 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:13:50 smithi191 ceph-mon[98311]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi130", "root=default"]}]: dispatch 2023-07-14T19:13:50.978 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:13:50 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:13:50.978 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:13:50 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:13:50.979 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:13:50 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-07-14T19:13:50.979 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:13:50 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-07-14T19:13:50.979 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:13:50 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-07-14T19:13:50.979 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:13:50 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:13:50.980 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:13:50 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:13:50.980 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:13:50 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T19:13:50.980 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:13:50 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:13:50.992 INFO:journalctl@ceph.osd.7.smithi130.stdout:Jul 14 19:13:50 smithi130 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-7[113739]: 2023-07-14T19:13:50.569+0000 7f6219035700 -1 osd.7 0 waiting for initial osdmap 2023-07-14T19:13:50.992 INFO:journalctl@ceph.osd.7.smithi130.stdout:Jul 14 19:13:50 smithi130 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-7[113739]: 2023-07-14T19:13:50.582+0000 7f6212e27700 -1 osd.7 49 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-07-14T19:13:50.993 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:13:50 smithi130 ceph-mon[97832]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 188 MiB used, 626 GiB / 626 GiB avail 2023-07-14T19:13:50.993 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:13:50 smithi130 ceph-mon[97832]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2023-07-14T19:13:50.993 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:13:50 smithi130 ceph-mon[97832]: osdmap e48: 8 total, 7 up, 8 in 2023-07-14T19:13:50.993 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:13:50 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-07-14T19:13:50.993 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:13:50 smithi130 ceph-mon[97832]: from='osd.7 [v2:172.21.15.130:6816/2384019528,v1:172.21.15.130:6817/2384019528]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi130", "root=default"]}]: dispatch 2023-07-14T19:13:50.994 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:13:50 smithi130 ceph-mon[97832]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi130", "root=default"]}]: dispatch 2023-07-14T19:13:50.994 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:13:50 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:13:50.994 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:13:50 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:13:50.994 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:13:50 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-07-14T19:13:50.994 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:13:50 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-07-14T19:13:50.994 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:13:50 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-07-14T19:13:50.994 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:13:50 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:13:50.994 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:13:50 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:13:50.995 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:13:50 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T19:13:50.995 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:13:50 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:13:51.176 INFO:teuthology.orchestra.run.smithi191.stderr:Inferring config /var/lib/ceph/adc52f04-2279-11ee-9b33-001a4aab830c/mon.c/config 2023-07-14T19:13:51.677 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:13:51 smithi191 ceph-mon[98311]: Detected new or changed devices on smithi130 2023-07-14T19:13:51.677 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:13:51 smithi191 ceph-mon[98311]: Adjusting osd_memory_target on smithi130 to 5722M 2023-07-14T19:13:51.678 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:13:51 smithi191 ceph-mon[98311]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi130", "root=default"]}]': finished 2023-07-14T19:13:51.678 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:13:51 smithi191 ceph-mon[98311]: osdmap e49: 8 total, 7 up, 8 in 2023-07-14T19:13:51.678 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:13:51 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-07-14T19:13:51.678 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:13:51 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-07-14T19:13:51.678 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:13:51 smithi191 ceph-mon[98311]: osd.7 [v2:172.21.15.130:6816/2384019528,v1:172.21.15.130:6817/2384019528] boot 2023-07-14T19:13:51.678 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:13:51 smithi191 ceph-mon[98311]: osdmap e50: 8 total, 8 up, 8 in 2023-07-14T19:13:51.679 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:13:51 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-07-14T19:13:51.971 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:13:51 smithi040 ceph-mon[96518]: Detected new or changed devices on smithi130 2023-07-14T19:13:51.972 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:13:51 smithi040 ceph-mon[96518]: Adjusting osd_memory_target on smithi130 to 5722M 2023-07-14T19:13:51.972 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:13:51 smithi040 ceph-mon[96518]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi130", "root=default"]}]': finished 2023-07-14T19:13:51.972 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:13:51 smithi040 ceph-mon[96518]: osdmap e49: 8 total, 7 up, 8 in 2023-07-14T19:13:51.972 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:13:51 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-07-14T19:13:51.972 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:13:51 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-07-14T19:13:51.972 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:13:51 smithi040 ceph-mon[96518]: osd.7 [v2:172.21.15.130:6816/2384019528,v1:172.21.15.130:6817/2384019528] boot 2023-07-14T19:13:51.972 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:13:51 smithi040 ceph-mon[96518]: osdmap e50: 8 total, 8 up, 8 in 2023-07-14T19:13:51.973 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:13:51 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-07-14T19:13:51.992 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:13:51 smithi130 ceph-mon[97832]: Detected new or changed devices on smithi130 2023-07-14T19:13:51.992 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:13:51 smithi130 ceph-mon[97832]: Adjusting osd_memory_target on smithi130 to 5722M 2023-07-14T19:13:51.993 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:13:51 smithi130 ceph-mon[97832]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi130", "root=default"]}]': finished 2023-07-14T19:13:51.993 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:13:51 smithi130 ceph-mon[97832]: osdmap e49: 8 total, 7 up, 8 in 2023-07-14T19:13:51.993 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:13:51 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-07-14T19:13:51.993 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:13:51 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-07-14T19:13:51.993 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:13:51 smithi130 ceph-mon[97832]: osd.7 [v2:172.21.15.130:6816/2384019528,v1:172.21.15.130:6817/2384019528] boot 2023-07-14T19:13:51.993 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:13:51 smithi130 ceph-mon[97832]: osdmap e50: 8 total, 8 up, 8 in 2023-07-14T19:13:51.993 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:13:51 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-07-14T19:13:52.386 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:13:52.384+0000 7f7e464ff700 1 -- 172.21.15.191:0/1128993784 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7e400febd0 msgr2=0x7f7e400fefe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:13:52.386 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:13:52.384+0000 7f7e464ff700 1 --2- 172.21.15.191:0/1128993784 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7e400febd0 0x7f7e400fefe0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f7e34009990 tx=0x7f7e34009ca0 comp rx=0 tx=0).stop 2023-07-14T19:13:52.387 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:13:52.384+0000 7f7e464ff700 1 -- 172.21.15.191:0/1128993784 shutdown_connections 2023-07-14T19:13:52.387 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:13:52.384+0000 7f7e464ff700 1 --2- 172.21.15.191:0/1128993784 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7e400febd0 0x7f7e400fefe0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:13:52.387 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:13:52.384+0000 7f7e464ff700 1 --2- 172.21.15.191:0/1128993784 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7e40100820 0x7f7e401070c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:13:52.387 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:13:52.384+0000 7f7e464ff700 1 --2- 172.21.15.191:0/1128993784 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7e400ffe70 0x7f7e401002e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:13:52.387 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:13:52.384+0000 7f7e464ff700 1 -- 172.21.15.191:0/1128993784 >> 172.21.15.191:0/1128993784 conn(0x7f7e400fa240 msgr2=0x7f7e400fc650 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:13:52.387 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:13:52.384+0000 7f7e464ff700 1 -- 172.21.15.191:0/1128993784 shutdown_connections 2023-07-14T19:13:52.388 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:13:52.384+0000 7f7e464ff700 1 -- 172.21.15.191:0/1128993784 wait complete. 2023-07-14T19:13:52.389 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:13:52.384+0000 7f7e464ff700 1 Processor -- start 2023-07-14T19:13:52.390 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:13:52.384+0000 7f7e464ff700 1 -- start start 2023-07-14T19:13:52.390 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:13:52.385+0000 7f7e464ff700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7e400febd0 0x7f7e40069580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:13:52.390 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:13:52.385+0000 7f7e464ff700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7e400ffe70 0x7f7e4019cb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:13:52.390 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:13:52.385+0000 7f7e464ff700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7e40100820 0x7f7e4019f090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:13:52.390 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:13:52.385+0000 7f7e464ff700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e4019f740 con 0x7f7e40100820 2023-07-14T19:13:52.391 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:13:52.385+0000 7f7e464ff700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e4019f8b0 con 0x7f7e400febd0 2023-07-14T19:13:52.391 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:13:52.385+0000 7f7e464ff700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e4019fa70 con 0x7f7e400ffe70 2023-07-14T19:13:52.391 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:13:52.385+0000 7f7e3f7fe700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7e400ffe70 0x7f7e4019cb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:13:52.391 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:13:52.385+0000 7f7e3f7fe700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7e400ffe70 0x7f7e4019cb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.191:59470/0 (socket says 172.21.15.191:59470) 2023-07-14T19:13:52.392 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:13:52.385+0000 7f7e3f7fe700 1 -- 172.21.15.191:0/3492393589 learned_addr learned my addr 172.21.15.191:0/3492393589 (peer_addr_for_me v2:172.21.15.191:0/0) 2023-07-14T19:13:52.392 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:13:52.385+0000 7f7e3ffff700 1 --2- 172.21.15.191:0/3492393589 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7e400febd0 0x7f7e40069580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:13:52.392 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:13:52.385+0000 7f7e44a9c700 1 --2- 172.21.15.191:0/3492393589 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7e40100820 0x7f7e4019f090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:13:52.392 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:13:52.385+0000 7f7e3f7fe700 1 -- 172.21.15.191:0/3492393589 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7e400febd0 msgr2=0x7f7e40069580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:13:52.392 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:13:52.385+0000 7f7e3f7fe700 1 --2- 172.21.15.191:0/3492393589 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7e400febd0 0x7f7e40069580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:13:52.392 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:13:52.385+0000 7f7e3f7fe700 1 -- 172.21.15.191:0/3492393589 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7e40100820 msgr2=0x7f7e4019f090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:13:52.392 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:13:52.385+0000 7f7e3f7fe700 1 --2- 172.21.15.191:0/3492393589 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7e40100820 0x7f7e4019f090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:13:52.393 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:13:52.385+0000 7f7e3f7fe700 1 -- 172.21.15.191:0/3492393589 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e340095f0 con 0x7f7e400ffe70 2023-07-14T19:13:52.393 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:13:52.385+0000 7f7e3ffff700 1 --2- 172.21.15.191:0/3492393589 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7e400febd0 0x7f7e40069580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T19:13:52.393 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:13:52.385+0000 7f7e44a9c700 1 --2- 172.21.15.191:0/3492393589 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7e40100820 0x7f7e4019f090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-14T19:13:52.393 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:13:52.385+0000 7f7e3f7fe700 1 --2- 172.21.15.191:0/3492393589 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7e400ffe70 0x7f7e4019cb40 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f7e3000d950 tx=0x7f7e3000dd10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:13:52.393 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:13:52.385+0000 7f7e3d7fa700 1 -- 172.21.15.191:0/3492393589 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e30009b40 con 0x7f7e400ffe70 2023-07-14T19:13:52.393 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:13:52.385+0000 7f7e3d7fa700 1 -- 172.21.15.191:0/3492393589 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f7e30009ca0 con 0x7f7e400ffe70 2023-07-14T19:13:52.393 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:13:52.385+0000 7f7e464ff700 1 -- 172.21.15.191:0/3492393589 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e4019fd50 con 0x7f7e400ffe70 2023-07-14T19:13:52.394 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:13:52.386+0000 7f7e3d7fa700 1 -- 172.21.15.191:0/3492393589 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e3000f620 con 0x7f7e400ffe70 2023-07-14T19:13:52.394 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:13:52.386+0000 7f7e464ff700 1 -- 172.21.15.191:0/3492393589 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e401a0240 con 0x7f7e400ffe70 2023-07-14T19:13:52.394 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:13:52.387+0000 7f7e464ff700 1 -- 172.21.15.191:0/3492393589 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7e4010a3a0 con 0x7f7e400ffe70 2023-07-14T19:13:52.394 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:13:52.388+0000 7f7e3d7fa700 1 -- 172.21.15.191:0/3492393589 <== mon.1 v2:172.21.15.191:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f7e30010460 con 0x7f7e400ffe70 2023-07-14T19:13:52.395 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:13:52.388+0000 7f7e3d7fa700 1 --2- 172.21.15.191:0/3492393589 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f7e280a11d0 0x7f7e280a3680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:13:52.395 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:13:52.388+0000 7f7e3d7fa700 1 -- 172.21.15.191:0/3492393589 <== mon.1 v2:172.21.15.191:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 5430+0+0 (secure 0 0 0) 0x7f7e300cc0b0 con 0x7f7e400ffe70 2023-07-14T19:13:52.395 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:13:52.388+0000 7f7e3ffff700 1 --2- 172.21.15.191:0/3492393589 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f7e280a11d0 0x7f7e280a3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:13:52.395 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:13:52.389+0000 7f7e3ffff700 1 --2- 172.21.15.191:0/3492393589 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f7e280a11d0 0x7f7e280a3680 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f7e34006010 tx=0x7f7e3400b540 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:13:52.396 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:13:52.392+0000 7f7e3d7fa700 1 -- 172.21.15.191:0/3492393589 <== mon.1 v2:172.21.15.191:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f7e30096340 con 0x7f7e400ffe70 2023-07-14T19:13:52.591 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:13:52.589+0000 7f7e464ff700 1 -- 172.21.15.191:0/3492393589 --> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi191:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) v1 -- 0x7f7e40060a80 con 0x7f7e280a11d0 2023-07-14T19:13:52.971 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:13:52 smithi040 ceph-mon[96518]: purged_snaps scrub starts 2023-07-14T19:13:52.971 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:13:52 smithi040 ceph-mon[96518]: purged_snaps scrub ok 2023-07-14T19:13:52.971 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:13:52 smithi040 ceph-mon[96518]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 188 MiB used, 626 GiB / 626 GiB avail 2023-07-14T19:13:52.972 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:13:52 smithi040 ceph-mon[96518]: osdmap e51: 8 total, 8 up, 8 in 2023-07-14T19:13:52.972 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:13:52 smithi191 ceph-mon[98311]: purged_snaps scrub starts 2023-07-14T19:13:52.972 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:13:52 smithi191 ceph-mon[98311]: purged_snaps scrub ok 2023-07-14T19:13:52.972 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:13:52 smithi191 ceph-mon[98311]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 188 MiB used, 626 GiB / 626 GiB avail 2023-07-14T19:13:52.972 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:13:52 smithi191 ceph-mon[98311]: osdmap e51: 8 total, 8 up, 8 in 2023-07-14T19:13:52.992 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:13:52 smithi130 ceph-mon[97832]: purged_snaps scrub starts 2023-07-14T19:13:52.993 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:13:52 smithi130 ceph-mon[97832]: purged_snaps scrub ok 2023-07-14T19:13:52.993 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:13:52 smithi130 ceph-mon[97832]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 188 MiB used, 626 GiB / 626 GiB avail 2023-07-14T19:13:52.993 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:13:52 smithi130 ceph-mon[97832]: osdmap e51: 8 total, 8 up, 8 in 2023-07-14T19:13:53.886 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:13:53 smithi191 ceph-mon[98311]: from='client.24298 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi191:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T19:13:53.886 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:13:53 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-14T19:13:53.886 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:13:53 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-14T19:13:53.887 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:13:53 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:13:53.887 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:13:53 smithi191 ceph-mon[98311]: osdmap e52: 8 total, 8 up, 8 in 2023-07-14T19:13:53.972 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:13:53 smithi040 ceph-mon[96518]: from='client.24298 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi191:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T19:13:53.972 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:13:53 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-14T19:13:53.972 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:13:53 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-14T19:13:53.972 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:13:53 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:13:53.973 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:13:53 smithi040 ceph-mon[96518]: osdmap e52: 8 total, 8 up, 8 in 2023-07-14T19:13:53.992 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:13:53 smithi130 ceph-mon[97832]: from='client.24298 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi191:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T19:13:53.992 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:13:53 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-14T19:13:53.993 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:13:53 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-14T19:13:53.993 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:13:53 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:13:53.993 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:13:53 smithi130 ceph-mon[97832]: osdmap e52: 8 total, 8 up, 8 in 2023-07-14T19:13:54.971 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:13:54 smithi040 ceph-mon[96518]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 215 MiB used, 715 GiB / 715 GiB avail 2023-07-14T19:13:54.972 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:13:54 smithi040 ceph-mon[96518]: osdmap e53: 8 total, 8 up, 8 in 2023-07-14T19:13:54.972 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:13:54 smithi191 ceph-mon[98311]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 215 MiB used, 715 GiB / 715 GiB avail 2023-07-14T19:13:54.972 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:13:54 smithi191 ceph-mon[98311]: osdmap e53: 8 total, 8 up, 8 in 2023-07-14T19:13:54.992 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:13:54 smithi130 ceph-mon[97832]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 215 MiB used, 715 GiB / 715 GiB avail 2023-07-14T19:13:54.992 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:13:54 smithi130 ceph-mon[97832]: osdmap e53: 8 total, 8 up, 8 in 2023-07-14T19:13:55.971 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:13:55 smithi040 ceph-mon[96518]: from='client.? 172.21.15.191:0/2426499566' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b76d4033-c1e2-457b-8571-c8a49fe443f0"}]: dispatch 2023-07-14T19:13:55.971 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:13:55 smithi040 ceph-mon[96518]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b76d4033-c1e2-457b-8571-c8a49fe443f0"}]: dispatch 2023-07-14T19:13:55.971 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:13:55 smithi040 ceph-mon[96518]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b76d4033-c1e2-457b-8571-c8a49fe443f0"}]': finished 2023-07-14T19:13:55.972 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:13:55 smithi040 ceph-mon[96518]: osdmap e54: 9 total, 8 up, 9 in 2023-07-14T19:13:55.972 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:13:55 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-07-14T19:13:55.972 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:13:55 smithi040 ceph-mon[96518]: from='client.? 172.21.15.191:0/1900894317' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-14T19:13:55.972 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:13:55 smithi191 ceph-mon[98311]: from='client.? 172.21.15.191:0/2426499566' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b76d4033-c1e2-457b-8571-c8a49fe443f0"}]: dispatch 2023-07-14T19:13:55.973 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:13:55 smithi191 ceph-mon[98311]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b76d4033-c1e2-457b-8571-c8a49fe443f0"}]: dispatch 2023-07-14T19:13:55.973 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:13:55 smithi191 ceph-mon[98311]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b76d4033-c1e2-457b-8571-c8a49fe443f0"}]': finished 2023-07-14T19:13:55.973 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:13:55 smithi191 ceph-mon[98311]: osdmap e54: 9 total, 8 up, 9 in 2023-07-14T19:13:55.973 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:13:55 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-07-14T19:13:55.973 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:13:55 smithi191 ceph-mon[98311]: from='client.? 172.21.15.191:0/1900894317' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-14T19:13:55.992 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:13:55 smithi130 ceph-mon[97832]: from='client.? 172.21.15.191:0/2426499566' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b76d4033-c1e2-457b-8571-c8a49fe443f0"}]: dispatch 2023-07-14T19:13:55.992 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:13:55 smithi130 ceph-mon[97832]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b76d4033-c1e2-457b-8571-c8a49fe443f0"}]: dispatch 2023-07-14T19:13:55.993 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:13:55 smithi130 ceph-mon[97832]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b76d4033-c1e2-457b-8571-c8a49fe443f0"}]': finished 2023-07-14T19:13:55.993 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:13:55 smithi130 ceph-mon[97832]: osdmap e54: 9 total, 8 up, 9 in 2023-07-14T19:13:55.993 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:13:55 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-07-14T19:13:55.993 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:13:55 smithi130 ceph-mon[97832]: from='client.? 172.21.15.191:0/1900894317' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-14T19:13:56.971 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:13:56 smithi040 ceph-mon[96518]: pgmap v165: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 215 MiB used, 715 GiB / 715 GiB avail 2023-07-14T19:13:56.972 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:13:56 smithi191 ceph-mon[98311]: pgmap v165: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 215 MiB used, 715 GiB / 715 GiB avail 2023-07-14T19:13:56.992 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:13:56 smithi130 ceph-mon[97832]: pgmap v165: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 215 MiB used, 715 GiB / 715 GiB avail 2023-07-14T19:13:58.971 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:13:58 smithi040 ceph-mon[96518]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 215 MiB used, 715 GiB / 715 GiB avail 2023-07-14T19:13:58.972 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:13:58 smithi191 ceph-mon[98311]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 215 MiB used, 715 GiB / 715 GiB avail 2023-07-14T19:13:58.992 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:13:58 smithi130 ceph-mon[97832]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 215 MiB used, 715 GiB / 715 GiB avail 2023-07-14T19:14:00.639 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:14:00 smithi191 ceph-mon[98311]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 215 MiB used, 715 GiB / 715 GiB avail 2023-07-14T19:14:00.971 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:14:00 smithi040 ceph-mon[96518]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 215 MiB used, 715 GiB / 715 GiB avail 2023-07-14T19:14:00.992 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:14:00 smithi130 ceph-mon[97832]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 215 MiB used, 715 GiB / 715 GiB avail 2023-07-14T19:14:01.722 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:14:01 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2023-07-14T19:14:01.722 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:14:01 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:14:01.971 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:14:01 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2023-07-14T19:14:01.972 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:14:01 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:14:01.992 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:14:01 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2023-07-14T19:14:01.992 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:14:01 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:14:02.668 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:14:02 smithi191 ceph-mon[98311]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 215 MiB used, 715 GiB / 715 GiB avail 2023-07-14T19:14:02.669 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:14:02 smithi191 ceph-mon[98311]: Deploying daemon osd.8 on smithi191 2023-07-14T19:14:02.971 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:14:02 smithi040 ceph-mon[96518]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 215 MiB used, 715 GiB / 715 GiB avail 2023-07-14T19:14:02.971 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:14:02 smithi040 ceph-mon[96518]: Deploying daemon osd.8 on smithi191 2023-07-14T19:14:02.992 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:14:02 smithi130 ceph-mon[97832]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 215 MiB used, 715 GiB / 715 GiB avail 2023-07-14T19:14:02.992 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:14:02 smithi130 ceph-mon[97832]: Deploying daemon osd.8 on smithi191 2023-07-14T19:14:04.722 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:14:04 smithi191 ceph-mon[98311]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 215 MiB used, 715 GiB / 715 GiB avail 2023-07-14T19:14:04.971 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:14:04 smithi040 ceph-mon[96518]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 215 MiB used, 715 GiB / 715 GiB avail 2023-07-14T19:14:04.992 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:14:04 smithi130 ceph-mon[97832]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 215 MiB used, 715 GiB / 715 GiB avail 2023-07-14T19:14:06.722 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:14:06 smithi191 ceph-mon[98311]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 215 MiB used, 715 GiB / 715 GiB avail 2023-07-14T19:14:06.722 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:14:06 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:14:06.722 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:14:06 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T19:14:06.722 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:14:06 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:14:06.971 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:14:06 smithi040 ceph-mon[96518]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 215 MiB used, 715 GiB / 715 GiB avail 2023-07-14T19:14:06.971 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:14:06 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:14:06.971 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:14:06 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T19:14:06.972 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:14:06 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:14:06.992 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:14:06 smithi130 ceph-mon[97832]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 215 MiB used, 715 GiB / 715 GiB avail 2023-07-14T19:14:06.993 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:14:06 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:14:06.993 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:14:06 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T19:14:06.993 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:14:06 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:14:08.755 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:14:08.754+0000 7f7e3d7fa700 1 -- 172.21.15.191:0/3492393589 <== mgr.14152 v2:172.21.15.40:6800/1693027118 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f7e40060a80 con 0x7f7e280a11d0 2023-07-14T19:14:08.756 INFO:teuthology.orchestra.run.smithi191.stdout:Created osd(s) 8 on host 'smithi191' 2023-07-14T19:14:08.758 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:14:08.757+0000 7f7e464ff700 1 -- 172.21.15.191:0/3492393589 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f7e280a11d0 msgr2=0x7f7e280a3680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:14:08.758 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:14:08.757+0000 7f7e464ff700 1 --2- 172.21.15.191:0/3492393589 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f7e280a11d0 0x7f7e280a3680 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f7e34006010 tx=0x7f7e3400b540 comp rx=0 tx=0).stop 2023-07-14T19:14:08.758 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:14:08.757+0000 7f7e464ff700 1 -- 172.21.15.191:0/3492393589 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7e400ffe70 msgr2=0x7f7e4019cb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:14:08.758 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:14:08.757+0000 7f7e464ff700 1 --2- 172.21.15.191:0/3492393589 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7e400ffe70 0x7f7e4019cb40 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f7e3000d950 tx=0x7f7e3000dd10 comp rx=0 tx=0).stop 2023-07-14T19:14:08.758 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:14:08.757+0000 7f7e464ff700 1 -- 172.21.15.191:0/3492393589 shutdown_connections 2023-07-14T19:14:08.758 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:14:08.757+0000 7f7e464ff700 1 --2- 172.21.15.191:0/3492393589 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f7e280a11d0 0x7f7e280a3680 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:14:08.759 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:14:08.757+0000 7f7e464ff700 1 --2- 172.21.15.191:0/3492393589 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7e40100820 0x7f7e4019f090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:14:08.759 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:14:08.757+0000 7f7e464ff700 1 --2- 172.21.15.191:0/3492393589 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7e400febd0 0x7f7e40069580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:14:08.759 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:14:08.757+0000 7f7e464ff700 1 --2- 172.21.15.191:0/3492393589 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7e400ffe70 0x7f7e4019cb40 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:14:08.759 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:14:08.757+0000 7f7e464ff700 1 -- 172.21.15.191:0/3492393589 >> 172.21.15.191:0/3492393589 conn(0x7f7e400fa240 msgr2=0x7f7e400fc490 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:14:08.759 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:14:08.757+0000 7f7e464ff700 1 -- 172.21.15.191:0/3492393589 shutdown_connections 2023-07-14T19:14:08.759 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:14:08.757+0000 7f7e464ff700 1 -- 172.21.15.191:0/3492393589 wait complete. 2023-07-14T19:14:08.915 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:14:08 smithi191 ceph-mon[98311]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 215 MiB used, 715 GiB / 715 GiB avail 2023-07-14T19:14:08.971 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:14:08 smithi040 ceph-mon[96518]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 215 MiB used, 715 GiB / 715 GiB avail 2023-07-14T19:14:08.992 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:14:08 smithi130 ceph-mon[97832]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 215 MiB used, 715 GiB / 715 GiB avail 2023-07-14T19:14:09.175 DEBUG:teuthology.orchestra.run.smithi191:osd.8> sudo journalctl -f -n 0 -u ceph-adc52f04-2279-11ee-9b33-001a4aab830c@osd.8.service 2023-07-14T19:14:09.177 INFO:tasks.cephadm:Deploying osd.9 on smithi040 with /dev/vg_nvme/lv_1... 2023-07-14T19:14:09.178 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid adc52f04-2279-11ee-9b33-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2023-07-14T19:14:09.213 INFO:journalctl@ceph.osd.8.smithi191.stdout:-- Logs begin at Fri 2023-07-14 18:50:54 UTC. -- 2023-07-14T19:14:09.749 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/adc52f04-2279-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T19:14:09.971 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:14:09 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:14:09.971 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:14:09 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:14:09.971 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:14:09 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:14:09.971 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:14:09 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:14:09.972 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:14:09 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:14:09.972 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:14:09 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:14:09.972 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:14:09 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:14:09.972 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:14:09 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:14:09.973 INFO:journalctl@ceph.osd.8.smithi191.stdout:Jul 14 19:14:09 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-8[115063]: 2023-07-14T19:14:09.916+0000 7f430c8f6640 -1 osd.8 0 log_to_monitors true 2023-07-14T19:14:09.992 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:14:09 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:14:09.992 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:14:09 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:14:09.993 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:14:09 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:14:09.993 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:14:09 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:14:10.972 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:14:10 smithi191 ceph-mon[98311]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 215 MiB used, 715 GiB / 715 GiB avail 2023-07-14T19:14:10.973 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:14:10 smithi191 ceph-mon[98311]: from='osd.8 [v2:172.21.15.191:6816/2386818417,v1:172.21.15.191:6817/2386818417]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2023-07-14T19:14:10.973 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:14:10 smithi191 ceph-mon[98311]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2023-07-14T19:14:10.992 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:14:10 smithi130 ceph-mon[97832]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 215 MiB used, 715 GiB / 715 GiB avail 2023-07-14T19:14:10.992 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:14:10 smithi130 ceph-mon[97832]: from='osd.8 [v2:172.21.15.191:6816/2386818417,v1:172.21.15.191:6817/2386818417]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2023-07-14T19:14:10.992 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:14:10 smithi130 ceph-mon[97832]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2023-07-14T19:14:11.221 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:14:10 smithi040 ceph-mon[96518]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 215 MiB used, 715 GiB / 715 GiB avail 2023-07-14T19:14:11.221 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:14:10 smithi040 ceph-mon[96518]: from='osd.8 [v2:172.21.15.191:6816/2386818417,v1:172.21.15.191:6817/2386818417]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2023-07-14T19:14:11.221 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:14:10 smithi040 ceph-mon[96518]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2023-07-14T19:14:11.396 INFO:teuthology.orchestra.run.smithi040.stderr:--> Zapping: /dev/vg_nvme/lv_1 2023-07-14T19:14:11.396 INFO:teuthology.orchestra.run.smithi040.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2023-07-14T19:14:11.397 INFO:teuthology.orchestra.run.smithi040.stderr: stderr: 10+0 records in 2023-07-14T19:14:11.397 INFO:teuthology.orchestra.run.smithi040.stderr:10+0 records out 2023-07-14T19:14:11.397 INFO:teuthology.orchestra.run.smithi040.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0105995 s, 989 MB/s 2023-07-14T19:14:11.397 INFO:teuthology.orchestra.run.smithi040.stderr:--> Zapping successful for: 2023-07-14T19:14:11.808 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid adc52f04-2279-11ee-9b33-001a4aab830c -- ceph orch daemon add osd smithi040:vg_nvme/lv_1 2023-07-14T19:14:12.221 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:14:11 smithi040 ceph-mon[96518]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2023-07-14T19:14:12.222 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:14:11 smithi040 ceph-mon[96518]: osdmap e55: 9 total, 8 up, 9 in 2023-07-14T19:14:12.222 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:14:11 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-07-14T19:14:12.222 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:14:11 smithi040 ceph-mon[96518]: from='osd.8 [v2:172.21.15.191:6816/2386818417,v1:172.21.15.191:6817/2386818417]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi191", "root=default"]}]: dispatch 2023-07-14T19:14:12.222 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:14:11 smithi040 ceph-mon[96518]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi191", "root=default"]}]: dispatch 2023-07-14T19:14:12.222 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:14:11 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:14:12.222 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:14:11 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:14:12.223 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:14:11 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-07-14T19:14:12.223 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:14:11 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-07-14T19:14:12.223 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:14:11 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-07-14T19:14:12.223 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:14:11 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:14:12.223 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:14:11 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:14:12.223 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:14:11 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T19:14:12.223 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:14:11 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:14:12.224 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:14:11 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T19:14:12.224 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:14:11 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:14:12.224 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:14:11 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T19:14:12.224 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:14:11 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:14:12.225 INFO:journalctl@ceph.osd.8.smithi191.stdout:Jul 14 19:14:11 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-8[115063]: 2023-07-14T19:14:11.748+0000 7f430276f700 -1 osd.8 0 waiting for initial osdmap 2023-07-14T19:14:12.225 INFO:journalctl@ceph.osd.8.smithi191.stdout:Jul 14 19:14:11 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-8[115063]: 2023-07-14T19:14:11.758+0000 7f42fc561700 -1 osd.8 56 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-07-14T19:14:12.225 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:14:11 smithi191 ceph-mon[98311]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2023-07-14T19:14:12.225 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:14:11 smithi191 ceph-mon[98311]: osdmap e55: 9 total, 8 up, 9 in 2023-07-14T19:14:12.225 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:14:11 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-07-14T19:14:12.225 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:14:11 smithi191 ceph-mon[98311]: from='osd.8 [v2:172.21.15.191:6816/2386818417,v1:172.21.15.191:6817/2386818417]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi191", "root=default"]}]: dispatch 2023-07-14T19:14:12.226 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:14:11 smithi191 ceph-mon[98311]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi191", "root=default"]}]: dispatch 2023-07-14T19:14:12.226 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:14:11 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:14:12.226 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:14:11 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:14:12.226 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:14:11 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-07-14T19:14:12.226 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:14:11 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-07-14T19:14:12.226 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:14:11 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-07-14T19:14:12.226 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:14:11 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:14:12.227 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:14:11 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:14:12.227 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:14:11 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T19:14:12.227 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:14:11 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:14:12.227 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:14:11 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T19:14:12.227 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:14:11 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:14:12.227 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:14:11 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T19:14:12.227 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:14:11 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:14:12.242 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:14:11 smithi130 ceph-mon[97832]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2023-07-14T19:14:12.242 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:14:11 smithi130 ceph-mon[97832]: osdmap e55: 9 total, 8 up, 9 in 2023-07-14T19:14:12.243 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:14:11 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-07-14T19:14:12.243 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:14:11 smithi130 ceph-mon[97832]: from='osd.8 [v2:172.21.15.191:6816/2386818417,v1:172.21.15.191:6817/2386818417]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi191", "root=default"]}]: dispatch 2023-07-14T19:14:12.243 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:14:11 smithi130 ceph-mon[97832]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi191", "root=default"]}]: dispatch 2023-07-14T19:14:12.243 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:14:11 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:14:12.243 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:14:11 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:14:12.243 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:14:11 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-07-14T19:14:12.243 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:14:11 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-07-14T19:14:12.244 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:14:11 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-07-14T19:14:12.244 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:14:11 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:14:12.244 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:14:11 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:14:12.244 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:14:11 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T19:14:12.244 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:14:11 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:14:12.244 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:14:11 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T19:14:12.245 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:14:11 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:14:12.245 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:14:11 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T19:14:12.245 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:14:11 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:14:12.368 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/adc52f04-2279-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T19:14:12.861 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:14:12 smithi040 ceph-mon[96518]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 215 MiB used, 715 GiB / 715 GiB avail 2023-07-14T19:14:12.861 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:14:12 smithi040 ceph-mon[96518]: Detected new or changed devices on smithi191 2023-07-14T19:14:12.861 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:14:12 smithi040 ceph-mon[96518]: Adjusting osd_memory_target on smithi191 to 5722M 2023-07-14T19:14:12.862 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:14:12 smithi040 ceph-mon[96518]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi191", "root=default"]}]': finished 2023-07-14T19:14:12.862 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:14:12 smithi040 ceph-mon[96518]: osdmap e56: 9 total, 8 up, 9 in 2023-07-14T19:14:12.862 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:14:12 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-07-14T19:14:12.862 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:14:12 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-07-14T19:14:13.222 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:14:12 smithi191 ceph-mon[98311]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 215 MiB used, 715 GiB / 715 GiB avail 2023-07-14T19:14:13.222 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:14:12 smithi191 ceph-mon[98311]: Detected new or changed devices on smithi191 2023-07-14T19:14:13.222 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:14:12 smithi191 ceph-mon[98311]: Adjusting osd_memory_target on smithi191 to 5722M 2023-07-14T19:14:13.223 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:14:12 smithi191 ceph-mon[98311]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi191", "root=default"]}]': finished 2023-07-14T19:14:13.223 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:14:12 smithi191 ceph-mon[98311]: osdmap e56: 9 total, 8 up, 9 in 2023-07-14T19:14:13.223 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:14:12 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-07-14T19:14:13.223 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:14:12 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-07-14T19:14:13.242 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:14:12 smithi130 ceph-mon[97832]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 215 MiB used, 715 GiB / 715 GiB avail 2023-07-14T19:14:13.242 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:14:12 smithi130 ceph-mon[97832]: Detected new or changed devices on smithi191 2023-07-14T19:14:13.243 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:14:12 smithi130 ceph-mon[97832]: Adjusting osd_memory_target on smithi191 to 5722M 2023-07-14T19:14:13.243 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:14:12 smithi130 ceph-mon[97832]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi191", "root=default"]}]': finished 2023-07-14T19:14:13.243 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:14:12 smithi130 ceph-mon[97832]: osdmap e56: 9 total, 8 up, 9 in 2023-07-14T19:14:13.243 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:14:12 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-07-14T19:14:13.243 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:14:12 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-07-14T19:14:13.571 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:14:13.568+0000 7ff99559b700 1 -- 172.21.15.40:0/310487552 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff9900fec30 msgr2=0x7ff9900ff040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:14:13.571 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:14:13.568+0000 7ff99559b700 1 --2- 172.21.15.40:0/310487552 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff9900fec30 0x7ff9900ff040 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7ff980009990 tx=0x7ff980009ca0 comp rx=0 tx=0).stop 2023-07-14T19:14:13.571 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:14:13.568+0000 7ff99559b700 1 -- 172.21.15.40:0/310487552 shutdown_connections 2023-07-14T19:14:13.571 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:14:13.568+0000 7ff99559b700 1 --2- 172.21.15.40:0/310487552 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff9901007c0 0x7ff990107080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:14:13.571 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:14:13.568+0000 7ff99559b700 1 --2- 172.21.15.40:0/310487552 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff9900fec30 0x7ff9900ff040 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:14:13.572 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:14:13.568+0000 7ff99559b700 1 --2- 172.21.15.40:0/310487552 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff9900ffe30 0x7ff990100280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:14:13.572 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:14:13.568+0000 7ff99559b700 1 -- 172.21.15.40:0/310487552 >> 172.21.15.40:0/310487552 conn(0x7ff9900fa260 msgr2=0x7ff9900fc6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:14:13.572 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:14:13.568+0000 7ff99559b700 1 -- 172.21.15.40:0/310487552 shutdown_connections 2023-07-14T19:14:13.572 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:14:13.568+0000 7ff99559b700 1 -- 172.21.15.40:0/310487552 wait complete. 2023-07-14T19:14:13.572 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:14:13.568+0000 7ff99559b700 1 Processor -- start 2023-07-14T19:14:13.573 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:14:13.568+0000 7ff99559b700 1 -- start start 2023-07-14T19:14:13.573 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:14:13.569+0000 7ff99559b700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff9900fec30 0x7ff990069620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:14:13.573 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:14:13.569+0000 7ff99559b700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff9900ffe30 0x7ff99019a710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:14:13.573 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:14:13.569+0000 7ff99559b700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff9901007c0 0x7ff99019cd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:14:13.573 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:14:13.569+0000 7ff99559b700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff99019d4f0 con 0x7ff9900fec30 2023-07-14T19:14:13.573 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:14:13.569+0000 7ff99559b700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff99019d660 con 0x7ff9900ffe30 2023-07-14T19:14:13.573 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:14:13.569+0000 7ff99559b700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff99019d820 con 0x7ff9901007c0 2023-07-14T19:14:13.574 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:14:13.569+0000 7ff98effd700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff9900fec30 0x7ff990069620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:14:13.574 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:14:13.569+0000 7ff98effd700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff9900fec30 0x7ff990069620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:60532/0 (socket says 172.21.15.40:60532) 2023-07-14T19:14:13.574 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:14:13.569+0000 7ff98effd700 1 -- 172.21.15.40:0/4018298213 learned_addr learned my addr 172.21.15.40:0/4018298213 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-07-14T19:14:13.574 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:14:13.569+0000 7ff98e7fc700 1 --2- 172.21.15.40:0/4018298213 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff9900ffe30 0x7ff99019a710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:14:13.575 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:14:13.569+0000 7ff98effd700 1 -- 172.21.15.40:0/4018298213 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff9901007c0 msgr2=0x7ff99019cd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:14:13.575 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:14:13.569+0000 7ff98f7fe700 1 --2- 172.21.15.40:0/4018298213 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff9901007c0 0x7ff99019cd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:14:13.575 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:14:13.569+0000 7ff98effd700 1 --2- 172.21.15.40:0/4018298213 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff9901007c0 0x7ff99019cd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:14:13.575 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:14:13.569+0000 7ff98effd700 1 -- 172.21.15.40:0/4018298213 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff9900ffe30 msgr2=0x7ff99019a710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:14:13.575 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:14:13.569+0000 7ff98effd700 1 --2- 172.21.15.40:0/4018298213 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff9900ffe30 0x7ff99019a710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:14:13.575 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:14:13.569+0000 7ff98effd700 1 -- 172.21.15.40:0/4018298213 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9800095f0 con 0x7ff9900fec30 2023-07-14T19:14:13.575 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:14:13.569+0000 7ff98e7fc700 1 --2- 172.21.15.40:0/4018298213 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff9900ffe30 0x7ff99019a710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T19:14:13.576 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:14:13.569+0000 7ff98f7fe700 1 --2- 172.21.15.40:0/4018298213 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff9901007c0 0x7ff99019cd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-14T19:14:13.576 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:14:13.569+0000 7ff98effd700 1 --2- 172.21.15.40:0/4018298213 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff9900fec30 0x7ff990069620 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7ff9800095c0 tx=0x7ff98000f690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:14:13.576 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:14:13.569+0000 7ff977fff700 1 -- 172.21.15.40:0/4018298213 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff98001d070 con 0x7ff9900fec30 2023-07-14T19:14:13.576 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:14:13.569+0000 7ff977fff700 1 -- 172.21.15.40:0/4018298213 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff98000fb80 con 0x7ff9900fec30 2023-07-14T19:14:13.576 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:14:13.569+0000 7ff99559b700 1 -- 172.21.15.40:0/4018298213 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff99019daa0 con 0x7ff9900fec30 2023-07-14T19:14:13.576 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:14:13.569+0000 7ff977fff700 1 -- 172.21.15.40:0/4018298213 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff980017650 con 0x7ff9900fec30 2023-07-14T19:14:13.577 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:14:13.570+0000 7ff99559b700 1 -- 172.21.15.40:0/4018298213 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff99019deb0 con 0x7ff9900fec30 2023-07-14T19:14:13.577 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:14:13.570+0000 7ff99559b700 1 -- 172.21.15.40:0/4018298213 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff990106280 con 0x7ff9900fec30 2023-07-14T19:14:13.578 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:14:13.572+0000 7ff977fff700 1 -- 172.21.15.40:0/4018298213 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7ff980022470 con 0x7ff9900fec30 2023-07-14T19:14:13.578 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:14:13.572+0000 7ff977fff700 1 --2- 172.21.15.40:0/4018298213 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7ff9780a1150 0x7ff9780a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:14:13.578 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:14:13.572+0000 7ff98e7fc700 1 --2- 172.21.15.40:0/4018298213 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7ff9780a1150 0x7ff9780a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:14:13.578 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:14:13.572+0000 7ff977fff700 1 -- 172.21.15.40:0/4018298213 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 5862+0+0 (secure 0 0 0) 0x7ff9800cdc20 con 0x7ff9900fec30 2023-07-14T19:14:13.578 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:14:13.573+0000 7ff98e7fc700 1 --2- 172.21.15.40:0/4018298213 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7ff9780a1150 0x7ff9780a3600 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7ff984005950 tx=0x7ff9840058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:14:13.578 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:14:13.576+0000 7ff977fff700 1 -- 172.21.15.40:0/4018298213 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7ff980097d90 con 0x7ff9900fec30 2023-07-14T19:14:13.783 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:14:13.782+0000 7ff99559b700 1 -- 172.21.15.40:0/4018298213 --> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi040:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) v1 -- 0x7ff990060a80 con 0x7ff9780a1150 2023-07-14T19:14:14.221 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:14:13 smithi040 ceph-mon[96518]: purged_snaps scrub starts 2023-07-14T19:14:14.222 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:14:13 smithi040 ceph-mon[96518]: purged_snaps scrub ok 2023-07-14T19:14:14.222 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:14:13 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-07-14T19:14:14.222 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:14:13 smithi040 ceph-mon[96518]: osd.8 [v2:172.21.15.191:6816/2386818417,v1:172.21.15.191:6817/2386818417] boot 2023-07-14T19:14:14.222 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:14:13 smithi040 ceph-mon[96518]: osdmap e57: 9 total, 9 up, 9 in 2023-07-14T19:14:14.222 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:14:13 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-07-14T19:14:14.223 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:14:13 smithi191 ceph-mon[98311]: purged_snaps scrub starts 2023-07-14T19:14:14.223 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:14:13 smithi191 ceph-mon[98311]: purged_snaps scrub ok 2023-07-14T19:14:14.223 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:14:13 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-07-14T19:14:14.223 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:14:13 smithi191 ceph-mon[98311]: osd.8 [v2:172.21.15.191:6816/2386818417,v1:172.21.15.191:6817/2386818417] boot 2023-07-14T19:14:14.223 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:14:13 smithi191 ceph-mon[98311]: osdmap e57: 9 total, 9 up, 9 in 2023-07-14T19:14:14.223 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:14:13 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-07-14T19:14:14.242 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:14:13 smithi130 ceph-mon[97832]: purged_snaps scrub starts 2023-07-14T19:14:14.242 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:14:13 smithi130 ceph-mon[97832]: purged_snaps scrub ok 2023-07-14T19:14:14.242 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:14:13 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-07-14T19:14:14.243 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:14:13 smithi130 ceph-mon[97832]: osd.8 [v2:172.21.15.191:6816/2386818417,v1:172.21.15.191:6817/2386818417] boot 2023-07-14T19:14:14.243 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:14:13 smithi130 ceph-mon[97832]: osdmap e57: 9 total, 9 up, 9 in 2023-07-14T19:14:14.243 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:14:13 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-07-14T19:14:14.928 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:14:14 smithi040 ceph-mon[96518]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 215 MiB used, 715 GiB / 715 GiB avail 2023-07-14T19:14:14.928 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:14:14 smithi040 ceph-mon[96518]: osdmap e58: 9 total, 9 up, 9 in 2023-07-14T19:14:14.928 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:14:14 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-14T19:14:14.929 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:14:14 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-14T19:14:14.929 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:14:14 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:14:15.222 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:14:14 smithi191 ceph-mon[98311]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 215 MiB used, 715 GiB / 715 GiB avail 2023-07-14T19:14:15.222 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:14:14 smithi191 ceph-mon[98311]: osdmap e58: 9 total, 9 up, 9 in 2023-07-14T19:14:15.222 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:14:14 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-14T19:14:15.223 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:14:14 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-14T19:14:15.223 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:14:14 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:14:15.242 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:14:14 smithi130 ceph-mon[97832]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 215 MiB used, 715 GiB / 715 GiB avail 2023-07-14T19:14:15.242 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:14:14 smithi130 ceph-mon[97832]: osdmap e58: 9 total, 9 up, 9 in 2023-07-14T19:14:15.243 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:14:14 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-14T19:14:15.243 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:14:14 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-14T19:14:15.243 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:14:14 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:14:15.971 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:14:15 smithi040 ceph-mon[96518]: from='client.14406 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi040:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T19:14:15.971 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:14:15 smithi040 ceph-mon[96518]: osdmap e59: 9 total, 9 up, 9 in 2023-07-14T19:14:16.222 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:14:15 smithi191 ceph-mon[98311]: from='client.14406 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi040:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T19:14:16.222 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:14:15 smithi191 ceph-mon[98311]: osdmap e59: 9 total, 9 up, 9 in 2023-07-14T19:14:16.242 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:14:15 smithi130 ceph-mon[97832]: from='client.14406 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi040:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T19:14:16.242 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:14:15 smithi130 ceph-mon[97832]: osdmap e59: 9 total, 9 up, 9 in 2023-07-14T19:14:17.221 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:14:16 smithi040 ceph-mon[96518]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 242 MiB used, 804 GiB / 805 GiB avail 2023-07-14T19:14:17.222 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:14:16 smithi040 ceph-mon[96518]: osdmap e60: 9 total, 9 up, 9 in 2023-07-14T19:14:17.222 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:14:16 smithi040 ceph-mon[96518]: from='client.? 172.21.15.40:0/906218234' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "df31bf2f-e30e-41b6-b2b9-0e26a07a88bd"}]: dispatch 2023-07-14T19:14:17.222 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:14:16 smithi040 ceph-mon[96518]: from='client.? 172.21.15.40:0/906218234' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "df31bf2f-e30e-41b6-b2b9-0e26a07a88bd"}]': finished 2023-07-14T19:14:17.222 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:14:16 smithi040 ceph-mon[96518]: osdmap e61: 10 total, 9 up, 10 in 2023-07-14T19:14:17.222 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:14:16 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-07-14T19:14:17.223 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:14:16 smithi191 ceph-mon[98311]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 242 MiB used, 804 GiB / 805 GiB avail 2023-07-14T19:14:17.223 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:14:16 smithi191 ceph-mon[98311]: osdmap e60: 9 total, 9 up, 9 in 2023-07-14T19:14:17.223 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:14:16 smithi191 ceph-mon[98311]: from='client.? 172.21.15.40:0/906218234' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "df31bf2f-e30e-41b6-b2b9-0e26a07a88bd"}]: dispatch 2023-07-14T19:14:17.223 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:14:16 smithi191 ceph-mon[98311]: from='client.? 172.21.15.40:0/906218234' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "df31bf2f-e30e-41b6-b2b9-0e26a07a88bd"}]': finished 2023-07-14T19:14:17.223 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:14:16 smithi191 ceph-mon[98311]: osdmap e61: 10 total, 9 up, 10 in 2023-07-14T19:14:17.223 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:14:16 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-07-14T19:14:17.242 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:14:16 smithi130 ceph-mon[97832]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 242 MiB used, 804 GiB / 805 GiB avail 2023-07-14T19:14:17.242 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:14:16 smithi130 ceph-mon[97832]: osdmap e60: 9 total, 9 up, 9 in 2023-07-14T19:14:17.242 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:14:16 smithi130 ceph-mon[97832]: from='client.? 172.21.15.40:0/906218234' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "df31bf2f-e30e-41b6-b2b9-0e26a07a88bd"}]: dispatch 2023-07-14T19:14:17.242 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:14:16 smithi130 ceph-mon[97832]: from='client.? 172.21.15.40:0/906218234' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "df31bf2f-e30e-41b6-b2b9-0e26a07a88bd"}]': finished 2023-07-14T19:14:17.243 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:14:16 smithi130 ceph-mon[97832]: osdmap e61: 10 total, 9 up, 10 in 2023-07-14T19:14:17.243 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:14:16 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-07-14T19:14:18.221 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:14:17 smithi040 ceph-mon[96518]: from='client.? 172.21.15.40:0/3438515922' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-14T19:14:18.222 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:14:17 smithi191 ceph-mon[98311]: from='client.? 172.21.15.40:0/3438515922' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-14T19:14:18.242 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:14:17 smithi130 ceph-mon[97832]: from='client.? 172.21.15.40:0/3438515922' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-14T19:14:19.221 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:14:18 smithi040 ceph-mon[96518]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 243 MiB used, 804 GiB / 805 GiB avail; 144 KiB/s, 0 objects/s recovering 2023-07-14T19:14:19.222 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:14:18 smithi191 ceph-mon[98311]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 243 MiB used, 804 GiB / 805 GiB avail; 144 KiB/s, 0 objects/s recovering 2023-07-14T19:14:19.242 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:14:18 smithi130 ceph-mon[97832]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 243 MiB used, 804 GiB / 805 GiB avail; 144 KiB/s, 0 objects/s recovering 2023-07-14T19:14:21.221 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:14:20 smithi040 ceph-mon[96518]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 243 MiB used, 804 GiB / 805 GiB avail; 115 KiB/s, 0 objects/s recovering 2023-07-14T19:14:21.222 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:14:20 smithi191 ceph-mon[98311]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 243 MiB used, 804 GiB / 805 GiB avail; 115 KiB/s, 0 objects/s recovering 2023-07-14T19:14:21.242 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:14:20 smithi130 ceph-mon[97832]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 243 MiB used, 804 GiB / 805 GiB avail; 115 KiB/s, 0 objects/s recovering 2023-07-14T19:14:23.221 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:14:22 smithi040 ceph-mon[96518]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 243 MiB used, 804 GiB / 805 GiB avail; 96 KiB/s, 0 objects/s recovering 2023-07-14T19:14:23.222 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:14:22 smithi191 ceph-mon[98311]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 243 MiB used, 804 GiB / 805 GiB avail; 96 KiB/s, 0 objects/s recovering 2023-07-14T19:14:23.242 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:14:22 smithi130 ceph-mon[97832]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 243 MiB used, 804 GiB / 805 GiB avail; 96 KiB/s, 0 objects/s recovering 2023-07-14T19:14:24.074 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:14:23 smithi040 ceph-mon[96518]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 243 MiB used, 804 GiB / 805 GiB avail; 72 KiB/s, 0 objects/s recovering 2023-07-14T19:14:24.074 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:14:23 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2023-07-14T19:14:24.074 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:14:23 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:14:24.074 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:14:23 smithi040 ceph-mon[96518]: Deploying daemon osd.9 on smithi040 2023-07-14T19:14:24.222 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:14:23 smithi191 ceph-mon[98311]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 243 MiB used, 804 GiB / 805 GiB avail; 72 KiB/s, 0 objects/s recovering 2023-07-14T19:14:24.222 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:14:23 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2023-07-14T19:14:24.222 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:14:23 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:14:24.222 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:14:23 smithi191 ceph-mon[98311]: Deploying daemon osd.9 on smithi040 2023-07-14T19:14:24.242 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:14:23 smithi130 ceph-mon[97832]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 243 MiB used, 804 GiB / 805 GiB avail; 72 KiB/s, 0 objects/s recovering 2023-07-14T19:14:24.242 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:14:23 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2023-07-14T19:14:24.243 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:14:23 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:14:24.243 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:14:23 smithi130 ceph-mon[97832]: Deploying daemon osd.9 on smithi040 2023-07-14T19:14:26.081 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:14:25 smithi040 ceph-mon[96518]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 243 MiB used, 804 GiB / 805 GiB avail; 64 KiB/s, 0 objects/s recovering 2023-07-14T19:14:26.222 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:14:25 smithi191 ceph-mon[98311]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 243 MiB used, 804 GiB / 805 GiB avail; 64 KiB/s, 0 objects/s recovering 2023-07-14T19:14:26.242 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:14:25 smithi130 ceph-mon[97832]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 243 MiB used, 804 GiB / 805 GiB avail; 64 KiB/s, 0 objects/s recovering 2023-07-14T19:14:28.534 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:14:28 smithi040 ceph-mon[96518]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 243 MiB used, 804 GiB / 805 GiB avail; 55 KiB/s, 0 objects/s recovering 2023-07-14T19:14:28.535 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:14:28 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:14:28.535 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:14:28 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:14:28.535 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:14:28 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T19:14:28.722 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:14:28 smithi191 ceph-mon[98311]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 243 MiB used, 804 GiB / 805 GiB avail; 55 KiB/s, 0 objects/s recovering 2023-07-14T19:14:28.722 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:14:28 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:14:28.723 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:14:28 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:14:28.723 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:14:28 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T19:14:28.742 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:14:28 smithi130 ceph-mon[97832]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 243 MiB used, 804 GiB / 805 GiB avail; 55 KiB/s, 0 objects/s recovering 2023-07-14T19:14:28.742 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:14:28 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:14:28.743 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:14:28 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:14:28.743 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:14:28 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T19:14:30.144 INFO:teuthology.orchestra.run.smithi040.stdout:Created osd(s) 9 on host 'smithi040' 2023-07-14T19:14:30.145 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:14:30.142+0000 7ff977fff700 1 -- 172.21.15.40:0/4018298213 <== mgr.14152 v2:172.21.15.40:6800/1693027118 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7ff990060a80 con 0x7ff9780a1150 2023-07-14T19:14:30.147 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:14:30.145+0000 7ff99559b700 1 -- 172.21.15.40:0/4018298213 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7ff9780a1150 msgr2=0x7ff9780a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:14:30.147 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:14:30.145+0000 7ff99559b700 1 --2- 172.21.15.40:0/4018298213 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7ff9780a1150 0x7ff9780a3600 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7ff984005950 tx=0x7ff9840058e0 comp rx=0 tx=0).stop 2023-07-14T19:14:30.147 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:14:30.145+0000 7ff99559b700 1 -- 172.21.15.40:0/4018298213 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff9900fec30 msgr2=0x7ff990069620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:14:30.147 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:14:30.145+0000 7ff99559b700 1 --2- 172.21.15.40:0/4018298213 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff9900fec30 0x7ff990069620 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7ff9800095c0 tx=0x7ff98000f690 comp rx=0 tx=0).stop 2023-07-14T19:14:30.147 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:14:30.145+0000 7ff99559b700 1 -- 172.21.15.40:0/4018298213 shutdown_connections 2023-07-14T19:14:30.147 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:14:30.145+0000 7ff99559b700 1 --2- 172.21.15.40:0/4018298213 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7ff9780a1150 0x7ff9780a3600 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:14:30.148 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:14:30.145+0000 7ff99559b700 1 --2- 172.21.15.40:0/4018298213 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff9901007c0 0x7ff99019cd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:14:30.148 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:14:30.145+0000 7ff99559b700 1 --2- 172.21.15.40:0/4018298213 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff9900fec30 0x7ff990069620 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:14:30.148 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:14:30.145+0000 7ff99559b700 1 --2- 172.21.15.40:0/4018298213 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff9900ffe30 0x7ff99019a710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:14:30.148 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:14:30.145+0000 7ff99559b700 1 -- 172.21.15.40:0/4018298213 >> 172.21.15.40:0/4018298213 conn(0x7ff9900fa260 msgr2=0x7ff9901033c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:14:30.148 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:14:30.145+0000 7ff99559b700 1 -- 172.21.15.40:0/4018298213 shutdown_connections 2023-07-14T19:14:30.148 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:14:30.145+0000 7ff99559b700 1 -- 172.21.15.40:0/4018298213 wait complete. 2023-07-14T19:14:30.555 DEBUG:teuthology.orchestra.run.smithi040:osd.9> sudo journalctl -f -n 0 -u ceph-adc52f04-2279-11ee-9b33-001a4aab830c@osd.9.service 2023-07-14T19:14:30.558 INFO:tasks.cephadm:Deploying osd.10 on smithi130 with /dev/vg_nvme/lv_1... 2023-07-14T19:14:30.558 DEBUG:teuthology.orchestra.run.smithi130:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid adc52f04-2279-11ee-9b33-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2023-07-14T19:14:30.588 INFO:journalctl@ceph.osd.9.smithi040.stdout:-- Logs begin at Fri 2023-07-14 18:50:26 UTC. -- 2023-07-14T19:14:30.721 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:14:30 smithi040 ceph-mon[96518]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 243 MiB used, 804 GiB / 805 GiB avail 2023-07-14T19:14:30.721 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:14:30 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:14:30.722 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:14:30 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:14:30.722 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:14:30 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:14:30.722 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:14:30 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:14:30.723 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:14:30 smithi191 ceph-mon[98311]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 243 MiB used, 804 GiB / 805 GiB avail 2023-07-14T19:14:30.723 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:14:30 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:14:30.723 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:14:30 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:14:30.723 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:14:30 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:14:30.723 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:14:30 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:14:30.742 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:14:30 smithi130 ceph-mon[97832]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 243 MiB used, 804 GiB / 805 GiB avail 2023-07-14T19:14:30.742 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:14:30 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:14:30.743 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:14:30 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:14:30.743 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:14:30 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:14:30.743 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:14:30 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:14:31.124 INFO:teuthology.orchestra.run.smithi130.stderr:Inferring config /var/lib/ceph/adc52f04-2279-11ee-9b33-001a4aab830c/mon.b/config 2023-07-14T19:14:31.427 INFO:journalctl@ceph.osd.9.smithi040.stdout:Jul 14 19:14:31 smithi040 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-9[125889]: 2023-07-14T19:14:31.396+0000 7fa2aa9b4640 -1 osd.9 0 log_to_monitors true 2023-07-14T19:14:31.427 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:14:31 smithi040 ceph-mon[96518]: from='osd.9 [v2:172.21.15.40:6826/2228380263,v1:172.21.15.40:6827/2228380263]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2023-07-14T19:14:31.659 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:14:31 smithi130 ceph-mon[97832]: from='osd.9 [v2:172.21.15.40:6826/2228380263,v1:172.21.15.40:6827/2228380263]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2023-07-14T19:14:31.722 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:14:31 smithi191 ceph-mon[98311]: from='osd.9 [v2:172.21.15.40:6826/2228380263,v1:172.21.15.40:6827/2228380263]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2023-07-14T19:14:32.585 INFO:teuthology.orchestra.run.smithi130.stderr:--> Zapping: /dev/vg_nvme/lv_1 2023-07-14T19:14:32.585 INFO:teuthology.orchestra.run.smithi130.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2023-07-14T19:14:32.585 INFO:teuthology.orchestra.run.smithi130.stderr: stderr: 10+0 records in 2023-07-14T19:14:32.585 INFO:teuthology.orchestra.run.smithi130.stderr:10+0 records out 2023-07-14T19:14:32.586 INFO:teuthology.orchestra.run.smithi130.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0113445 s, 924 MB/s 2023-07-14T19:14:32.586 INFO:teuthology.orchestra.run.smithi130.stderr:--> Zapping successful for: 2023-07-14T19:14:32.717 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:14:32 smithi130 ceph-mon[97832]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 243 MiB used, 804 GiB / 805 GiB avail 2023-07-14T19:14:32.718 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:14:32 smithi130 ceph-mon[97832]: from='osd.9 [v2:172.21.15.40:6826/2228380263,v1:172.21.15.40:6827/2228380263]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2023-07-14T19:14:32.718 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:14:32 smithi130 ceph-mon[97832]: osdmap e62: 10 total, 9 up, 10 in 2023-07-14T19:14:32.718 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:14:32 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-07-14T19:14:32.718 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:14:32 smithi130 ceph-mon[97832]: from='osd.9 [v2:172.21.15.40:6826/2228380263,v1:172.21.15.40:6827/2228380263]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi040", "root=default"]}]: dispatch 2023-07-14T19:14:32.721 INFO:journalctl@ceph.osd.9.smithi040.stdout:Jul 14 19:14:32 smithi040 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-9[125889]: 2023-07-14T19:14:32.430+0000 7fa2a002c700 -1 osd.9 0 waiting for initial osdmap 2023-07-14T19:14:32.721 INFO:journalctl@ceph.osd.9.smithi040.stdout:Jul 14 19:14:32 smithi040 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-9[125889]: 2023-07-14T19:14:32.441+0000 7fa29be22700 -1 osd.9 63 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-07-14T19:14:32.721 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:14:32 smithi040 ceph-mon[96518]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 243 MiB used, 804 GiB / 805 GiB avail 2023-07-14T19:14:32.722 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:14:32 smithi040 ceph-mon[96518]: from='osd.9 [v2:172.21.15.40:6826/2228380263,v1:172.21.15.40:6827/2228380263]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2023-07-14T19:14:32.722 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:14:32 smithi040 ceph-mon[96518]: osdmap e62: 10 total, 9 up, 10 in 2023-07-14T19:14:32.722 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:14:32 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-07-14T19:14:32.722 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:14:32 smithi040 ceph-mon[96518]: from='osd.9 [v2:172.21.15.40:6826/2228380263,v1:172.21.15.40:6827/2228380263]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi040", "root=default"]}]: dispatch 2023-07-14T19:14:32.722 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:14:32 smithi191 ceph-mon[98311]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 243 MiB used, 804 GiB / 805 GiB avail 2023-07-14T19:14:32.723 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:14:32 smithi191 ceph-mon[98311]: from='osd.9 [v2:172.21.15.40:6826/2228380263,v1:172.21.15.40:6827/2228380263]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2023-07-14T19:14:32.723 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:14:32 smithi191 ceph-mon[98311]: osdmap e62: 10 total, 9 up, 10 in 2023-07-14T19:14:32.723 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:14:32 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-07-14T19:14:32.723 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:14:32 smithi191 ceph-mon[98311]: from='osd.9 [v2:172.21.15.40:6826/2228380263,v1:172.21.15.40:6827/2228380263]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi040", "root=default"]}]: dispatch 2023-07-14T19:14:32.978 DEBUG:teuthology.orchestra.run.smithi130:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid adc52f04-2279-11ee-9b33-001a4aab830c -- ceph orch daemon add osd smithi130:vg_nvme/lv_1 2023-07-14T19:14:33.536 INFO:teuthology.orchestra.run.smithi130.stderr:Inferring config /var/lib/ceph/adc52f04-2279-11ee-9b33-001a4aab830c/mon.b/config 2023-07-14T19:14:33.721 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:14:33 smithi040 ceph-mon[96518]: from='osd.9 [v2:172.21.15.40:6826/2228380263,v1:172.21.15.40:6827/2228380263]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi040", "root=default"]}]': finished 2023-07-14T19:14:33.721 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:14:33 smithi040 ceph-mon[96518]: osdmap e63: 10 total, 9 up, 10 in 2023-07-14T19:14:33.722 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:14:33 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-07-14T19:14:33.722 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:14:33 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-07-14T19:14:33.722 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:14:33 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:14:33.722 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:14:33 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:14:33.722 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:14:33 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-07-14T19:14:33.722 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:14:33 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-07-14T19:14:33.722 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:14:33 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-07-14T19:14:33.723 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:14:33 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2023-07-14T19:14:33.723 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:14:33 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:14:33.723 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:14:33 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:14:33.723 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:14:33 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T19:14:33.723 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:14:33 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:14:33.723 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:14:33 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T19:14:33.723 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:14:33 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:14:33.724 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:14:33 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T19:14:33.724 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:14:33 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:14:33.724 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:14:33 smithi191 ceph-mon[98311]: from='osd.9 [v2:172.21.15.40:6826/2228380263,v1:172.21.15.40:6827/2228380263]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi040", "root=default"]}]': finished 2023-07-14T19:14:33.724 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:14:33 smithi191 ceph-mon[98311]: osdmap e63: 10 total, 9 up, 10 in 2023-07-14T19:14:33.724 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:14:33 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-07-14T19:14:33.725 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:14:33 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-07-14T19:14:33.725 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:14:33 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:14:33.725 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:14:33 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:14:33.725 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:14:33 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-07-14T19:14:33.725 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:14:33 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-07-14T19:14:33.725 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:14:33 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-07-14T19:14:33.725 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:14:33 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2023-07-14T19:14:33.726 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:14:33 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:14:33.726 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:14:33 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:14:33.726 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:14:33 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T19:14:33.726 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:14:33 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:14:33.726 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:14:33 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T19:14:33.726 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:14:33 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:14:33.727 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:14:33 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T19:14:33.727 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:14:33 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:14:33.742 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:14:33 smithi130 ceph-mon[97832]: from='osd.9 [v2:172.21.15.40:6826/2228380263,v1:172.21.15.40:6827/2228380263]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi040", "root=default"]}]': finished 2023-07-14T19:14:33.742 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:14:33 smithi130 ceph-mon[97832]: osdmap e63: 10 total, 9 up, 10 in 2023-07-14T19:14:33.742 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:14:33 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-07-14T19:14:33.743 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:14:33 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-07-14T19:14:33.743 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:14:33 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:14:33.743 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:14:33 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:14:33.743 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:14:33 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-07-14T19:14:33.743 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:14:33 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-07-14T19:14:33.743 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:14:33 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-07-14T19:14:33.743 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:14:33 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2023-07-14T19:14:33.744 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:14:33 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:14:33.744 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:14:33 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:14:33.744 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:14:33 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T19:14:33.744 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:14:33 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:14:33.744 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:14:33 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T19:14:33.744 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:14:33 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:14:33.744 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:14:33 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T19:14:33.745 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:14:33 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:14:34.680 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:14:34 smithi130 ceph-mon[97832]: purged_snaps scrub starts 2023-07-14T19:14:34.680 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:14:34 smithi130 ceph-mon[97832]: purged_snaps scrub ok 2023-07-14T19:14:34.680 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:14:34 smithi130 ceph-mon[97832]: pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 243 MiB used, 804 GiB / 805 GiB avail 2023-07-14T19:14:34.681 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:14:34 smithi130 ceph-mon[97832]: Detected new or changed devices on smithi040 2023-07-14T19:14:34.681 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:14:34 smithi130 ceph-mon[97832]: Adjusting osd_memory_target on smithi040 to 4293M 2023-07-14T19:14:34.681 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:14:34 smithi130 ceph-mon[97832]: osd.9 [v2:172.21.15.40:6826/2228380263,v1:172.21.15.40:6827/2228380263] boot 2023-07-14T19:14:34.681 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:14:34 smithi130 ceph-mon[97832]: osdmap e64: 10 total, 10 up, 10 in 2023-07-14T19:14:34.681 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:14:34 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-07-14T19:14:34.722 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:14:34 smithi040 ceph-mon[96518]: purged_snaps scrub starts 2023-07-14T19:14:34.722 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:14:34 smithi040 ceph-mon[96518]: purged_snaps scrub ok 2023-07-14T19:14:34.722 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:14:34 smithi040 ceph-mon[96518]: pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 243 MiB used, 804 GiB / 805 GiB avail 2023-07-14T19:14:34.722 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:14:34 smithi040 ceph-mon[96518]: Detected new or changed devices on smithi040 2023-07-14T19:14:34.722 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:14:34 smithi040 ceph-mon[96518]: Adjusting osd_memory_target on smithi040 to 4293M 2023-07-14T19:14:34.722 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:14:34 smithi040 ceph-mon[96518]: osd.9 [v2:172.21.15.40:6826/2228380263,v1:172.21.15.40:6827/2228380263] boot 2023-07-14T19:14:34.723 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:14:34 smithi040 ceph-mon[96518]: osdmap e64: 10 total, 10 up, 10 in 2023-07-14T19:14:34.723 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:14:34 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-07-14T19:14:34.723 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:14:34 smithi191 ceph-mon[98311]: purged_snaps scrub starts 2023-07-14T19:14:34.723 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:14:34 smithi191 ceph-mon[98311]: purged_snaps scrub ok 2023-07-14T19:14:34.723 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:14:34 smithi191 ceph-mon[98311]: pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 243 MiB used, 804 GiB / 805 GiB avail 2023-07-14T19:14:34.724 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:14:34 smithi191 ceph-mon[98311]: Detected new or changed devices on smithi040 2023-07-14T19:14:34.724 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:14:34 smithi191 ceph-mon[98311]: Adjusting osd_memory_target on smithi040 to 4293M 2023-07-14T19:14:34.724 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:14:34 smithi191 ceph-mon[98311]: osd.9 [v2:172.21.15.40:6826/2228380263,v1:172.21.15.40:6827/2228380263] boot 2023-07-14T19:14:34.724 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:14:34 smithi191 ceph-mon[98311]: osdmap e64: 10 total, 10 up, 10 in 2023-07-14T19:14:34.724 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:14:34 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-07-14T19:14:34.867 INFO:teuthology.orchestra.run.smithi130.stderr:2023-07-14T19:14:34.865+0000 7f540f9c9700 1 -- 172.21.15.130:0/3863069669 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f54081018e0 msgr2=0x7f54081081a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:14:34.867 INFO:teuthology.orchestra.run.smithi130.stderr:2023-07-14T19:14:34.865+0000 7f540f9c9700 1 --2- 172.21.15.130:0/3863069669 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f54081018e0 0x7f54081081a0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f540000b0a0 tx=0x7f540000b3b0 comp rx=0 tx=0).stop 2023-07-14T19:14:34.867 INFO:teuthology.orchestra.run.smithi130.stderr:2023-07-14T19:14:34.865+0000 7f540f9c9700 1 -- 172.21.15.130:0/3863069669 shutdown_connections 2023-07-14T19:14:34.868 INFO:teuthology.orchestra.run.smithi130.stderr:2023-07-14T19:14:34.865+0000 7f540f9c9700 1 --2- 172.21.15.130:0/3863069669 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f54081018e0 0x7f54081081a0 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:14:34.868 INFO:teuthology.orchestra.run.smithi130.stderr:2023-07-14T19:14:34.865+0000 7f540f9c9700 1 --2- 172.21.15.130:0/3863069669 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f54080ffd50 0x7f5408100160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:14:34.868 INFO:teuthology.orchestra.run.smithi130.stderr:2023-07-14T19:14:34.865+0000 7f540f9c9700 1 --2- 172.21.15.130:0/3863069669 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5408100f50 0x7f54081013a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:14:34.868 INFO:teuthology.orchestra.run.smithi130.stderr:2023-07-14T19:14:34.865+0000 7f540f9c9700 1 -- 172.21.15.130:0/3863069669 >> 172.21.15.130:0/3863069669 conn(0x7f54080fb380 msgr2=0x7f54080fd7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:14:34.868 INFO:teuthology.orchestra.run.smithi130.stderr:2023-07-14T19:14:34.865+0000 7f540f9c9700 1 -- 172.21.15.130:0/3863069669 shutdown_connections 2023-07-14T19:14:34.868 INFO:teuthology.orchestra.run.smithi130.stderr:2023-07-14T19:14:34.865+0000 7f540f9c9700 1 -- 172.21.15.130:0/3863069669 wait complete. 2023-07-14T19:14:34.869 INFO:teuthology.orchestra.run.smithi130.stderr:2023-07-14T19:14:34.865+0000 7f540f9c9700 1 Processor -- start 2023-07-14T19:14:34.869 INFO:teuthology.orchestra.run.smithi130.stderr:2023-07-14T19:14:34.865+0000 7f540f9c9700 1 -- start start 2023-07-14T19:14:34.869 INFO:teuthology.orchestra.run.smithi130.stderr:2023-07-14T19:14:34.866+0000 7f540f9c9700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f54080ffd50 0x7f5408195ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:14:34.869 INFO:teuthology.orchestra.run.smithi130.stderr:2023-07-14T19:14:34.866+0000 7f540f9c9700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5408100f50 0x7f5408195ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:14:34.869 INFO:teuthology.orchestra.run.smithi130.stderr:2023-07-14T19:14:34.866+0000 7f540f9c9700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f54081018e0 0x7f540819d070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:14:34.869 INFO:teuthology.orchestra.run.smithi130.stderr:2023-07-14T19:14:34.866+0000 7f540f9c9700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f540819d640 con 0x7f54080ffd50 2023-07-14T19:14:34.869 INFO:teuthology.orchestra.run.smithi130.stderr:2023-07-14T19:14:34.866+0000 7f540f9c9700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f540819d780 con 0x7f54081018e0 2023-07-14T19:14:34.869 INFO:teuthology.orchestra.run.smithi130.stderr:2023-07-14T19:14:34.866+0000 7f540f9c9700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f540819d910 con 0x7f5408100f50 2023-07-14T19:14:34.870 INFO:teuthology.orchestra.run.smithi130.stderr:2023-07-14T19:14:34.866+0000 7f540df66700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f54081018e0 0x7f540819d070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:14:34.870 INFO:teuthology.orchestra.run.smithi130.stderr:2023-07-14T19:14:34.866+0000 7f540df66700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f54081018e0 0x7f540819d070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.130:39886/0 (socket says 172.21.15.130:39886) 2023-07-14T19:14:34.870 INFO:teuthology.orchestra.run.smithi130.stderr:2023-07-14T19:14:34.866+0000 7f540cf64700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5408100f50 0x7f5408195ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:14:34.870 INFO:teuthology.orchestra.run.smithi130.stderr:2023-07-14T19:14:34.866+0000 7f540df66700 1 -- 172.21.15.130:0/1373937275 learned_addr learned my addr 172.21.15.130:0/1373937275 (peer_addr_for_me v2:172.21.15.130:0/0) 2023-07-14T19:14:34.870 INFO:teuthology.orchestra.run.smithi130.stderr:2023-07-14T19:14:34.866+0000 7f540d765700 1 --2- 172.21.15.130:0/1373937275 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f54080ffd50 0x7f5408195ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:14:34.870 INFO:teuthology.orchestra.run.smithi130.stderr:2023-07-14T19:14:34.866+0000 7f540df66700 1 -- 172.21.15.130:0/1373937275 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5408100f50 msgr2=0x7f5408195ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:14:34.870 INFO:teuthology.orchestra.run.smithi130.stderr:2023-07-14T19:14:34.866+0000 7f540df66700 1 --2- 172.21.15.130:0/1373937275 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5408100f50 0x7f5408195ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:14:34.871 INFO:teuthology.orchestra.run.smithi130.stderr:2023-07-14T19:14:34.866+0000 7f540df66700 1 -- 172.21.15.130:0/1373937275 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f54080ffd50 msgr2=0x7f5408195ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:14:34.871 INFO:teuthology.orchestra.run.smithi130.stderr:2023-07-14T19:14:34.866+0000 7f540df66700 1 --2- 172.21.15.130:0/1373937275 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f54080ffd50 0x7f5408195ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:14:34.871 INFO:teuthology.orchestra.run.smithi130.stderr:2023-07-14T19:14:34.866+0000 7f540df66700 1 -- 172.21.15.130:0/1373937275 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5400009d00 con 0x7f54081018e0 2023-07-14T19:14:34.871 INFO:teuthology.orchestra.run.smithi130.stderr:2023-07-14T19:14:34.866+0000 7f540df66700 1 --2- 172.21.15.130:0/1373937275 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f54081018e0 0x7f540819d070 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f54000062a0 tx=0x7f5400004d00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:14:34.871 INFO:teuthology.orchestra.run.smithi130.stderr:2023-07-14T19:14:34.866+0000 7f53fe7fc700 1 -- 172.21.15.130:0/1373937275 <== mon.2 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f540000e040 con 0x7f54081018e0 2023-07-14T19:14:34.871 INFO:teuthology.orchestra.run.smithi130.stderr:2023-07-14T19:14:34.866+0000 7f540f9c9700 1 -- 172.21.15.130:0/1373937275 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f540819db90 con 0x7f54081018e0 2023-07-14T19:14:34.871 INFO:teuthology.orchestra.run.smithi130.stderr:2023-07-14T19:14:34.866+0000 7f53fe7fc700 1 -- 172.21.15.130:0/1373937275 <== mon.2 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f540000cec0 con 0x7f54081018e0 2023-07-14T19:14:34.872 INFO:teuthology.orchestra.run.smithi130.stderr:2023-07-14T19:14:34.866+0000 7f53fe7fc700 1 -- 172.21.15.130:0/1373937275 <== mon.2 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f540001bb30 con 0x7f54081018e0 2023-07-14T19:14:34.872 INFO:teuthology.orchestra.run.smithi130.stderr:2023-07-14T19:14:34.866+0000 7f540f9c9700 1 -- 172.21.15.130:0/1373937275 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f540819e050 con 0x7f54081018e0 2023-07-14T19:14:34.872 INFO:teuthology.orchestra.run.smithi130.stderr:2023-07-14T19:14:34.867+0000 7f540f9c9700 1 -- 172.21.15.130:0/1373937275 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5408107300 con 0x7f54081018e0 2023-07-14T19:14:34.873 INFO:teuthology.orchestra.run.smithi130.stderr:2023-07-14T19:14:34.868+0000 7f53fe7fc700 1 -- 172.21.15.130:0/1373937275 <== mon.2 v2:172.21.15.130:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f540001bc90 con 0x7f54081018e0 2023-07-14T19:14:34.873 INFO:teuthology.orchestra.run.smithi130.stderr:2023-07-14T19:14:34.869+0000 7f53fe7fc700 1 --2- 172.21.15.130:0/1373937275 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f53f40a1150 0x7f53f40a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:14:34.873 INFO:teuthology.orchestra.run.smithi130.stderr:2023-07-14T19:14:34.869+0000 7f53fe7fc700 1 -- 172.21.15.130:0/1373937275 <== mon.2 v2:172.21.15.130:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 6261+0+0 (secure 0 0 0) 0x7f54000ce360 con 0x7f54081018e0 2023-07-14T19:14:34.873 INFO:teuthology.orchestra.run.smithi130.stderr:2023-07-14T19:14:34.869+0000 7f540d765700 1 --2- 172.21.15.130:0/1373937275 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f53f40a1150 0x7f53f40a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:14:34.873 INFO:teuthology.orchestra.run.smithi130.stderr:2023-07-14T19:14:34.870+0000 7f540d765700 1 --2- 172.21.15.130:0/1373937275 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f53f40a1150 0x7f53f40a3600 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f5408100db0 tx=0x7f540400b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:14:34.874 INFO:teuthology.orchestra.run.smithi130.stderr:2023-07-14T19:14:34.873+0000 7f53fe7fc700 1 -- 172.21.15.130:0/1373937275 <== mon.2 v2:172.21.15.130:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f5400098340 con 0x7f54081018e0 2023-07-14T19:14:35.066 INFO:teuthology.orchestra.run.smithi130.stderr:2023-07-14T19:14:35.065+0000 7f540f9c9700 1 -- 172.21.15.130:0/1373937275 --> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi130:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) v1 -- 0x7f5408060a80 con 0x7f53f40a1150 2023-07-14T19:14:35.721 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:14:35 smithi040 ceph-mon[96518]: osdmap e65: 10 total, 10 up, 10 in 2023-07-14T19:14:35.721 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:14:35 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-14T19:14:35.721 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:14:35 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-14T19:14:35.722 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:14:35 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:14:35.722 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:14:35 smithi191 ceph-mon[98311]: osdmap e65: 10 total, 10 up, 10 in 2023-07-14T19:14:35.722 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:14:35 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-14T19:14:35.722 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:14:35 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-14T19:14:35.723 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:14:35 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:14:35.742 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:14:35 smithi130 ceph-mon[97832]: osdmap e65: 10 total, 10 up, 10 in 2023-07-14T19:14:35.743 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:14:35 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-14T19:14:35.743 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:14:35 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-14T19:14:35.743 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:14:35 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:14:36.492 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:14:36 smithi130 ceph-mon[97832]: pgmap v197: 1 pgs: 1 active+clean; 577 KiB data, 270 MiB used, 894 GiB / 894 GiB avail 2023-07-14T19:14:36.492 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:14:36 smithi130 ceph-mon[97832]: from='client.24356 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi130:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T19:14:36.721 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:14:36 smithi040 ceph-mon[96518]: pgmap v197: 1 pgs: 1 active+clean; 577 KiB data, 270 MiB used, 894 GiB / 894 GiB avail 2023-07-14T19:14:36.721 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:14:36 smithi040 ceph-mon[96518]: from='client.24356 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi130:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T19:14:36.722 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:14:36 smithi191 ceph-mon[98311]: pgmap v197: 1 pgs: 1 active+clean; 577 KiB data, 270 MiB used, 894 GiB / 894 GiB avail 2023-07-14T19:14:36.722 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:14:36 smithi191 ceph-mon[98311]: from='client.24356 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi130:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T19:14:37.971 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:14:37 smithi040 ceph-mon[96518]: from='client.? 172.21.15.130:0/2365583022' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4a93d621-49ee-4ad7-9e49-750614faf912"}]: dispatch 2023-07-14T19:14:37.971 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:14:37 smithi040 ceph-mon[96518]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4a93d621-49ee-4ad7-9e49-750614faf912"}]: dispatch 2023-07-14T19:14:37.971 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:14:37 smithi040 ceph-mon[96518]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "4a93d621-49ee-4ad7-9e49-750614faf912"}]': finished 2023-07-14T19:14:37.972 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:14:37 smithi040 ceph-mon[96518]: osdmap e66: 11 total, 10 up, 11 in 2023-07-14T19:14:37.972 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:14:37 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-07-14T19:14:37.972 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:14:37 smithi191 ceph-mon[98311]: from='client.? 172.21.15.130:0/2365583022' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4a93d621-49ee-4ad7-9e49-750614faf912"}]: dispatch 2023-07-14T19:14:37.972 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:14:37 smithi191 ceph-mon[98311]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4a93d621-49ee-4ad7-9e49-750614faf912"}]: dispatch 2023-07-14T19:14:37.973 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:14:37 smithi191 ceph-mon[98311]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "4a93d621-49ee-4ad7-9e49-750614faf912"}]': finished 2023-07-14T19:14:37.973 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:14:37 smithi191 ceph-mon[98311]: osdmap e66: 11 total, 10 up, 11 in 2023-07-14T19:14:37.973 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:14:37 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-07-14T19:14:37.992 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:14:37 smithi130 ceph-mon[97832]: from='client.? 172.21.15.130:0/2365583022' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4a93d621-49ee-4ad7-9e49-750614faf912"}]: dispatch 2023-07-14T19:14:37.992 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:14:37 smithi130 ceph-mon[97832]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4a93d621-49ee-4ad7-9e49-750614faf912"}]: dispatch 2023-07-14T19:14:37.993 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:14:37 smithi130 ceph-mon[97832]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "4a93d621-49ee-4ad7-9e49-750614faf912"}]': finished 2023-07-14T19:14:37.993 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:14:37 smithi130 ceph-mon[97832]: osdmap e66: 11 total, 10 up, 11 in 2023-07-14T19:14:37.993 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:14:37 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-07-14T19:14:38.971 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:14:38 smithi040 ceph-mon[96518]: pgmap v198: 1 pgs: 1 active+clean; 577 KiB data, 270 MiB used, 894 GiB / 894 GiB avail 2023-07-14T19:14:38.971 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:14:38 smithi040 ceph-mon[96518]: from='client.? 172.21.15.130:0/2023713579' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-14T19:14:38.972 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:14:38 smithi191 ceph-mon[98311]: pgmap v198: 1 pgs: 1 active+clean; 577 KiB data, 270 MiB used, 894 GiB / 894 GiB avail 2023-07-14T19:14:38.972 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:14:38 smithi191 ceph-mon[98311]: from='client.? 172.21.15.130:0/2023713579' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-14T19:14:38.992 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:14:38 smithi130 ceph-mon[97832]: pgmap v198: 1 pgs: 1 active+clean; 577 KiB data, 270 MiB used, 894 GiB / 894 GiB avail 2023-07-14T19:14:38.993 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:14:38 smithi130 ceph-mon[97832]: from='client.? 172.21.15.130:0/2023713579' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-14T19:14:40.971 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:14:40 smithi040 ceph-mon[96518]: pgmap v200: 1 pgs: 1 active+clean; 577 KiB data, 270 MiB used, 894 GiB / 894 GiB avail 2023-07-14T19:14:40.972 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:14:40 smithi191 ceph-mon[98311]: pgmap v200: 1 pgs: 1 active+clean; 577 KiB data, 270 MiB used, 894 GiB / 894 GiB avail 2023-07-14T19:14:40.992 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:14:40 smithi130 ceph-mon[97832]: pgmap v200: 1 pgs: 1 active+clean; 577 KiB data, 270 MiB used, 894 GiB / 894 GiB avail 2023-07-14T19:14:42.707 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:14:42 smithi130 ceph-mon[97832]: pgmap v201: 1 pgs: 1 active+clean; 577 KiB data, 270 MiB used, 894 GiB / 894 GiB avail 2023-07-14T19:14:42.971 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:14:42 smithi040 ceph-mon[96518]: pgmap v201: 1 pgs: 1 active+clean; 577 KiB data, 270 MiB used, 894 GiB / 894 GiB avail 2023-07-14T19:14:42.972 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:14:42 smithi191 ceph-mon[98311]: pgmap v201: 1 pgs: 1 active+clean; 577 KiB data, 270 MiB used, 894 GiB / 894 GiB avail 2023-07-14T19:14:44.863 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:14:44 smithi130 ceph-mon[97832]: pgmap v202: 1 pgs: 1 active+clean; 577 KiB data, 271 MiB used, 894 GiB / 894 GiB avail 2023-07-14T19:14:44.863 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:14:44 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2023-07-14T19:14:44.863 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:14:44 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:14:44.971 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:14:44 smithi040 ceph-mon[96518]: pgmap v202: 1 pgs: 1 active+clean; 577 KiB data, 271 MiB used, 894 GiB / 894 GiB avail 2023-07-14T19:14:44.971 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:14:44 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2023-07-14T19:14:44.971 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:14:44 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:14:44.972 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:14:44 smithi191 ceph-mon[98311]: pgmap v202: 1 pgs: 1 active+clean; 577 KiB data, 271 MiB used, 894 GiB / 894 GiB avail 2023-07-14T19:14:44.972 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:14:44 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2023-07-14T19:14:44.972 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:14:44 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:14:45.971 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:14:45 smithi040 ceph-mon[96518]: Deploying daemon osd.10 on smithi130 2023-07-14T19:14:45.972 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:14:45 smithi191 ceph-mon[98311]: Deploying daemon osd.10 on smithi130 2023-07-14T19:14:45.992 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:14:45 smithi130 ceph-mon[97832]: Deploying daemon osd.10 on smithi130 2023-07-14T19:14:46.737 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:14:46 smithi130 ceph-mon[97832]: pgmap v203: 1 pgs: 1 active+clean; 577 KiB data, 271 MiB used, 894 GiB / 894 GiB avail 2023-07-14T19:14:46.971 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:14:46 smithi040 ceph-mon[96518]: pgmap v203: 1 pgs: 1 active+clean; 577 KiB data, 271 MiB used, 894 GiB / 894 GiB avail 2023-07-14T19:14:46.972 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:14:46 smithi191 ceph-mon[98311]: pgmap v203: 1 pgs: 1 active+clean; 577 KiB data, 271 MiB used, 894 GiB / 894 GiB avail 2023-07-14T19:14:48.699 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:14:48 smithi130 ceph-mon[97832]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 271 MiB used, 894 GiB / 894 GiB avail 2023-07-14T19:14:48.971 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:14:48 smithi040 ceph-mon[96518]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 271 MiB used, 894 GiB / 894 GiB avail 2023-07-14T19:14:48.972 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:14:48 smithi191 ceph-mon[98311]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 271 MiB used, 894 GiB / 894 GiB avail 2023-07-14T19:14:49.865 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:14:49 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:14:49.865 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:14:49 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:14:49.865 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:14:49 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T19:14:50.221 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:14:49 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:14:50.221 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:14:49 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:14:50.221 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:14:49 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T19:14:50.222 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:14:49 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:14:50.222 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:14:49 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:14:50.222 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:14:49 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T19:14:50.992 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:14:50 smithi130 ceph-mon[97832]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 271 MiB used, 894 GiB / 894 GiB avail 2023-07-14T19:14:51.221 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:14:50 smithi040 ceph-mon[96518]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 271 MiB used, 894 GiB / 894 GiB avail 2023-07-14T19:14:51.222 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:14:50 smithi191 ceph-mon[98311]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 271 MiB used, 894 GiB / 894 GiB avail 2023-07-14T19:14:51.579 INFO:teuthology.orchestra.run.smithi130.stdout:Created osd(s) 10 on host 'smithi130' 2023-07-14T19:14:51.579 INFO:teuthology.orchestra.run.smithi130.stderr:2023-07-14T19:14:51.577+0000 7f53fe7fc700 1 -- 172.21.15.130:0/1373937275 <== mgr.14152 v2:172.21.15.40:6800/1693027118 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+38 (secure 0 0 0) 0x7f5408060a80 con 0x7f53f40a1150 2023-07-14T19:14:51.581 INFO:teuthology.orchestra.run.smithi130.stderr:2023-07-14T19:14:51.579+0000 7f540f9c9700 1 -- 172.21.15.130:0/1373937275 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f53f40a1150 msgr2=0x7f53f40a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:14:51.582 INFO:teuthology.orchestra.run.smithi130.stderr:2023-07-14T19:14:51.579+0000 7f540f9c9700 1 --2- 172.21.15.130:0/1373937275 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f53f40a1150 0x7f53f40a3600 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f5408100db0 tx=0x7f540400b410 comp rx=0 tx=0).stop 2023-07-14T19:14:51.582 INFO:teuthology.orchestra.run.smithi130.stderr:2023-07-14T19:14:51.579+0000 7f540f9c9700 1 -- 172.21.15.130:0/1373937275 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f54081018e0 msgr2=0x7f540819d070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:14:51.582 INFO:teuthology.orchestra.run.smithi130.stderr:2023-07-14T19:14:51.579+0000 7f540f9c9700 1 --2- 172.21.15.130:0/1373937275 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f54081018e0 0x7f540819d070 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f54000062a0 tx=0x7f5400004d00 comp rx=0 tx=0).stop 2023-07-14T19:14:51.582 INFO:teuthology.orchestra.run.smithi130.stderr:2023-07-14T19:14:51.579+0000 7f540f9c9700 1 -- 172.21.15.130:0/1373937275 shutdown_connections 2023-07-14T19:14:51.582 INFO:teuthology.orchestra.run.smithi130.stderr:2023-07-14T19:14:51.579+0000 7f540f9c9700 1 --2- 172.21.15.130:0/1373937275 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f53f40a1150 0x7f53f40a3600 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:14:51.583 INFO:teuthology.orchestra.run.smithi130.stderr:2023-07-14T19:14:51.579+0000 7f540f9c9700 1 --2- 172.21.15.130:0/1373937275 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f54081018e0 0x7f540819d070 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:14:51.583 INFO:teuthology.orchestra.run.smithi130.stderr:2023-07-14T19:14:51.579+0000 7f540f9c9700 1 --2- 172.21.15.130:0/1373937275 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f54080ffd50 0x7f5408195ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:14:51.583 INFO:teuthology.orchestra.run.smithi130.stderr:2023-07-14T19:14:51.579+0000 7f540f9c9700 1 --2- 172.21.15.130:0/1373937275 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5408100f50 0x7f5408195ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:14:51.583 INFO:teuthology.orchestra.run.smithi130.stderr:2023-07-14T19:14:51.579+0000 7f540f9c9700 1 -- 172.21.15.130:0/1373937275 >> 172.21.15.130:0/1373937275 conn(0x7f54080fb380 msgr2=0x7f5408104440 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:14:51.583 INFO:teuthology.orchestra.run.smithi130.stderr:2023-07-14T19:14:51.580+0000 7f540f9c9700 1 -- 172.21.15.130:0/1373937275 shutdown_connections 2023-07-14T19:14:51.583 INFO:teuthology.orchestra.run.smithi130.stderr:2023-07-14T19:14:51.580+0000 7f540f9c9700 1 -- 172.21.15.130:0/1373937275 wait complete. 2023-07-14T19:14:52.034 DEBUG:teuthology.orchestra.run.smithi130:osd.10> sudo journalctl -f -n 0 -u ceph-adc52f04-2279-11ee-9b33-001a4aab830c@osd.10.service 2023-07-14T19:14:52.036 INFO:tasks.cephadm:Deploying osd.11 on smithi191 with /dev/vg_nvme/lv_1... 2023-07-14T19:14:52.037 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid adc52f04-2279-11ee-9b33-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2023-07-14T19:14:52.065 INFO:journalctl@ceph.osd.10.smithi130.stdout:-- Logs begin at Fri 2023-07-14 18:50:32 UTC. -- 2023-07-14T19:14:52.472 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:14:52 smithi191 ceph-mon[98311]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 271 MiB used, 894 GiB / 894 GiB avail 2023-07-14T19:14:52.472 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:14:52 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:14:52.472 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:14:52 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:14:52.472 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:14:52 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:14:52.473 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:14:52 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T19:14:52.473 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:14:52 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:14:52.473 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:14:52 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:14:52.473 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:14:52 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T19:14:52.473 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:14:52 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:14:52.603 INFO:teuthology.orchestra.run.smithi191.stderr:Inferring config /var/lib/ceph/adc52f04-2279-11ee-9b33-001a4aab830c/mon.c/config 2023-07-14T19:14:52.665 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:14:52 smithi130 ceph-mon[97832]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 271 MiB used, 894 GiB / 894 GiB avail 2023-07-14T19:14:52.665 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:14:52 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:14:52.665 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:14:52 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:14:52.666 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:14:52 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:14:52.666 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:14:52 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T19:14:52.666 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:14:52 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:14:52.666 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:14:52 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:14:52.666 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:14:52 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T19:14:52.666 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:14:52 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:14:52.721 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:14:52 smithi040 ceph-mon[96518]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 271 MiB used, 894 GiB / 894 GiB avail 2023-07-14T19:14:52.721 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:14:52 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:14:52.721 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:14:52 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:14:52.721 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:14:52 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:14:52.722 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:14:52 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T19:14:52.722 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:14:52 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:14:52.722 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:14:52 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:14:52.722 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:14:52 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T19:14:52.722 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:14:52 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:14:52.941 INFO:journalctl@ceph.osd.10.smithi130.stdout:Jul 14 19:14:52 smithi130 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-10[119160]: 2023-07-14T19:14:52.693+0000 7fa20db4d640 -1 osd.10 0 log_to_monitors true 2023-07-14T19:14:53.472 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:14:53 smithi191 ceph-mon[98311]: from='osd.10 [v2:172.21.15.130:6824/2417067798,v1:172.21.15.130:6825/2417067798]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2023-07-14T19:14:53.472 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:14:53 smithi191 ceph-mon[98311]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2023-07-14T19:14:53.697 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:14:53 smithi130 ceph-mon[97832]: from='osd.10 [v2:172.21.15.130:6824/2417067798,v1:172.21.15.130:6825/2417067798]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2023-07-14T19:14:53.697 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:14:53 smithi130 ceph-mon[97832]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2023-07-14T19:14:53.721 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:14:53 smithi040 ceph-mon[96518]: from='osd.10 [v2:172.21.15.130:6824/2417067798,v1:172.21.15.130:6825/2417067798]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2023-07-14T19:14:53.721 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:14:53 smithi040 ceph-mon[96518]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2023-07-14T19:14:54.053 INFO:teuthology.orchestra.run.smithi191.stderr:--> Zapping: /dev/vg_nvme/lv_1 2023-07-14T19:14:54.053 INFO:teuthology.orchestra.run.smithi191.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2023-07-14T19:14:54.053 INFO:teuthology.orchestra.run.smithi191.stderr: stderr: 10+0 records in 2023-07-14T19:14:54.054 INFO:teuthology.orchestra.run.smithi191.stderr:10+0 records out 2023-07-14T19:14:54.054 INFO:teuthology.orchestra.run.smithi191.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0131225 s, 799 MB/s 2023-07-14T19:14:54.054 INFO:teuthology.orchestra.run.smithi191.stderr:--> Zapping successful for: 2023-07-14T19:14:54.419 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:14:54 smithi191 ceph-mon[98311]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 271 MiB used, 894 GiB / 894 GiB avail 2023-07-14T19:14:54.419 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:14:54 smithi191 ceph-mon[98311]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2023-07-14T19:14:54.419 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:14:54 smithi191 ceph-mon[98311]: osdmap e67: 11 total, 10 up, 11 in 2023-07-14T19:14:54.419 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:14:54 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-07-14T19:14:54.419 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:14:54 smithi191 ceph-mon[98311]: from='osd.10 [v2:172.21.15.130:6824/2417067798,v1:172.21.15.130:6825/2417067798]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi130", "root=default"]}]: dispatch 2023-07-14T19:14:54.420 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:14:54 smithi191 ceph-mon[98311]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi130", "root=default"]}]: dispatch 2023-07-14T19:14:54.420 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:14:54 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:14:54.420 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:14:54 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:14:54.420 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:14:54 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-07-14T19:14:54.420 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:14:54 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-07-14T19:14:54.421 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:14:54 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-07-14T19:14:54.421 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:14:54 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2023-07-14T19:14:54.421 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:14:54 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:14:54.421 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:14:54 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:14:54.421 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:14:54 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T19:14:54.421 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:14:54 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:14:54.444 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid adc52f04-2279-11ee-9b33-001a4aab830c -- ceph orch daemon add osd smithi191:vg_nvme/lv_1 2023-07-14T19:14:54.492 INFO:journalctl@ceph.osd.10.smithi130.stdout:Jul 14 19:14:54 smithi130 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-10[119160]: 2023-07-14T19:14:54.296+0000 7fa2031c5700 -1 osd.10 0 waiting for initial osdmap 2023-07-14T19:14:54.492 INFO:journalctl@ceph.osd.10.smithi130.stdout:Jul 14 19:14:54 smithi130 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-10[119160]: 2023-07-14T19:14:54.307+0000 7fa1fc7b6700 -1 osd.10 68 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-07-14T19:14:54.493 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:14:54 smithi130 ceph-mon[97832]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 271 MiB used, 894 GiB / 894 GiB avail 2023-07-14T19:14:54.493 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:14:54 smithi130 ceph-mon[97832]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2023-07-14T19:14:54.493 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:14:54 smithi130 ceph-mon[97832]: osdmap e67: 11 total, 10 up, 11 in 2023-07-14T19:14:54.493 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:14:54 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-07-14T19:14:54.493 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:14:54 smithi130 ceph-mon[97832]: from='osd.10 [v2:172.21.15.130:6824/2417067798,v1:172.21.15.130:6825/2417067798]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi130", "root=default"]}]: dispatch 2023-07-14T19:14:54.494 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:14:54 smithi130 ceph-mon[97832]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi130", "root=default"]}]: dispatch 2023-07-14T19:14:54.494 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:14:54 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:14:54.494 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:14:54 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:14:54.494 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:14:54 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-07-14T19:14:54.494 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:14:54 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-07-14T19:14:54.494 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:14:54 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-07-14T19:14:54.494 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:14:54 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2023-07-14T19:14:54.495 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:14:54 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:14:54.495 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:14:54 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:14:54.495 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:14:54 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T19:14:54.495 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:14:54 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:14:54.721 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:14:54 smithi040 ceph-mon[96518]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 271 MiB used, 894 GiB / 894 GiB avail 2023-07-14T19:14:54.721 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:14:54 smithi040 ceph-mon[96518]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2023-07-14T19:14:54.722 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:14:54 smithi040 ceph-mon[96518]: osdmap e67: 11 total, 10 up, 11 in 2023-07-14T19:14:54.722 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:14:54 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-07-14T19:14:54.722 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:14:54 smithi040 ceph-mon[96518]: from='osd.10 [v2:172.21.15.130:6824/2417067798,v1:172.21.15.130:6825/2417067798]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi130", "root=default"]}]: dispatch 2023-07-14T19:14:54.722 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:14:54 smithi040 ceph-mon[96518]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi130", "root=default"]}]: dispatch 2023-07-14T19:14:54.722 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:14:54 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:14:54.722 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:14:54 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:14:54.723 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:14:54 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-07-14T19:14:54.723 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:14:54 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-07-14T19:14:54.723 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:14:54 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-07-14T19:14:54.723 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:14:54 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2023-07-14T19:14:54.723 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:14:54 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:14:54.723 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:14:54 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:14:54.723 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:14:54 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T19:14:54.723 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:14:54 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:14:54.996 INFO:teuthology.orchestra.run.smithi191.stderr:Inferring config /var/lib/ceph/adc52f04-2279-11ee-9b33-001a4aab830c/mon.c/config 2023-07-14T19:14:55.472 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:14:55 smithi191 ceph-mon[98311]: Detected new or changed devices on smithi130 2023-07-14T19:14:55.472 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:14:55 smithi191 ceph-mon[98311]: Adjusting osd_memory_target on smithi130 to 4292M 2023-07-14T19:14:55.473 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:14:55 smithi191 ceph-mon[98311]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi130", "root=default"]}]': finished 2023-07-14T19:14:55.473 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:14:55 smithi191 ceph-mon[98311]: osdmap e68: 11 total, 10 up, 11 in 2023-07-14T19:14:55.473 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:14:55 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-07-14T19:14:55.473 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:14:55 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-07-14T19:14:55.721 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:14:55 smithi040 ceph-mon[96518]: Detected new or changed devices on smithi130 2023-07-14T19:14:55.721 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:14:55 smithi040 ceph-mon[96518]: Adjusting osd_memory_target on smithi130 to 4292M 2023-07-14T19:14:55.721 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:14:55 smithi040 ceph-mon[96518]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi130", "root=default"]}]': finished 2023-07-14T19:14:55.722 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:14:55 smithi040 ceph-mon[96518]: osdmap e68: 11 total, 10 up, 11 in 2023-07-14T19:14:55.722 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:14:55 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-07-14T19:14:55.722 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:14:55 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-07-14T19:14:55.742 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:14:55 smithi130 ceph-mon[97832]: Detected new or changed devices on smithi130 2023-07-14T19:14:55.743 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:14:55 smithi130 ceph-mon[97832]: Adjusting osd_memory_target on smithi130 to 4292M 2023-07-14T19:14:55.743 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:14:55 smithi130 ceph-mon[97832]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi130", "root=default"]}]': finished 2023-07-14T19:14:55.743 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:14:55 smithi130 ceph-mon[97832]: osdmap e68: 11 total, 10 up, 11 in 2023-07-14T19:14:55.743 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:14:55 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-07-14T19:14:55.743 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:14:55 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-07-14T19:14:56.423 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:14:56.422+0000 7f61344ef700 1 -- 172.21.15.191:0/3872177901 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f612c100b30 msgr2=0x7f612c0fef40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:14:56.424 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:14:56.422+0000 7f61344ef700 1 --2- 172.21.15.191:0/3872177901 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f612c100b30 0x7f612c0fef40 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f6128009990 tx=0x7f6128009ca0 comp rx=0 tx=0).stop 2023-07-14T19:14:56.424 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:14:56.422+0000 7f61344ef700 1 -- 172.21.15.191:0/3872177901 shutdown_connections 2023-07-14T19:14:56.424 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:14:56.422+0000 7f61344ef700 1 --2- 172.21.15.191:0/3872177901 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f612c0ff5d0 0x7f612c107140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:14:56.424 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:14:56.422+0000 7f61344ef700 1 --2- 172.21.15.191:0/3872177901 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f612c1001e0 0x7f612c1005f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:14:56.425 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:14:56.422+0000 7f61344ef700 1 --2- 172.21.15.191:0/3872177901 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f612c100b30 0x7f612c0fef40 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:14:56.425 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:14:56.422+0000 7f61344ef700 1 -- 172.21.15.191:0/3872177901 >> 172.21.15.191:0/3872177901 conn(0x7f612c0f99c0 msgr2=0x7f612c0fbdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:14:56.425 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:14:56.422+0000 7f61344ef700 1 -- 172.21.15.191:0/3872177901 shutdown_connections 2023-07-14T19:14:56.425 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:14:56.422+0000 7f61344ef700 1 -- 172.21.15.191:0/3872177901 wait complete. 2023-07-14T19:14:56.427 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:14:56.423+0000 7f61344ef700 1 Processor -- start 2023-07-14T19:14:56.427 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:14:56.423+0000 7f61344ef700 1 -- start start 2023-07-14T19:14:56.427 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:14:56.423+0000 7f61344ef700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f612c0ff5d0 0x7f612c193860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:14:56.428 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:14:56.423+0000 7f61344ef700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f612c1001e0 0x7f612c193da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:14:56.428 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:14:56.423+0000 7f61344ef700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f612c100b30 0x7f612c19ae20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:14:56.428 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:14:56.423+0000 7f61344ef700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f612c19b3f0 con 0x7f612c100b30 2023-07-14T19:14:56.428 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:14:56.423+0000 7f61344ef700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f612c19b530 con 0x7f612c0ff5d0 2023-07-14T19:14:56.429 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:14:56.423+0000 7f61344ef700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f612c19b6c0 con 0x7f612c1001e0 2023-07-14T19:14:56.429 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:14:56.423+0000 7f6131a8a700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f612c1001e0 0x7f612c193da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:14:56.429 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:14:56.423+0000 7f6131a8a700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f612c1001e0 0x7f612c193da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.191:59622/0 (socket says 172.21.15.191:59622) 2023-07-14T19:14:56.429 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:14:56.423+0000 7f6131a8a700 1 -- 172.21.15.191:0/413079700 learned_addr learned my addr 172.21.15.191:0/413079700 (peer_addr_for_me v2:172.21.15.191:0/0) 2023-07-14T19:14:56.430 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:14:56.423+0000 7f613228b700 1 --2- 172.21.15.191:0/413079700 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f612c0ff5d0 0x7f612c193860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:14:56.430 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:14:56.423+0000 7f6131a8a700 1 -- 172.21.15.191:0/413079700 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f612c0ff5d0 msgr2=0x7f612c193860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:14:56.430 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:14:56.423+0000 7f6131a8a700 1 --2- 172.21.15.191:0/413079700 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f612c0ff5d0 0x7f612c193860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:14:56.430 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:14:56.423+0000 7f6131a8a700 1 -- 172.21.15.191:0/413079700 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f612c100b30 msgr2=0x7f612c19ae20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:14:56.430 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:14:56.423+0000 7f6131a8a700 1 --2- 172.21.15.191:0/413079700 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f612c100b30 0x7f612c19ae20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:14:56.431 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:14:56.423+0000 7f6131a8a700 1 -- 172.21.15.191:0/413079700 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61280095f0 con 0x7f612c1001e0 2023-07-14T19:14:56.431 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:14:56.423+0000 7f613228b700 1 --2- 172.21.15.191:0/413079700 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f612c0ff5d0 0x7f612c193860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T19:14:56.431 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:14:56.423+0000 7f6131a8a700 1 --2- 172.21.15.191:0/413079700 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f612c1001e0 0x7f612c193da0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f6128005910 tx=0x7f612800f710 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:14:56.431 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:14:56.423+0000 7f61237fe700 1 -- 172.21.15.191:0/413079700 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f612801d070 con 0x7f612c1001e0 2023-07-14T19:14:56.432 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:14:56.423+0000 7f61344ef700 1 -- 172.21.15.191:0/413079700 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f612c19b940 con 0x7f612c1001e0 2023-07-14T19:14:56.432 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:14:56.423+0000 7f61237fe700 1 -- 172.21.15.191:0/413079700 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f612800fbc0 con 0x7f612c1001e0 2023-07-14T19:14:56.432 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:14:56.423+0000 7f61237fe700 1 -- 172.21.15.191:0/413079700 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6128017650 con 0x7f612c1001e0 2023-07-14T19:14:56.433 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:14:56.424+0000 7f61344ef700 1 -- 172.21.15.191:0/413079700 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f612c19bde0 con 0x7f612c1001e0 2023-07-14T19:14:56.433 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:14:56.425+0000 7f61344ef700 1 -- 172.21.15.191:0/413079700 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f612c106290 con 0x7f612c1001e0 2023-07-14T19:14:56.434 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:14:56.425+0000 7f61237fe700 1 -- 172.21.15.191:0/413079700 <== mon.1 v2:172.21.15.191:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f6128022470 con 0x7f612c1001e0 2023-07-14T19:14:56.434 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:14:56.426+0000 7f61237fe700 1 --2- 172.21.15.191:0/413079700 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f61180a3360 0x7f61180a5810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:14:56.434 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:14:56.426+0000 7f61237fe700 1 -- 172.21.15.191:0/413079700 <== mon.1 v2:172.21.15.191:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 6723+0+0 (secure 0 0 0) 0x7f61280cd620 con 0x7f612c1001e0 2023-07-14T19:14:56.434 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:14:56.426+0000 7f613228b700 1 --2- 172.21.15.191:0/413079700 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f61180a3360 0x7f61180a5810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:14:56.434 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:14:56.427+0000 7f613228b700 1 --2- 172.21.15.191:0/413079700 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f61180a3360 0x7f61180a5810 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f612c0689f0 tx=0x7f611c00b5f0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:14:56.435 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:14:56.431+0000 7f61237fe700 1 -- 172.21.15.191:0/413079700 <== mon.1 v2:172.21.15.191:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f6128097430 con 0x7f612c1001e0 2023-07-14T19:14:56.624 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:14:56.623+0000 7f61344ef700 1 -- 172.21.15.191:0/413079700 --> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi191:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) v1 -- 0x7f612c060a80 con 0x7f61180a3360 2023-07-14T19:14:56.721 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:14:56 smithi040 ceph-mon[96518]: purged_snaps scrub starts 2023-07-14T19:14:56.721 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:14:56 smithi040 ceph-mon[96518]: purged_snaps scrub ok 2023-07-14T19:14:56.722 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:14:56 smithi040 ceph-mon[96518]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 271 MiB used, 894 GiB / 894 GiB avail 2023-07-14T19:14:56.722 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:14:56 smithi040 ceph-mon[96518]: osd.10 [v2:172.21.15.130:6824/2417067798,v1:172.21.15.130:6825/2417067798] boot 2023-07-14T19:14:56.723 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:14:56 smithi040 ceph-mon[96518]: osdmap e69: 11 total, 11 up, 11 in 2023-07-14T19:14:56.723 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:14:56 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-07-14T19:14:56.723 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:14:56 smithi040 ceph-mon[96518]: osdmap e70: 11 total, 11 up, 11 in 2023-07-14T19:14:56.724 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:14:56 smithi191 ceph-mon[98311]: purged_snaps scrub starts 2023-07-14T19:14:56.725 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:14:56 smithi191 ceph-mon[98311]: purged_snaps scrub ok 2023-07-14T19:14:56.725 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:14:56 smithi191 ceph-mon[98311]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 271 MiB used, 894 GiB / 894 GiB avail 2023-07-14T19:14:56.725 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:14:56 smithi191 ceph-mon[98311]: osd.10 [v2:172.21.15.130:6824/2417067798,v1:172.21.15.130:6825/2417067798] boot 2023-07-14T19:14:56.725 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:14:56 smithi191 ceph-mon[98311]: osdmap e69: 11 total, 11 up, 11 in 2023-07-14T19:14:56.726 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:14:56 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-07-14T19:14:56.726 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:14:56 smithi191 ceph-mon[98311]: osdmap e70: 11 total, 11 up, 11 in 2023-07-14T19:14:56.742 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:14:56 smithi130 ceph-mon[97832]: purged_snaps scrub starts 2023-07-14T19:14:56.743 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:14:56 smithi130 ceph-mon[97832]: purged_snaps scrub ok 2023-07-14T19:14:56.743 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:14:56 smithi130 ceph-mon[97832]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 271 MiB used, 894 GiB / 894 GiB avail 2023-07-14T19:14:56.743 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:14:56 smithi130 ceph-mon[97832]: osd.10 [v2:172.21.15.130:6824/2417067798,v1:172.21.15.130:6825/2417067798] boot 2023-07-14T19:14:56.743 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:14:56 smithi130 ceph-mon[97832]: osdmap e69: 11 total, 11 up, 11 in 2023-07-14T19:14:56.743 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:14:56 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-07-14T19:14:56.743 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:14:56 smithi130 ceph-mon[97832]: osdmap e70: 11 total, 11 up, 11 in 2023-07-14T19:14:57.722 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:14:57 smithi191 ceph-mon[98311]: from='client.24373 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi191:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T19:14:57.722 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:14:57 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-14T19:14:57.722 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:14:57 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-14T19:14:57.723 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:14:57 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:14:57.723 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:14:57 smithi191 ceph-mon[98311]: osdmap e71: 11 total, 11 up, 11 in 2023-07-14T19:14:57.971 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:14:57 smithi040 ceph-mon[96518]: from='client.24373 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi191:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T19:14:57.971 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:14:57 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-14T19:14:57.972 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:14:57 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-14T19:14:57.972 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:14:57 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:14:57.972 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:14:57 smithi040 ceph-mon[96518]: osdmap e71: 11 total, 11 up, 11 in 2023-07-14T19:14:57.992 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:14:57 smithi130 ceph-mon[97832]: from='client.24373 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi191:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T19:14:57.993 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:14:57 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-14T19:14:57.993 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:14:57 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-14T19:14:57.993 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:14:57 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:14:57.993 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:14:57 smithi130 ceph-mon[97832]: osdmap e71: 11 total, 11 up, 11 in 2023-07-14T19:14:58.971 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:14:58 smithi040 ceph-mon[96518]: pgmap v213: 1 pgs: 1 remapped+peering; 577 KiB data, 298 MiB used, 983 GiB / 983 GiB avail 2023-07-14T19:14:58.971 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:14:58 smithi040 ceph-mon[96518]: osdmap e72: 11 total, 11 up, 11 in 2023-07-14T19:14:58.972 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:14:58 smithi191 ceph-mon[98311]: pgmap v213: 1 pgs: 1 remapped+peering; 577 KiB data, 298 MiB used, 983 GiB / 983 GiB avail 2023-07-14T19:14:58.972 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:14:58 smithi191 ceph-mon[98311]: osdmap e72: 11 total, 11 up, 11 in 2023-07-14T19:14:58.992 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:14:58 smithi130 ceph-mon[97832]: pgmap v213: 1 pgs: 1 remapped+peering; 577 KiB data, 298 MiB used, 983 GiB / 983 GiB avail 2023-07-14T19:14:58.992 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:14:58 smithi130 ceph-mon[97832]: osdmap e72: 11 total, 11 up, 11 in 2023-07-14T19:14:59.971 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:14:59 smithi040 ceph-mon[96518]: from='client.? 172.21.15.191:0/1497398598' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f43a2a6f-82f3-42ac-b77c-4f1b34da6918"}]: dispatch 2023-07-14T19:14:59.972 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:14:59 smithi040 ceph-mon[96518]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f43a2a6f-82f3-42ac-b77c-4f1b34da6918"}]: dispatch 2023-07-14T19:14:59.972 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:14:59 smithi040 ceph-mon[96518]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f43a2a6f-82f3-42ac-b77c-4f1b34da6918"}]': finished 2023-07-14T19:14:59.972 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:14:59 smithi040 ceph-mon[96518]: osdmap e73: 12 total, 11 up, 12 in 2023-07-14T19:14:59.973 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:14:59 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-07-14T19:14:59.973 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:14:59 smithi191 ceph-mon[98311]: from='client.? 172.21.15.191:0/1497398598' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f43a2a6f-82f3-42ac-b77c-4f1b34da6918"}]: dispatch 2023-07-14T19:14:59.974 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:14:59 smithi191 ceph-mon[98311]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f43a2a6f-82f3-42ac-b77c-4f1b34da6918"}]: dispatch 2023-07-14T19:14:59.974 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:14:59 smithi191 ceph-mon[98311]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f43a2a6f-82f3-42ac-b77c-4f1b34da6918"}]': finished 2023-07-14T19:14:59.975 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:14:59 smithi191 ceph-mon[98311]: osdmap e73: 12 total, 11 up, 12 in 2023-07-14T19:14:59.975 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:14:59 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-07-14T19:14:59.992 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:14:59 smithi130 ceph-mon[97832]: from='client.? 172.21.15.191:0/1497398598' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f43a2a6f-82f3-42ac-b77c-4f1b34da6918"}]: dispatch 2023-07-14T19:14:59.993 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:14:59 smithi130 ceph-mon[97832]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f43a2a6f-82f3-42ac-b77c-4f1b34da6918"}]: dispatch 2023-07-14T19:14:59.993 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:14:59 smithi130 ceph-mon[97832]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f43a2a6f-82f3-42ac-b77c-4f1b34da6918"}]': finished 2023-07-14T19:14:59.993 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:14:59 smithi130 ceph-mon[97832]: osdmap e73: 12 total, 11 up, 12 in 2023-07-14T19:14:59.994 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:14:59 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-07-14T19:15:00.971 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:15:00 smithi040 ceph-mon[96518]: pgmap v216: 1 pgs: 1 remapped+peering; 577 KiB data, 298 MiB used, 983 GiB / 983 GiB avail 2023-07-14T19:15:00.971 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:15:00 smithi040 ceph-mon[96518]: from='client.? 172.21.15.191:0/1552611682' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-14T19:15:00.971 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:15:00 smithi191 ceph-mon[98311]: pgmap v216: 1 pgs: 1 remapped+peering; 577 KiB data, 298 MiB used, 983 GiB / 983 GiB avail 2023-07-14T19:15:00.972 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:15:00 smithi191 ceph-mon[98311]: from='client.? 172.21.15.191:0/1552611682' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-14T19:15:00.992 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:15:00 smithi130 ceph-mon[97832]: pgmap v216: 1 pgs: 1 remapped+peering; 577 KiB data, 298 MiB used, 983 GiB / 983 GiB avail 2023-07-14T19:15:00.992 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:15:00 smithi130 ceph-mon[97832]: from='client.? 172.21.15.191:0/1552611682' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-14T19:15:02.971 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:15:02 smithi040 ceph-mon[96518]: pgmap v218: 1 pgs: 1 remapped+peering; 577 KiB data, 298 MiB used, 983 GiB / 983 GiB avail 2023-07-14T19:15:02.972 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:15:02 smithi191 ceph-mon[98311]: pgmap v218: 1 pgs: 1 remapped+peering; 577 KiB data, 298 MiB used, 983 GiB / 983 GiB avail 2023-07-14T19:15:02.992 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:15:02 smithi130 ceph-mon[97832]: pgmap v218: 1 pgs: 1 remapped+peering; 577 KiB data, 298 MiB used, 983 GiB / 983 GiB avail 2023-07-14T19:15:04.971 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:15:04 smithi040 ceph-mon[96518]: pgmap v219: 1 pgs: 1 active+clean; 577 KiB data, 298 MiB used, 983 GiB / 983 GiB avail; 96 KiB/s, 0 objects/s recovering 2023-07-14T19:15:04.972 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:15:04 smithi191 ceph-mon[98311]: pgmap v219: 1 pgs: 1 active+clean; 577 KiB data, 298 MiB used, 983 GiB / 983 GiB avail; 96 KiB/s, 0 objects/s recovering 2023-07-14T19:15:04.992 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:15:04 smithi130 ceph-mon[97832]: pgmap v219: 1 pgs: 1 active+clean; 577 KiB data, 298 MiB used, 983 GiB / 983 GiB avail; 96 KiB/s, 0 objects/s recovering 2023-07-14T19:15:05.971 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:15:05 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2023-07-14T19:15:05.971 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:15:05 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:15:05.972 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:15:05 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2023-07-14T19:15:05.972 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:15:05 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:15:05.992 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:15:05 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2023-07-14T19:15:05.992 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:15:05 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:15:06.722 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:15:06 smithi191 ceph-mon[98311]: pgmap v220: 1 pgs: 1 active+clean; 577 KiB data, 298 MiB used, 983 GiB / 983 GiB avail; 77 KiB/s, 0 objects/s recovering 2023-07-14T19:15:06.722 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:15:06 smithi191 ceph-mon[98311]: Deploying daemon osd.11 on smithi191 2023-07-14T19:15:06.971 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:15:06 smithi040 ceph-mon[96518]: pgmap v220: 1 pgs: 1 active+clean; 577 KiB data, 298 MiB used, 983 GiB / 983 GiB avail; 77 KiB/s, 0 objects/s recovering 2023-07-14T19:15:06.971 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:15:06 smithi040 ceph-mon[96518]: Deploying daemon osd.11 on smithi191 2023-07-14T19:15:06.992 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:15:06 smithi130 ceph-mon[97832]: pgmap v220: 1 pgs: 1 active+clean; 577 KiB data, 298 MiB used, 983 GiB / 983 GiB avail; 77 KiB/s, 0 objects/s recovering 2023-07-14T19:15:06.992 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:15:06 smithi130 ceph-mon[97832]: Deploying daemon osd.11 on smithi191 2023-07-14T19:15:08.722 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:15:08 smithi191 ceph-mon[98311]: pgmap v221: 1 pgs: 1 active+clean; 577 KiB data, 298 MiB used, 983 GiB / 983 GiB avail; 68 KiB/s, 0 objects/s recovering 2023-07-14T19:15:08.971 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:15:08 smithi040 ceph-mon[96518]: pgmap v221: 1 pgs: 1 active+clean; 577 KiB data, 298 MiB used, 983 GiB / 983 GiB avail; 68 KiB/s, 0 objects/s recovering 2023-07-14T19:15:08.992 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:15:08 smithi130 ceph-mon[97832]: pgmap v221: 1 pgs: 1 active+clean; 577 KiB data, 298 MiB used, 983 GiB / 983 GiB avail; 68 KiB/s, 0 objects/s recovering 2023-07-14T19:15:10.907 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:15:10 smithi191 ceph-mon[98311]: pgmap v222: 1 pgs: 1 active+clean; 577 KiB data, 298 MiB used, 983 GiB / 983 GiB avail; 58 KiB/s, 0 objects/s recovering 2023-07-14T19:15:10.908 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:15:10 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:15:10.908 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:15:10 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:15:10.908 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:15:10 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T19:15:10.971 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:15:10 smithi040 ceph-mon[96518]: pgmap v222: 1 pgs: 1 active+clean; 577 KiB data, 298 MiB used, 983 GiB / 983 GiB avail; 58 KiB/s, 0 objects/s recovering 2023-07-14T19:15:10.971 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:15:10 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:15:10.972 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:15:10 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:15:10.972 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:15:10 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T19:15:10.992 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:15:10 smithi130 ceph-mon[97832]: pgmap v222: 1 pgs: 1 active+clean; 577 KiB data, 298 MiB used, 983 GiB / 983 GiB avail; 58 KiB/s, 0 objects/s recovering 2023-07-14T19:15:10.992 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:15:10 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:15:10.992 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:15:10 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:15:10.993 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:15:10 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T19:15:12.592 INFO:teuthology.orchestra.run.smithi191.stdout:Created osd(s) 11 on host 'smithi191' 2023-07-14T19:15:12.592 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:15:12.589+0000 7f61237fe700 1 -- 172.21.15.191:0/413079700 <== mgr.14152 v2:172.21.15.40:6800/1693027118 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+38 (secure 0 0 0) 0x7f612c060a80 con 0x7f61180a3360 2023-07-14T19:15:12.593 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:15:12.591+0000 7f61344ef700 1 -- 172.21.15.191:0/413079700 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f61180a3360 msgr2=0x7f61180a5810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:15:12.594 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:15:12.591+0000 7f61344ef700 1 --2- 172.21.15.191:0/413079700 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f61180a3360 0x7f61180a5810 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f612c0689f0 tx=0x7f611c00b5f0 comp rx=0 tx=0).stop 2023-07-14T19:15:12.594 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:15:12.592+0000 7f61344ef700 1 -- 172.21.15.191:0/413079700 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f612c1001e0 msgr2=0x7f612c193da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:15:12.594 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:15:12.592+0000 7f61344ef700 1 --2- 172.21.15.191:0/413079700 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f612c1001e0 0x7f612c193da0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f6128005910 tx=0x7f612800f710 comp rx=0 tx=0).stop 2023-07-14T19:15:12.594 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:15:12.592+0000 7f61344ef700 1 -- 172.21.15.191:0/413079700 shutdown_connections 2023-07-14T19:15:12.594 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:15:12.592+0000 7f61344ef700 1 --2- 172.21.15.191:0/413079700 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f61180a3360 0x7f61180a5810 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:15:12.595 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:15:12.592+0000 7f61344ef700 1 --2- 172.21.15.191:0/413079700 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f612c100b30 0x7f612c19ae20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:15:12.595 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:15:12.592+0000 7f61344ef700 1 --2- 172.21.15.191:0/413079700 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f612c0ff5d0 0x7f612c193860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:15:12.595 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:15:12.592+0000 7f61344ef700 1 --2- 172.21.15.191:0/413079700 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f612c1001e0 0x7f612c193da0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:15:12.595 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:15:12.592+0000 7f61344ef700 1 -- 172.21.15.191:0/413079700 >> 172.21.15.191:0/413079700 conn(0x7f612c0f99c0 msgr2=0x7f612c1033d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:15:12.595 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:15:12.592+0000 7f61344ef700 1 -- 172.21.15.191:0/413079700 shutdown_connections 2023-07-14T19:15:12.596 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:15:12.592+0000 7f61344ef700 1 -- 172.21.15.191:0/413079700 wait complete. 2023-07-14T19:15:12.971 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:15:12 smithi040 ceph-mon[96518]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 298 MiB used, 983 GiB / 983 GiB avail; 49 KiB/s, 0 objects/s recovering 2023-07-14T19:15:12.971 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:15:12 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:15:12.972 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:15:12 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:15:12.972 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:15:12 smithi191 ceph-mon[98311]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 298 MiB used, 983 GiB / 983 GiB avail; 49 KiB/s, 0 objects/s recovering 2023-07-14T19:15:12.972 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:15:12 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:15:12.972 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:15:12 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:15:12.974 DEBUG:teuthology.orchestra.run.smithi191:osd.11> sudo journalctl -f -n 0 -u ceph-adc52f04-2279-11ee-9b33-001a4aab830c@osd.11.service 2023-07-14T19:15:12.976 INFO:tasks.cephadm:Waiting for 12 OSDs to come up... 2023-07-14T19:15:12.976 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid adc52f04-2279-11ee-9b33-001a4aab830c -- ceph osd stat -f json 2023-07-14T19:15:12.992 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:15:12 smithi130 ceph-mon[97832]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 298 MiB used, 983 GiB / 983 GiB avail; 49 KiB/s, 0 objects/s recovering 2023-07-14T19:15:12.992 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:15:12 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:15:12.993 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:15:12 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:15:13.005 INFO:journalctl@ceph.osd.11.smithi191.stdout:-- Logs begin at Fri 2023-07-14 18:50:54 UTC. -- 2023-07-14T19:15:13.550 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/adc52f04-2279-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T19:15:13.828 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:15:13 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:15:13.829 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:15:13 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:15:13.947 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:15:13 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:15:13.947 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:15:13 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:15:13.992 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:15:13 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:15:13.992 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:15:13 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:15:14.222 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jul 14 19:15:13 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-11[120628]: 2023-07-14T19:15:13.852+0000 7f5f0737d640 -1 osd.11 0 log_to_monitors true 2023-07-14T19:15:14.797 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:14.793+0000 7fa27fda2700 1 -- 172.21.15.40:0/3287301162 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa2780fe530 msgr2=0x7fa2780fe940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:15:14.798 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:14.793+0000 7fa27fda2700 1 --2- 172.21.15.40:0/3287301162 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa2780fe530 0x7fa2780fe940 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fa274009990 tx=0x7fa274009ca0 comp rx=0 tx=0).stop 2023-07-14T19:15:14.798 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:14.793+0000 7fa27fda2700 1 -- 172.21.15.40:0/3287301162 shutdown_connections 2023-07-14T19:15:14.798 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:14.793+0000 7fa27fda2700 1 --2- 172.21.15.40:0/3287301162 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa2780ff830 0x7fa2780ffca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:15:14.798 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:14.793+0000 7fa27fda2700 1 --2- 172.21.15.40:0/3287301162 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa2780fe530 0x7fa2780fe940 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:15:14.798 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:14.793+0000 7fa27fda2700 1 --2- 172.21.15.40:0/3287301162 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa2780fee80 0x7fa2780ff2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:15:14.798 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:14.793+0000 7fa27fda2700 1 -- 172.21.15.40:0/3287301162 >> 172.21.15.40:0/3287301162 conn(0x7fa2780fa1e0 msgr2=0x7fa2780fc630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:15:14.799 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:14.793+0000 7fa27fda2700 1 -- 172.21.15.40:0/3287301162 shutdown_connections 2023-07-14T19:15:14.799 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:14.793+0000 7fa27fda2700 1 -- 172.21.15.40:0/3287301162 wait complete. 2023-07-14T19:15:14.799 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:14.794+0000 7fa27fda2700 1 Processor -- start 2023-07-14T19:15:14.799 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:14.794+0000 7fa27fda2700 1 -- start start 2023-07-14T19:15:14.799 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:14.794+0000 7fa27fda2700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa2780fe530 0x7fa278197d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:15:14.799 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:14.794+0000 7fa27fda2700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa2780fee80 0x7fa2781982d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:15:14.799 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:14.794+0000 7fa27fda2700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa2780ff830 0x7fa27819f310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:15:14.800 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:14.794+0000 7fa27fda2700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa27819f8e0 con 0x7fa2780fee80 2023-07-14T19:15:14.800 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:14.794+0000 7fa27fda2700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa27819fa20 con 0x7fa2780ff830 2023-07-14T19:15:14.800 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:14.794+0000 7fa27fda2700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa27819fbe0 con 0x7fa2780fe530 2023-07-14T19:15:14.800 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:14.794+0000 7fa27d33d700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa2780fee80 0x7fa2781982d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:15:14.800 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:14.794+0000 7fa27db3e700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa2780fe530 0x7fa278197d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:15:14.800 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:14.794+0000 7fa27d33d700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa2780fee80 0x7fa2781982d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:60700/0 (socket says 172.21.15.40:60700) 2023-07-14T19:15:14.800 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:14.795+0000 7fa27d33d700 1 -- 172.21.15.40:0/2477995299 learned_addr learned my addr 172.21.15.40:0/2477995299 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-07-14T19:15:14.801 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:14.795+0000 7fa27e33f700 1 --2- 172.21.15.40:0/2477995299 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa2780ff830 0x7fa27819f310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:15:14.801 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:14.795+0000 7fa27d33d700 1 -- 172.21.15.40:0/2477995299 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa2780fe530 msgr2=0x7fa278197d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:15:14.801 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:14.795+0000 7fa27d33d700 1 --2- 172.21.15.40:0/2477995299 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa2780fe530 0x7fa278197d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:15:14.801 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:14.795+0000 7fa27d33d700 1 -- 172.21.15.40:0/2477995299 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa2780ff830 msgr2=0x7fa27819f310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:15:14.801 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:14.795+0000 7fa27d33d700 1 --2- 172.21.15.40:0/2477995299 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa2780ff830 0x7fa27819f310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:15:14.801 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:14.795+0000 7fa27d33d700 1 -- 172.21.15.40:0/2477995299 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2740095f0 con 0x7fa2780fee80 2023-07-14T19:15:14.801 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:14.795+0000 7fa27d33d700 1 --2- 172.21.15.40:0/2477995299 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa2780fee80 0x7fa2781982d0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fa26800e9a0 tx=0x7fa26800ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:15:14.801 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:14.795+0000 7fa26effd700 1 -- 172.21.15.40:0/2477995299 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa268009790 con 0x7fa2780fee80 2023-07-14T19:15:14.802 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:14.795+0000 7fa27fda2700 1 -- 172.21.15.40:0/2477995299 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa27819fec0 con 0x7fa2780fee80 2023-07-14T19:15:14.802 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:14.795+0000 7fa26effd700 1 -- 172.21.15.40:0/2477995299 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa268010430 con 0x7fa2780fee80 2023-07-14T19:15:14.802 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:14.795+0000 7fa26effd700 1 -- 172.21.15.40:0/2477995299 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa268018680 con 0x7fa2780fee80 2023-07-14T19:15:14.802 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:14.795+0000 7fa27fda2700 1 -- 172.21.15.40:0/2477995299 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa2781a02d0 con 0x7fa2780fee80 2023-07-14T19:15:14.803 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:14.796+0000 7fa27fda2700 1 -- 172.21.15.40:0/2477995299 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa27810d660 con 0x7fa2780fee80 2023-07-14T19:15:14.803 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:14.797+0000 7fa26effd700 1 -- 172.21.15.40:0/2477995299 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7fa2680105a0 con 0x7fa2780fee80 2023-07-14T19:15:14.803 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:14.798+0000 7fa26effd700 1 --2- 172.21.15.40:0/2477995299 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7fa2640a1150 0x7fa2640a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:15:14.803 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:14.798+0000 7fa26effd700 1 -- 172.21.15.40:0/2477995299 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 6831+0+0 (secure 0 0 0) 0x7fa268014070 con 0x7fa2780fee80 2023-07-14T19:15:14.803 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:14.798+0000 7fa27db3e700 1 --2- 172.21.15.40:0/2477995299 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7fa2640a1150 0x7fa2640a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:15:14.804 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:14.798+0000 7fa27db3e700 1 --2- 172.21.15.40:0/2477995299 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7fa2640a1150 0x7fa2640a3600 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fa27400b2e0 tx=0x7fa274005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:15:14.804 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:14.801+0000 7fa26effd700 1 -- 172.21.15.40:0/2477995299 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fa268096410 con 0x7fa2780fee80 2023-07-14T19:15:14.953 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:15:14 smithi191 ceph-mon[98311]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 298 MiB used, 983 GiB / 983 GiB avail; 48 KiB/s, 0 objects/s recovering 2023-07-14T19:15:14.953 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:15:14 smithi191 ceph-mon[98311]: from='osd.11 [v2:172.21.15.191:6824/2886059446,v1:172.21.15.191:6825/2886059446]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2023-07-14T19:15:14.953 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:15:14 smithi191 ceph-mon[98311]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2023-07-14T19:15:14.971 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:15:14 smithi040 ceph-mon[96518]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 298 MiB used, 983 GiB / 983 GiB avail; 48 KiB/s, 0 objects/s recovering 2023-07-14T19:15:14.971 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:15:14 smithi040 ceph-mon[96518]: from='osd.11 [v2:172.21.15.191:6824/2886059446,v1:172.21.15.191:6825/2886059446]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2023-07-14T19:15:14.971 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:15:14 smithi040 ceph-mon[96518]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2023-07-14T19:15:14.992 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:15:14 smithi130 ceph-mon[97832]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 298 MiB used, 983 GiB / 983 GiB avail; 48 KiB/s, 0 objects/s recovering 2023-07-14T19:15:14.992 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:15:14 smithi130 ceph-mon[97832]: from='osd.11 [v2:172.21.15.191:6824/2886059446,v1:172.21.15.191:6825/2886059446]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2023-07-14T19:15:14.993 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:15:14 smithi130 ceph-mon[97832]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2023-07-14T19:15:15.000 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:14.998+0000 7fa27fda2700 1 -- 172.21.15.40:0/2477995299 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) v1 -- 0x7fa27810d820 con 0x7fa2780fee80 2023-07-14T19:15:15.000 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:14.998+0000 7fa26effd700 1 -- 172.21.15.40:0/2477995299 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v74) v1 ==== 74+0+133 (secure 0 0 0) 0x7fa268099a30 con 0x7fa2780fee80 2023-07-14T19:15:15.001 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-07-14T19:15:15.002 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:15.000+0000 7fa27fda2700 1 -- 172.21.15.40:0/2477995299 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7fa2640a1150 msgr2=0x7fa2640a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:15:15.003 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:15.000+0000 7fa27fda2700 1 --2- 172.21.15.40:0/2477995299 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7fa2640a1150 0x7fa2640a3600 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fa27400b2e0 tx=0x7fa274005fb0 comp rx=0 tx=0).stop 2023-07-14T19:15:15.003 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:15.000+0000 7fa27fda2700 1 -- 172.21.15.40:0/2477995299 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa2780fee80 msgr2=0x7fa2781982d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:15:15.003 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:15.000+0000 7fa27fda2700 1 --2- 172.21.15.40:0/2477995299 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa2780fee80 0x7fa2781982d0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fa26800e9a0 tx=0x7fa26800ecb0 comp rx=0 tx=0).stop 2023-07-14T19:15:15.003 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:15.001+0000 7fa27fda2700 1 -- 172.21.15.40:0/2477995299 shutdown_connections 2023-07-14T19:15:15.003 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:15.001+0000 7fa27fda2700 1 --2- 172.21.15.40:0/2477995299 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7fa2640a1150 0x7fa2640a3600 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:15:15.003 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:15.001+0000 7fa27fda2700 1 --2- 172.21.15.40:0/2477995299 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa2780fe530 0x7fa278197d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:15:15.004 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:15.001+0000 7fa27fda2700 1 --2- 172.21.15.40:0/2477995299 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa2780ff830 0x7fa27819f310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:15:15.004 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:15.001+0000 7fa27fda2700 1 --2- 172.21.15.40:0/2477995299 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa2780fee80 0x7fa2781982d0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:15:15.004 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:15.001+0000 7fa27fda2700 1 -- 172.21.15.40:0/2477995299 >> 172.21.15.40:0/2477995299 conn(0x7fa2780fa1e0 msgr2=0x7fa2780fc1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:15:15.004 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:15.001+0000 7fa27fda2700 1 -- 172.21.15.40:0/2477995299 shutdown_connections 2023-07-14T19:15:15.004 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:15.001+0000 7fa27fda2700 1 -- 172.21.15.40:0/2477995299 wait complete. 2023-07-14T19:15:15.412 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":74,"num_osds":12,"num_up_osds":11,"osd_up_since":1689362095,"num_in_osds":12,"osd_in_since":1689362098,"num_remapped_pgs":0} 2023-07-14T19:15:15.971 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:15:15 smithi040 ceph-mon[96518]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2023-07-14T19:15:15.972 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:15:15 smithi040 ceph-mon[96518]: osdmap e74: 12 total, 11 up, 12 in 2023-07-14T19:15:15.972 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:15:15 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-07-14T19:15:15.972 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:15:15 smithi040 ceph-mon[96518]: from='osd.11 [v2:172.21.15.191:6824/2886059446,v1:172.21.15.191:6825/2886059446]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi191", "root=default"]}]: dispatch 2023-07-14T19:15:15.973 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:15:15 smithi040 ceph-mon[96518]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi191", "root=default"]}]: dispatch 2023-07-14T19:15:15.973 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:15:15 smithi040 ceph-mon[96518]: from='client.? 172.21.15.40:0/2477995299' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-14T19:15:15.973 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:15:15 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:15:15.973 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:15:15 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:15:15.974 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:15:15 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-07-14T19:15:15.974 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:15:15 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-07-14T19:15:15.974 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:15:15 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-07-14T19:15:15.974 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:15:15 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2023-07-14T19:15:15.975 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:15:15 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:15:15.975 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:15:15 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:15:15.975 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:15:15 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T19:15:15.975 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:15:15 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:15:15.975 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:15:15 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T19:15:15.976 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:15:15 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:15:15.976 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:15:15 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T19:15:15.976 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:15:15 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:15:15.977 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jul 14 19:15:15 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-11[120628]: 2023-07-14T19:15:15.637+0000 7f5efc9f5700 -1 osd.11 0 waiting for initial osdmap 2023-07-14T19:15:15.977 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jul 14 19:15:15 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-11[120628]: 2023-07-14T19:15:15.650+0000 7f5ef67e7700 -1 osd.11 75 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-07-14T19:15:15.977 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:15:15 smithi191 ceph-mon[98311]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2023-07-14T19:15:15.978 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:15:15 smithi191 ceph-mon[98311]: osdmap e74: 12 total, 11 up, 12 in 2023-07-14T19:15:15.978 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:15:15 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-07-14T19:15:15.978 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:15:15 smithi191 ceph-mon[98311]: from='osd.11 [v2:172.21.15.191:6824/2886059446,v1:172.21.15.191:6825/2886059446]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi191", "root=default"]}]: dispatch 2023-07-14T19:15:15.978 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:15:15 smithi191 ceph-mon[98311]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi191", "root=default"]}]: dispatch 2023-07-14T19:15:15.979 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:15:15 smithi191 ceph-mon[98311]: from='client.? 172.21.15.40:0/2477995299' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-14T19:15:15.979 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:15:15 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:15:15.979 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:15:15 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:15:15.979 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:15:15 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-07-14T19:15:15.979 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:15:15 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-07-14T19:15:15.980 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:15:15 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-07-14T19:15:15.980 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:15:15 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2023-07-14T19:15:15.980 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:15:15 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:15:15.980 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:15:15 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:15:15.981 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:15:15 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T19:15:15.981 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:15:15 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:15:15.981 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:15:15 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T19:15:15.981 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:15:15 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:15:15.981 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:15:15 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T19:15:15.982 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:15:15 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:15:15.992 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:15:15 smithi130 ceph-mon[97832]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2023-07-14T19:15:15.992 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:15:15 smithi130 ceph-mon[97832]: osdmap e74: 12 total, 11 up, 12 in 2023-07-14T19:15:15.993 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:15:15 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-07-14T19:15:15.993 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:15:15 smithi130 ceph-mon[97832]: from='osd.11 [v2:172.21.15.191:6824/2886059446,v1:172.21.15.191:6825/2886059446]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi191", "root=default"]}]: dispatch 2023-07-14T19:15:15.993 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:15:15 smithi130 ceph-mon[97832]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi191", "root=default"]}]: dispatch 2023-07-14T19:15:15.993 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:15:15 smithi130 ceph-mon[97832]: from='client.? 172.21.15.40:0/2477995299' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-14T19:15:15.993 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:15:15 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:15:15.993 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:15:15 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:15:15.994 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:15:15 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-07-14T19:15:15.994 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:15:15 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-07-14T19:15:15.994 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:15:15 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-07-14T19:15:15.994 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:15:15 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2023-07-14T19:15:15.994 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:15:15 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:15:15.994 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:15:15 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:15:15.994 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:15:15 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T19:15:15.994 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:15:15 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:15:15.995 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:15:15 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T19:15:15.995 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:15:15 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:15:15.995 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:15:15 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T19:15:15.995 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:15:15 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:15:16.413 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid adc52f04-2279-11ee-9b33-001a4aab830c -- ceph osd stat -f json 2023-07-14T19:15:16.721 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:15:16 smithi040 ceph-mon[96518]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 298 MiB used, 983 GiB / 983 GiB avail 2023-07-14T19:15:16.721 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:15:16 smithi040 ceph-mon[96518]: Detected new or changed devices on smithi191 2023-07-14T19:15:16.722 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:15:16 smithi040 ceph-mon[96518]: Adjusting osd_memory_target on smithi191 to 4292M 2023-07-14T19:15:16.722 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:15:16 smithi040 ceph-mon[96518]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi191", "root=default"]}]': finished 2023-07-14T19:15:16.722 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:15:16 smithi040 ceph-mon[96518]: osdmap e75: 12 total, 11 up, 12 in 2023-07-14T19:15:16.723 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:15:16 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-07-14T19:15:16.723 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:15:16 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-07-14T19:15:16.972 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:15:16 smithi191 ceph-mon[98311]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 298 MiB used, 983 GiB / 983 GiB avail 2023-07-14T19:15:16.972 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:15:16 smithi191 ceph-mon[98311]: Detected new or changed devices on smithi191 2023-07-14T19:15:16.972 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:15:16 smithi191 ceph-mon[98311]: Adjusting osd_memory_target on smithi191 to 4292M 2023-07-14T19:15:16.972 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:15:16 smithi191 ceph-mon[98311]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi191", "root=default"]}]': finished 2023-07-14T19:15:16.973 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:15:16 smithi191 ceph-mon[98311]: osdmap e75: 12 total, 11 up, 12 in 2023-07-14T19:15:16.973 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:15:16 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-07-14T19:15:16.973 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:15:16 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-07-14T19:15:16.991 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/adc52f04-2279-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T19:15:16.992 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:15:16 smithi130 ceph-mon[97832]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 298 MiB used, 983 GiB / 983 GiB avail 2023-07-14T19:15:16.992 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:15:16 smithi130 ceph-mon[97832]: Detected new or changed devices on smithi191 2023-07-14T19:15:16.992 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:15:16 smithi130 ceph-mon[97832]: Adjusting osd_memory_target on smithi191 to 4292M 2023-07-14T19:15:16.992 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:15:16 smithi130 ceph-mon[97832]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi191", "root=default"]}]': finished 2023-07-14T19:15:16.993 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:15:16 smithi130 ceph-mon[97832]: osdmap e75: 12 total, 11 up, 12 in 2023-07-14T19:15:16.993 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:15:16 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-07-14T19:15:16.993 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:15:16 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-07-14T19:15:17.721 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:15:17 smithi040 ceph-mon[96518]: purged_snaps scrub starts 2023-07-14T19:15:17.721 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:15:17 smithi040 ceph-mon[96518]: purged_snaps scrub ok 2023-07-14T19:15:17.721 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:15:17 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-07-14T19:15:17.721 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:15:17 smithi040 ceph-mon[96518]: osd.11 [v2:172.21.15.191:6824/2886059446,v1:172.21.15.191:6825/2886059446] boot 2023-07-14T19:15:17.722 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:15:17 smithi040 ceph-mon[96518]: osdmap e76: 12 total, 12 up, 12 in 2023-07-14T19:15:17.722 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:15:17 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-07-14T19:15:17.972 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:15:17 smithi191 ceph-mon[98311]: purged_snaps scrub starts 2023-07-14T19:15:17.972 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:15:17 smithi191 ceph-mon[98311]: purged_snaps scrub ok 2023-07-14T19:15:17.972 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:15:17 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-07-14T19:15:17.972 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:15:17 smithi191 ceph-mon[98311]: osd.11 [v2:172.21.15.191:6824/2886059446,v1:172.21.15.191:6825/2886059446] boot 2023-07-14T19:15:17.973 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:15:17 smithi191 ceph-mon[98311]: osdmap e76: 12 total, 12 up, 12 in 2023-07-14T19:15:17.973 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:15:17 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-07-14T19:15:17.995 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:15:17 smithi130 ceph-mon[97832]: purged_snaps scrub starts 2023-07-14T19:15:17.995 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:15:17 smithi130 ceph-mon[97832]: purged_snaps scrub ok 2023-07-14T19:15:17.996 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:15:17 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-07-14T19:15:17.996 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:15:17 smithi130 ceph-mon[97832]: osd.11 [v2:172.21.15.191:6824/2886059446,v1:172.21.15.191:6825/2886059446] boot 2023-07-14T19:15:17.996 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:15:17 smithi130 ceph-mon[97832]: osdmap e76: 12 total, 12 up, 12 in 2023-07-14T19:15:17.996 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:15:17 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-07-14T19:15:18.193 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:18.189+0000 7fc43b43a700 1 -- 172.21.15.40:0/3745022881 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc4340febb0 msgr2=0x7fc4340fefc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:15:18.193 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:18.189+0000 7fc43b43a700 1 --2- 172.21.15.40:0/3745022881 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc4340febb0 0x7fc4340fefc0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fc430009990 tx=0x7fc430009ca0 comp rx=0 tx=0).stop 2023-07-14T19:15:18.193 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:18.189+0000 7fc43b43a700 1 -- 172.21.15.40:0/3745022881 shutdown_connections 2023-07-14T19:15:18.194 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:18.189+0000 7fc43b43a700 1 --2- 172.21.15.40:0/3745022881 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc434100740 0x7fc434107000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:15:18.194 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:18.189+0000 7fc43b43a700 1 --2- 172.21.15.40:0/3745022881 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc4340febb0 0x7fc4340fefc0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:15:18.194 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:18.189+0000 7fc43b43a700 1 --2- 172.21.15.40:0/3745022881 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc4340ffdb0 0x7fc434100200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:15:18.194 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:18.189+0000 7fc43b43a700 1 -- 172.21.15.40:0/3745022881 >> 172.21.15.40:0/3745022881 conn(0x7fc4340fa1e0 msgr2=0x7fc4340fc630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:15:18.194 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:18.189+0000 7fc43b43a700 1 -- 172.21.15.40:0/3745022881 shutdown_connections 2023-07-14T19:15:18.194 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:18.190+0000 7fc43b43a700 1 -- 172.21.15.40:0/3745022881 wait complete. 2023-07-14T19:15:18.194 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:18.190+0000 7fc43b43a700 1 Processor -- start 2023-07-14T19:15:18.195 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:18.190+0000 7fc43b43a700 1 -- start start 2023-07-14T19:15:18.195 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:18.190+0000 7fc43b43a700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc4340febb0 0x7fc434068c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:15:18.195 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:18.190+0000 7fc43b43a700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc4340ffdb0 0x7fc43419a720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:15:18.195 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:18.190+0000 7fc43b43a700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc434100740 0x7fc43419cd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:15:18.195 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:18.190+0000 7fc43b43a700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc43419d510 con 0x7fc4340febb0 2023-07-14T19:15:18.195 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:18.190+0000 7fc43b43a700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc43419d680 con 0x7fc434100740 2023-07-14T19:15:18.195 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:18.190+0000 7fc43b43a700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc43419d840 con 0x7fc4340ffdb0 2023-07-14T19:15:18.196 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:18.190+0000 7fc4391d6700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc4340febb0 0x7fc434068c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:15:18.196 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:18.190+0000 7fc4391d6700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc4340febb0 0x7fc434068c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:60714/0 (socket says 172.21.15.40:60714) 2023-07-14T19:15:18.196 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:18.190+0000 7fc4391d6700 1 -- 172.21.15.40:0/3443179276 learned_addr learned my addr 172.21.15.40:0/3443179276 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-07-14T19:15:18.196 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:18.191+0000 7fc4399d7700 1 --2- 172.21.15.40:0/3443179276 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc434100740 0x7fc43419cd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:15:18.196 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:18.191+0000 7fc4389d5700 1 --2- 172.21.15.40:0/3443179276 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc4340ffdb0 0x7fc43419a720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:15:18.196 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:18.191+0000 7fc4391d6700 1 -- 172.21.15.40:0/3443179276 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc4340ffdb0 msgr2=0x7fc43419a720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:15:18.197 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:18.191+0000 7fc4391d6700 1 --2- 172.21.15.40:0/3443179276 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc4340ffdb0 0x7fc43419a720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:15:18.197 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:18.191+0000 7fc4391d6700 1 -- 172.21.15.40:0/3443179276 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc434100740 msgr2=0x7fc43419cd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:15:18.197 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:18.191+0000 7fc4391d6700 1 --2- 172.21.15.40:0/3443179276 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc434100740 0x7fc43419cd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:15:18.197 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:18.191+0000 7fc4391d6700 1 -- 172.21.15.40:0/3443179276 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4300095f0 con 0x7fc4340febb0 2023-07-14T19:15:18.197 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:18.191+0000 7fc4391d6700 1 --2- 172.21.15.40:0/3443179276 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc4340febb0 0x7fc434068c80 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fc4300095c0 tx=0x7fc43000f690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:15:18.197 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:18.191+0000 7fc42a7fc700 1 -- 172.21.15.40:0/3443179276 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc43001d070 con 0x7fc4340febb0 2023-07-14T19:15:18.197 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:18.191+0000 7fc42a7fc700 1 -- 172.21.15.40:0/3443179276 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc43000fb80 con 0x7fc4340febb0 2023-07-14T19:15:18.198 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:18.191+0000 7fc43b43a700 1 -- 172.21.15.40:0/3443179276 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc43419dac0 con 0x7fc4340febb0 2023-07-14T19:15:18.198 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:18.191+0000 7fc42a7fc700 1 -- 172.21.15.40:0/3443179276 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc430017650 con 0x7fc4340febb0 2023-07-14T19:15:18.198 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:18.191+0000 7fc43b43a700 1 -- 172.21.15.40:0/3443179276 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc43419df30 con 0x7fc4340febb0 2023-07-14T19:15:18.199 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:18.192+0000 7fc43b43a700 1 -- 172.21.15.40:0/3443179276 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc434106200 con 0x7fc4340febb0 2023-07-14T19:15:18.199 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:18.193+0000 7fc42a7fc700 1 -- 172.21.15.40:0/3443179276 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7fc430022470 con 0x7fc4340febb0 2023-07-14T19:15:18.199 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:18.194+0000 7fc42a7fc700 1 --2- 172.21.15.40:0/3443179276 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7fc4200a11d0 0x7fc4200a3680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:15:18.199 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:18.194+0000 7fc4389d5700 1 --2- 172.21.15.40:0/3443179276 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7fc4200a11d0 0x7fc4200a3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:15:18.200 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:18.194+0000 7fc42a7fc700 1 -- 172.21.15.40:0/3443179276 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fc4300ce0b0 con 0x7fc4340febb0 2023-07-14T19:15:18.200 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:18.194+0000 7fc4389d5700 1 --2- 172.21.15.40:0/3443179276 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7fc4200a11d0 0x7fc4200a3680 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fc424005950 tx=0x7fc4240058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:15:18.200 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:18.197+0000 7fc42a7fc700 1 -- 172.21.15.40:0/3443179276 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fc430097d30 con 0x7fc4340febb0 2023-07-14T19:15:18.396 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:18.394+0000 7fc43b43a700 1 -- 172.21.15.40:0/3443179276 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) v1 -- 0x7fc43402cab0 con 0x7fc4340febb0 2023-07-14T19:15:18.396 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:18.394+0000 7fc42a7fc700 1 -- 172.21.15.40:0/3443179276 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v77) v1 ==== 74+0+133 (secure 0 0 0) 0x7fc43009b350 con 0x7fc4340febb0 2023-07-14T19:15:18.396 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-07-14T19:15:18.398 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:18.396+0000 7fc43b43a700 1 -- 172.21.15.40:0/3443179276 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7fc4200a11d0 msgr2=0x7fc4200a3680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:15:18.398 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:18.396+0000 7fc43b43a700 1 --2- 172.21.15.40:0/3443179276 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7fc4200a11d0 0x7fc4200a3680 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fc424005950 tx=0x7fc4240058e0 comp rx=0 tx=0).stop 2023-07-14T19:15:18.398 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:18.396+0000 7fc43b43a700 1 -- 172.21.15.40:0/3443179276 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc4340febb0 msgr2=0x7fc434068c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:15:18.399 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:18.396+0000 7fc43b43a700 1 --2- 172.21.15.40:0/3443179276 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc4340febb0 0x7fc434068c80 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fc4300095c0 tx=0x7fc43000f690 comp rx=0 tx=0).stop 2023-07-14T19:15:18.399 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:18.396+0000 7fc43b43a700 1 -- 172.21.15.40:0/3443179276 shutdown_connections 2023-07-14T19:15:18.399 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:18.396+0000 7fc43b43a700 1 --2- 172.21.15.40:0/3443179276 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7fc4200a11d0 0x7fc4200a3680 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:15:18.399 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:18.396+0000 7fc43b43a700 1 --2- 172.21.15.40:0/3443179276 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc434100740 0x7fc43419cd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:15:18.399 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:18.396+0000 7fc43b43a700 1 --2- 172.21.15.40:0/3443179276 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc4340febb0 0x7fc434068c80 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:15:18.399 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:18.396+0000 7fc43b43a700 1 --2- 172.21.15.40:0/3443179276 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc4340ffdb0 0x7fc43419a720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:15:18.399 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:18.396+0000 7fc43b43a700 1 -- 172.21.15.40:0/3443179276 >> 172.21.15.40:0/3443179276 conn(0x7fc4340fa1e0 msgr2=0x7fc434103340 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:15:18.400 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:18.396+0000 7fc43b43a700 1 -- 172.21.15.40:0/3443179276 shutdown_connections 2023-07-14T19:15:18.400 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:18.397+0000 7fc43b43a700 1 -- 172.21.15.40:0/3443179276 wait complete. 2023-07-14T19:15:18.721 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:15:18 smithi040 ceph-mon[96518]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 983 GiB / 983 GiB avail 2023-07-14T19:15:18.721 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:15:18 smithi040 ceph-mon[96518]: osdmap e77: 12 total, 12 up, 12 in 2023-07-14T19:15:18.722 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:15:18 smithi040 ceph-mon[96518]: from='client.? 172.21.15.40:0/3443179276' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-14T19:15:18.853 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":77,"num_osds":12,"num_up_osds":12,"osd_up_since":1689362116,"num_in_osds":12,"osd_in_since":1689362098,"num_remapped_pgs":0} 2023-07-14T19:15:18.854 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell --fsid adc52f04-2279-11ee-9b33-001a4aab830c -- ceph osd dump --format=json 2023-07-14T19:15:18.972 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:15:18 smithi191 ceph-mon[98311]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 983 GiB / 983 GiB avail 2023-07-14T19:15:18.972 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:15:18 smithi191 ceph-mon[98311]: osdmap e77: 12 total, 12 up, 12 in 2023-07-14T19:15:18.972 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:15:18 smithi191 ceph-mon[98311]: from='client.? 172.21.15.40:0/3443179276' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-14T19:15:18.992 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:15:18 smithi130 ceph-mon[97832]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 983 GiB / 983 GiB avail 2023-07-14T19:15:18.992 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:15:18 smithi130 ceph-mon[97832]: osdmap e77: 12 total, 12 up, 12 in 2023-07-14T19:15:18.992 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:15:18 smithi130 ceph-mon[97832]: from='client.? 172.21.15.40:0/3443179276' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-14T19:15:19.410 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/adc52f04-2279-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T19:15:20.717 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:20.714+0000 7f0299627700 1 -- 172.21.15.40:0/3427251406 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f02940ff580 msgr2=0x7f02940ff950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:15:20.718 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:20.714+0000 7f0299627700 1 --2- 172.21.15.40:0/3427251406 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f02940ff580 0x7f02940ff950 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f02880099c0 tx=0x7f0288009cd0 comp rx=0 tx=0).stop 2023-07-14T19:15:20.718 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:20.714+0000 7f0299627700 1 -- 172.21.15.40:0/3427251406 shutdown_connections 2023-07-14T19:15:20.718 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:20.714+0000 7f0299627700 1 --2- 172.21.15.40:0/3427251406 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0294103f50 0x7f029410bb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:15:20.718 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:20.714+0000 7f0299627700 1 --2- 172.21.15.40:0/3427251406 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f02940ff580 0x7f02940ff950 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:15:20.718 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:20.714+0000 7f0299627700 1 --2- 172.21.15.40:0/3427251406 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f02940fff20 0x7f0294103820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:15:20.718 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:20.714+0000 7f0299627700 1 -- 172.21.15.40:0/3427251406 >> 172.21.15.40:0/3427251406 conn(0x7f0294075280 msgr2=0x7f0294075680 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:15:20.719 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:20.714+0000 7f0299627700 1 -- 172.21.15.40:0/3427251406 shutdown_connections 2023-07-14T19:15:20.719 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:20.714+0000 7f0299627700 1 -- 172.21.15.40:0/3427251406 wait complete. 2023-07-14T19:15:20.719 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:20.714+0000 7f0299627700 1 Processor -- start 2023-07-14T19:15:20.719 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:20.715+0000 7f0299627700 1 -- start start 2023-07-14T19:15:20.719 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:20.715+0000 7f0299627700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f02940ff580 0x7f029419aa50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:15:20.719 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:20.715+0000 7f0299627700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f02940fff20 0x7f029419af90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:15:20.719 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:20.715+0000 7f0299627700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0294103f50 0x7f0294194a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:15:20.720 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:20.715+0000 7f0299627700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02941951f0 con 0x7f02940ff580 2023-07-14T19:15:20.720 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:20.715+0000 7f0299627700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0294195360 con 0x7f02940fff20 2023-07-14T19:15:20.720 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:20.715+0000 7f0299627700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0294195520 con 0x7f0294103f50 2023-07-14T19:15:20.720 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:20.715+0000 7f0292ffd700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f02940ff580 0x7f029419aa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:15:20.720 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:20.715+0000 7f0292ffd700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f02940ff580 0x7f029419aa50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:60730/0 (socket says 172.21.15.40:60730) 2023-07-14T19:15:20.721 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:20.715+0000 7f0292ffd700 1 -- 172.21.15.40:0/682808349 learned_addr learned my addr 172.21.15.40:0/682808349 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-07-14T19:15:20.721 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:20.715+0000 7f02927fc700 1 --2- 172.21.15.40:0/682808349 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f02940fff20 0x7f029419af90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:15:20.721 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:20.715+0000 7f0292ffd700 1 -- 172.21.15.40:0/682808349 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0294103f50 msgr2=0x7f0294194a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:15:20.721 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:20.715+0000 7f0292ffd700 1 --2- 172.21.15.40:0/682808349 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0294103f50 0x7f0294194a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:15:20.721 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:20.715+0000 7f0292ffd700 1 -- 172.21.15.40:0/682808349 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f02940fff20 msgr2=0x7f029419af90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:15:20.721 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:20.715+0000 7f0292ffd700 1 --2- 172.21.15.40:0/682808349 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f02940fff20 0x7f029419af90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:15:20.722 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:20.715+0000 7f0292ffd700 1 -- 172.21.15.40:0/682808349 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02880095f0 con 0x7f02940ff580 2023-07-14T19:15:20.722 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:20.715+0000 7f02927fc700 1 --2- 172.21.15.40:0/682808349 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f02940fff20 0x7f029419af90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-14T19:15:20.722 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:20.716+0000 7f0292ffd700 1 --2- 172.21.15.40:0/682808349 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f02940ff580 0x7f029419aa50 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f0288009990 tx=0x7f0288009580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:15:20.722 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:20.716+0000 7f027bfff700 1 -- 172.21.15.40:0/682808349 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f028801d070 con 0x7f02940ff580 2023-07-14T19:15:20.722 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:20.716+0000 7f0299627700 1 -- 172.21.15.40:0/682808349 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f02941957a0 con 0x7f02940ff580 2023-07-14T19:15:20.722 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:20.716+0000 7f027bfff700 1 -- 172.21.15.40:0/682808349 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f0288004010 con 0x7f02940ff580 2023-07-14T19:15:20.723 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:20.716+0000 7f027bfff700 1 -- 172.21.15.40:0/682808349 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0288022d40 con 0x7f02940ff580 2023-07-14T19:15:20.724 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:20.716+0000 7f0299627700 1 -- 172.21.15.40:0/682808349 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f02941a19d0 con 0x7f02940ff580 2023-07-14T19:15:20.724 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:20.717+0000 7f0299627700 1 -- 172.21.15.40:0/682808349 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f029404e340 con 0x7f02940ff580 2023-07-14T19:15:20.724 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:20.718+0000 7f027bfff700 1 -- 172.21.15.40:0/682808349 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f0288022470 con 0x7f02940ff580 2023-07-14T19:15:20.724 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:20.718+0000 7f027bfff700 1 --2- 172.21.15.40:0/682808349 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f027c0a1150 0x7f027c0a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:15:20.724 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:20.719+0000 7f027bfff700 1 -- 172.21.15.40:0/682808349 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f02880cdc60 con 0x7f02940ff580 2023-07-14T19:15:20.725 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:20.719+0000 7f02927fc700 1 --2- 172.21.15.40:0/682808349 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f027c0a1150 0x7f027c0a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:15:20.725 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:20.719+0000 7f02927fc700 1 --2- 172.21.15.40:0/682808349 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f027c0a1150 0x7f027c0a3600 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f0284005fd0 tx=0x7f0284005ee0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:15:20.725 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:20.722+0000 7f027bfff700 1 -- 172.21.15.40:0/682808349 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f02880978e0 con 0x7f02940ff580 2023-07-14T19:15:20.922 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:20.920+0000 7f0299627700 1 -- 172.21.15.40:0/682808349 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0294195f70 con 0x7f02940ff580 2023-07-14T19:15:20.922 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:20.921+0000 7f027bfff700 1 -- 172.21.15.40:0/682808349 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v77) v1 ==== 74+0+18424 (secure 0 0 0) 0x7f028809af00 con 0x7f02940ff580 2023-07-14T19:15:20.923 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-07-14T19:15:20.923 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":77,"fsid":"adc52f04-2279-11ee-9b33-001a4aab830c","created":"2023-07-14T19:08:56.893730+0000","modified":"2023-07-14T19:15:17.644846+0000","last_up_change":"2023-07-14T19:15:16.632979+0000","last_in_change":"2023-07-14T19:14:58.961702+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-07-14T19:12:03.271333+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"0d79356a-95d2-49ad-857b-2092ae9f7e13","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":3599703125},{"type":"v1","addr":"172.21.15.40:6803","nonce":3599703125}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":3599703125},{"type":"v1","addr":"172.21.15.40:6805","nonce":3599703125}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":3599703125},{"type":"v1","addr":"172.21.15.40:6809","nonce":3599703125}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":3599703125},{"type":"v1","addr":"172.21.15.40:6807","nonce":3599703125}]},"public_addr":"172.21.15.40:6803/3599703125","cluster_addr":"172.21.15.40:6805/3599703125","heartbeat_back_addr":"172.21.15.40:6809/3599703125","heartbeat_front_addr":"172.21.15.40:6807/3599703125","state":["exists","up"]},{"osd":1,"uuid":"40f8a3d1-a5e4-473c-b9f2-7600ec22be0d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6800","nonce":2512760702},{"type":"v1","addr":"172.21.15.130:6801","nonce":2512760702}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6802","nonce":2512760702},{"type":"v1","addr":"172.21.15.130:6803","nonce":2512760702}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6806","nonce":2512760702},{"type":"v1","addr":"172.21.15.130:6807","nonce":2512760702}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6804","nonce":2512760702},{"type":"v1","addr":"172.21.15.130:6805","nonce":2512760702}]},"public_addr":"172.21.15.130:6801/2512760702","cluster_addr":"172.21.15.130:6803/2512760702","heartbeat_back_addr":"172.21.15.130:6807/2512760702","heartbeat_front_addr":"172.21.15.130:6805/2512760702","state":["exists","up"]},{"osd":2,"uuid":"5a157891-cb7a-4ea6-9239-f48a7c67c003","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6800","nonce":589569135},{"type":"v1","addr":"172.21.15.191:6801","nonce":589569135}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6802","nonce":589569135},{"type":"v1","addr":"172.21.15.191:6803","nonce":589569135}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6806","nonce":589569135},{"type":"v1","addr":"172.21.15.191:6807","nonce":589569135}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6804","nonce":589569135},{"type":"v1","addr":"172.21.15.191:6805","nonce":589569135}]},"public_addr":"172.21.15.191:6801/589569135","cluster_addr":"172.21.15.191:6803/589569135","heartbeat_back_addr":"172.21.15.191:6807/589569135","heartbeat_front_addr":"172.21.15.191:6805/589569135","state":["exists","up"]},{"osd":3,"uuid":"8020e7dc-ac40-487f-857a-8da40bccb459","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":996287860},{"type":"v1","addr":"172.21.15.40:6811","nonce":996287860}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6812","nonce":996287860},{"type":"v1","addr":"172.21.15.40:6813","nonce":996287860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6816","nonce":996287860},{"type":"v1","addr":"172.21.15.40:6817","nonce":996287860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":996287860},{"type":"v1","addr":"172.21.15.40:6815","nonce":996287860}]},"public_addr":"172.21.15.40:6811/996287860","cluster_addr":"172.21.15.40:6813/996287860","heartbeat_back_addr":"172.21.15.40:6817/996287860","heartbeat_front_addr":"172.21.15.40:6815/996287860","state":["exists","up"]},{"osd":4,"uuid":"c5501b35-f62d-482d-8d22-115c88f0298e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6808","nonce":3507448720},{"type":"v1","addr":"172.21.15.130:6809","nonce":3507448720}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6810","nonce":3507448720},{"type":"v1","addr":"172.21.15.130:6811","nonce":3507448720}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6814","nonce":3507448720},{"type":"v1","addr":"172.21.15.130:6815","nonce":3507448720}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6812","nonce":3507448720},{"type":"v1","addr":"172.21.15.130:6813","nonce":3507448720}]},"public_addr":"172.21.15.130:6809/3507448720","cluster_addr":"172.21.15.130:6811/3507448720","heartbeat_back_addr":"172.21.15.130:6815/3507448720","heartbeat_front_addr":"172.21.15.130:6813/3507448720","state":["exists","up"]},{"osd":5,"uuid":"7770b6d6-5874-493a-986a-84db482cee42","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6808","nonce":1781944663},{"type":"v1","addr":"172.21.15.191:6809","nonce":1781944663}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6810","nonce":1781944663},{"type":"v1","addr":"172.21.15.191:6811","nonce":1781944663}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6814","nonce":1781944663},{"type":"v1","addr":"172.21.15.191:6815","nonce":1781944663}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6812","nonce":1781944663},{"type":"v1","addr":"172.21.15.191:6813","nonce":1781944663}]},"public_addr":"172.21.15.191:6809/1781944663","cluster_addr":"172.21.15.191:6811/1781944663","heartbeat_back_addr":"172.21.15.191:6815/1781944663","heartbeat_front_addr":"172.21.15.191:6813/1781944663","state":["exists","up"]},{"osd":6,"uuid":"ef65ad00-dfee-4300-9961-dfc8573aab06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6818","nonce":693186773},{"type":"v1","addr":"172.21.15.40:6819","nonce":693186773}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6820","nonce":693186773},{"type":"v1","addr":"172.21.15.40:6821","nonce":693186773}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6824","nonce":693186773},{"type":"v1","addr":"172.21.15.40:6825","nonce":693186773}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6822","nonce":693186773},{"type":"v1","addr":"172.21.15.40:6823","nonce":693186773}]},"public_addr":"172.21.15.40:6819/693186773","cluster_addr":"172.21.15.40:6821/693186773","heartbeat_back_addr":"172.21.15.40:6825/693186773","heartbeat_front_addr":"172.21.15.40:6823/693186773","state":["exists","up"]},{"osd":7,"uuid":"1700fbfa-cbfc-4f76-97be-727a3b100740","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":50,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6816","nonce":2384019528},{"type":"v1","addr":"172.21.15.130:6817","nonce":2384019528}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6818","nonce":2384019528},{"type":"v1","addr":"172.21.15.130:6819","nonce":2384019528}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6822","nonce":2384019528},{"type":"v1","addr":"172.21.15.130:6823","nonce":2384019528}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6820","nonce":2384019528},{"type":"v1","addr":"172.21.15.130:6821","nonce":2384019528}]},"public_addr":"172.21.15.130:6817/2384019528","cluster_addr":"172.21.15.130:6819/2384019528","heartbeat_back_addr":"172.21.15.130:6823/2384019528","heartbeat_front_addr":"172.21.15.130:6821/2384019528","state":["exists","up"]},{"osd":8,"uuid":"b76d4033-c1e2-457b-8571-c8a49fe443f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":57,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6816","nonce":2386818417},{"type":"v1","addr":"172.21.15.191:6817","nonce":2386818417}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6818","nonce":2386818417},{"type":"v1","addr":"172.21.15.191:6819","nonce":2386818417}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6822","nonce":2386818417},{"type":"v1","addr":"172.21.15.191:6823","nonce":2386818417}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6820","nonce":2386818417},{"type":"v1","addr":"172.21.15.191:6821","nonce":2386818417}]},"public_addr":"172.21.15.191:6817/2386818417","cluster_addr":"172.21.15.191:6819/2386818417","heartbeat_back_addr":"172.21.15.191:6823/2386818417","heartbeat_front_addr":"172.21.15.191:6821/2386818417","state":["exists","up"]},{"osd":9,"uuid":"df31bf2f-e30e-41b6-b2b9-0e26a07a88bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":64,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6826","nonce":2228380263},{"type":"v1","addr":"172.21.15.40:6827","nonce":2228380263}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6828","nonce":2228380263},{"type":"v1","addr":"172.21.15.40:6829","nonce":2228380263}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":2228380263},{"type":"v1","addr":"172.21.15.40:6833","nonce":2228380263}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6830","nonce":2228380263},{"type":"v1","addr":"172.21.15.40:6831","nonce":2228380263}]},"public_addr":"172.21.15.40:6827/2228380263","cluster_addr":"172.21.15.40:6829/2228380263","heartbeat_back_addr":"172.21.15.40:6833/2228380263","heartbeat_front_addr":"172.21.15.40:6831/2228380263","state":["exists","up"]},{"osd":10,"uuid":"4a93d621-49ee-4ad7-9e49-750614faf912","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6824","nonce":2417067798},{"type":"v1","addr":"172.21.15.130:6825","nonce":2417067798}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6826","nonce":2417067798},{"type":"v1","addr":"172.21.15.130:6827","nonce":2417067798}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6830","nonce":2417067798},{"type":"v1","addr":"172.21.15.130:6831","nonce":2417067798}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6828","nonce":2417067798},{"type":"v1","addr":"172.21.15.130:6829","nonce":2417067798}]},"public_addr":"172.21.15.130:6825/2417067798","cluster_addr":"172.21.15.130:6827/2417067798","heartbeat_back_addr":"172.21.15.130:6831/2417067798","heartbeat_front_addr":"172.21.15.130:6829/2417067798","state":["exists","up"]},{"osd":11,"uuid":"f43a2a6f-82f3-42ac-b77c-4f1b34da6918","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":76,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6824","nonce":2886059446},{"type":"v1","addr":"172.21.15.191:6825","nonce":2886059446}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6826","nonce":2886059446},{"type":"v1","addr":"172.21.15.191:6827","nonce":2886059446}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6830","nonce":2886059446},{"type":"v1","addr":"172.21.15.191:6831","nonce":2886059446}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6828","nonce":2886059446},{"type":"v1","addr":"172.21.15.191:6829","nonce":2886059446}]},"public_addr":"172.21.15.191:6825/2886059446","cluster_addr":"172.21.15.191:6827/2886059446","heartbeat_back_addr":"172.21.15.191:6831/2886059446","heartbeat_front_addr":"172.21.15.191:6829/2886059446","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:11:18.320495+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:11:39.901199+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:12:00.869042+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:12:23.292742+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:12:44.587019+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:13:06.494893+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:13:28.378374+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:13:50.033491+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:14:10.937062+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:14:32.358498+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:14:53.666584+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:15:14.837566+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.40:6801/3698757264":"2023-07-15T19:09:50.732458+0000","172.21.15.40:6800/3698757264":"2023-07-15T19:09:50.732458+0000","172.21.15.40:6800/2569015667":"2023-07-15T19:09:24.441543+0000","172.21.15.40:0/3653961003":"2023-07-15T19:09:24.441543+0000","172.21.15.40:0/3568861680":"2023-07-15T19:09:24.441543+0000","172.21.15.40:0/3249773172":"2023-07-15T19:09:50.732458+0000","172.21.15.40:0/4084215480":"2023-07-15T19:09:24.441543+0000","172.21.15.40:0/1163941760":"2023-07-15T19:09:50.732458+0000","172.21.15.40:6801/2569015667":"2023-07-15T19:09:24.441543+0000","172.21.15.40:0/2326639960":"2023-07-15T19:09:50.732458+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-07-14T19:15:20.925 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:20.924+0000 7f0299627700 1 -- 172.21.15.40:0/682808349 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f027c0a1150 msgr2=0x7f027c0a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:15:20.925 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:20.924+0000 7f0299627700 1 --2- 172.21.15.40:0/682808349 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f027c0a1150 0x7f027c0a3600 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f0284005fd0 tx=0x7f0284005ee0 comp rx=0 tx=0).stop 2023-07-14T19:15:20.925 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:20.924+0000 7f0299627700 1 -- 172.21.15.40:0/682808349 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f02940ff580 msgr2=0x7f029419aa50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:15:20.925 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:20.924+0000 7f0299627700 1 --2- 172.21.15.40:0/682808349 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f02940ff580 0x7f029419aa50 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f0288009990 tx=0x7f0288009580 comp rx=0 tx=0).stop 2023-07-14T19:15:20.925 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:20.924+0000 7f0299627700 1 -- 172.21.15.40:0/682808349 shutdown_connections 2023-07-14T19:15:20.925 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:20.924+0000 7f0299627700 1 --2- 172.21.15.40:0/682808349 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f027c0a1150 0x7f027c0a3600 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:15:20.926 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:20.924+0000 7f0299627700 1 --2- 172.21.15.40:0/682808349 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0294103f50 0x7f0294194a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:15:20.926 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:20.924+0000 7f0299627700 1 --2- 172.21.15.40:0/682808349 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f02940ff580 0x7f029419aa50 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:15:20.926 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:20.924+0000 7f0299627700 1 --2- 172.21.15.40:0/682808349 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f02940fff20 0x7f029419af90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:15:20.926 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:20.924+0000 7f0299627700 1 -- 172.21.15.40:0/682808349 >> 172.21.15.40:0/682808349 conn(0x7f0294075280 msgr2=0x7f02940fbe00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:15:20.926 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:20.924+0000 7f0299627700 1 -- 172.21.15.40:0/682808349 shutdown_connections 2023-07-14T19:15:20.926 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:20.924+0000 7f0299627700 1 -- 172.21.15.40:0/682808349 wait complete. 2023-07-14T19:15:20.947 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:15:20 smithi040 ceph-mon[96518]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T19:15:20.972 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:15:20 smithi191 ceph-mon[98311]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T19:15:20.992 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:15:20 smithi130 ceph-mon[97832]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T19:15:21.358 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-07-14T19:12:03.271333+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-07-14T19:15:21.358 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell --fsid adc52f04-2279-11ee-9b33-001a4aab830c -- ceph osd pool get .mgr pg_num 2023-07-14T19:15:21.916 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/adc52f04-2279-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T19:15:21.971 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:15:21 smithi040 ceph-mon[96518]: from='client.? 172.21.15.40:0/682808349' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T19:15:21.972 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:15:21 smithi191 ceph-mon[98311]: from='client.? 172.21.15.40:0/682808349' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T19:15:21.992 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:15:21 smithi130 ceph-mon[97832]: from='client.? 172.21.15.40:0/682808349' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T19:15:22.958 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:15:22 smithi040 ceph-mon[96518]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T19:15:22.972 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:15:22 smithi191 ceph-mon[98311]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T19:15:22.992 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:15:22 smithi130 ceph-mon[97832]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T19:15:23.160 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:23.158+0000 7f606cc9a700 1 -- 172.21.15.40:0/2561036777 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f60680fef40 msgr2=0x7f60680ff310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:15:23.160 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:23.158+0000 7f606cc9a700 1 --2- 172.21.15.40:0/2561036777 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f60680fef40 0x7f60680ff310 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f605c0099c0 tx=0x7f605c009cd0 comp rx=0 tx=0).stop 2023-07-14T19:15:23.160 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:23.159+0000 7f606cc9a700 1 -- 172.21.15.40:0/2561036777 shutdown_connections 2023-07-14T19:15:23.160 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:23.159+0000 7f606cc9a700 1 --2- 172.21.15.40:0/2561036777 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6068101150 0x7f606810bb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:15:23.161 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:23.159+0000 7f606cc9a700 1 --2- 172.21.15.40:0/2561036777 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f60680fef40 0x7f60680ff310 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:15:23.161 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:23.159+0000 7f606cc9a700 1 --2- 172.21.15.40:0/2561036777 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f60681007a0 0x7f6068100c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:15:23.161 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:23.159+0000 7f606cc9a700 1 -- 172.21.15.40:0/2561036777 >> 172.21.15.40:0/2561036777 conn(0x7f6068075c90 msgr2=0x7f6068076090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:15:23.161 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:23.159+0000 7f606cc9a700 1 -- 172.21.15.40:0/2561036777 shutdown_connections 2023-07-14T19:15:23.161 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:23.159+0000 7f606cc9a700 1 -- 172.21.15.40:0/2561036777 wait complete. 2023-07-14T19:15:23.162 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:23.160+0000 7f606cc9a700 1 Processor -- start 2023-07-14T19:15:23.163 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:23.160+0000 7f606cc9a700 1 -- start start 2023-07-14T19:15:23.163 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:23.160+0000 7f606cc9a700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f60680fef40 0x7f6068196c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:15:23.163 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:23.160+0000 7f606cc9a700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f60681007a0 0x7f6068197140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:15:23.163 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:23.160+0000 7f606cc9a700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6068101150 0x7f60681a1c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:15:23.163 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:23.160+0000 7f606cc9a700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60681a21c0 con 0x7f60681007a0 2023-07-14T19:15:23.163 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:23.160+0000 7f606cc9a700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60681a2300 con 0x7f6068101150 2023-07-14T19:15:23.164 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:23.160+0000 7f606cc9a700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60681a2440 con 0x7f60680fef40 2023-07-14T19:15:23.164 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:23.160+0000 7f6065d9b700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f60681007a0 0x7f6068197140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:15:23.164 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:23.160+0000 7f6065d9b700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f60681007a0 0x7f6068197140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:60752/0 (socket says 172.21.15.40:60752) 2023-07-14T19:15:23.164 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:23.160+0000 7f6065d9b700 1 -- 172.21.15.40:0/1084012239 learned_addr learned my addr 172.21.15.40:0/1084012239 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-07-14T19:15:23.164 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:23.160+0000 7f606659c700 1 --2- 172.21.15.40:0/1084012239 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f60680fef40 0x7f6068196c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:15:23.164 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:23.160+0000 7f6066d9d700 1 --2- 172.21.15.40:0/1084012239 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6068101150 0x7f60681a1c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:15:23.164 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:23.161+0000 7f6065d9b700 1 -- 172.21.15.40:0/1084012239 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f60680fef40 msgr2=0x7f6068196c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:15:23.165 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:23.161+0000 7f6065d9b700 1 --2- 172.21.15.40:0/1084012239 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f60680fef40 0x7f6068196c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:15:23.165 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:23.161+0000 7f6065d9b700 1 -- 172.21.15.40:0/1084012239 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6068101150 msgr2=0x7f60681a1c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:15:23.165 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:23.161+0000 7f6065d9b700 1 --2- 172.21.15.40:0/1084012239 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6068101150 0x7f60681a1c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:15:23.165 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:23.161+0000 7f6065d9b700 1 -- 172.21.15.40:0/1084012239 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f605c0095f0 con 0x7f60681007a0 2023-07-14T19:15:23.165 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:23.161+0000 7f606659c700 1 --2- 172.21.15.40:0/1084012239 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f60680fef40 0x7f6068196c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T19:15:23.165 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:23.161+0000 7f6065d9b700 1 --2- 172.21.15.40:0/1084012239 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f60681007a0 0x7f6068197140 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f605800c9b0 tx=0x7f605800cd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:15:23.165 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:23.161+0000 7f60577fe700 1 -- 172.21.15.40:0/1084012239 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6058007b00 con 0x7f60681007a0 2023-07-14T19:15:23.165 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:23.161+0000 7f60577fe700 1 -- 172.21.15.40:0/1084012239 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f6058007c60 con 0x7f60681007a0 2023-07-14T19:15:23.166 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:23.161+0000 7f60577fe700 1 -- 172.21.15.40:0/1084012239 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6058018660 con 0x7f60681007a0 2023-07-14T19:15:23.166 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:23.161+0000 7f606cc9a700 1 -- 172.21.15.40:0/1084012239 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f60681a26a0 con 0x7f60681007a0 2023-07-14T19:15:23.166 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:23.161+0000 7f606cc9a700 1 -- 172.21.15.40:0/1084012239 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f60681a2b20 con 0x7f60681007a0 2023-07-14T19:15:23.166 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:23.162+0000 7f606cc9a700 1 -- 172.21.15.40:0/1084012239 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f606804e340 con 0x7f60681007a0 2023-07-14T19:15:23.167 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:23.163+0000 7f60577fe700 1 -- 172.21.15.40:0/1084012239 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f6058018880 con 0x7f60681007a0 2023-07-14T19:15:23.167 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:23.164+0000 7f60577fe700 1 --2- 172.21.15.40:0/1084012239 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f60500a1150 0x7f60500a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:15:23.167 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:23.164+0000 7f606659c700 1 --2- 172.21.15.40:0/1084012239 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f60500a1150 0x7f60500a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:15:23.167 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:23.164+0000 7f60577fe700 1 -- 172.21.15.40:0/1084012239 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f60580cc530 con 0x7f60681007a0 2023-07-14T19:15:23.167 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:23.164+0000 7f606659c700 1 --2- 172.21.15.40:0/1084012239 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f60500a1150 0x7f60500a3600 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f605c000c00 tx=0x7f605c005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:15:23.169 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:23.168+0000 7f60577fe700 1 -- 172.21.15.40:0/1084012239 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f6058096120 con 0x7f60681007a0 2023-07-14T19:15:23.362 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:23.361+0000 7f606cc9a700 1 -- 172.21.15.40:0/1084012239 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f60681a2da0 con 0x7f60681007a0 2023-07-14T19:15:23.362 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:23.361+0000 7f60577fe700 1 -- 172.21.15.40:0/1084012239 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v77) v1 ==== 93+0+10 (secure 0 0 0) 0x7f6058099740 con 0x7f60681007a0 2023-07-14T19:15:23.363 INFO:teuthology.orchestra.run.smithi040.stdout:pg_num: 1 2023-07-14T19:15:23.364 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:23.363+0000 7f606cc9a700 1 -- 172.21.15.40:0/1084012239 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f60500a1150 msgr2=0x7f60500a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:15:23.365 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:23.364+0000 7f606cc9a700 1 --2- 172.21.15.40:0/1084012239 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f60500a1150 0x7f60500a3600 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f605c000c00 tx=0x7f605c005fb0 comp rx=0 tx=0).stop 2023-07-14T19:15:23.365 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:23.364+0000 7f606cc9a700 1 -- 172.21.15.40:0/1084012239 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f60681007a0 msgr2=0x7f6068197140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:15:23.365 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:23.364+0000 7f606cc9a700 1 --2- 172.21.15.40:0/1084012239 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f60681007a0 0x7f6068197140 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f605800c9b0 tx=0x7f605800cd70 comp rx=0 tx=0).stop 2023-07-14T19:15:23.365 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:23.364+0000 7f606cc9a700 1 -- 172.21.15.40:0/1084012239 shutdown_connections 2023-07-14T19:15:23.365 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:23.364+0000 7f606cc9a700 1 --2- 172.21.15.40:0/1084012239 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f60500a1150 0x7f60500a3600 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:15:23.365 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:23.364+0000 7f606cc9a700 1 --2- 172.21.15.40:0/1084012239 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f60680fef40 0x7f6068196c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:15:23.365 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:23.364+0000 7f606cc9a700 1 --2- 172.21.15.40:0/1084012239 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6068101150 0x7f60681a1c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:15:23.366 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:23.364+0000 7f606cc9a700 1 --2- 172.21.15.40:0/1084012239 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f60681007a0 0x7f6068197140 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:15:23.366 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:23.364+0000 7f606cc9a700 1 -- 172.21.15.40:0/1084012239 >> 172.21.15.40:0/1084012239 conn(0x7f6068075c90 msgr2=0x7f60680fd080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:15:23.366 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:23.364+0000 7f606cc9a700 1 -- 172.21.15.40:0/1084012239 shutdown_connections 2023-07-14T19:15:23.366 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:23.364+0000 7f606cc9a700 1 -- 172.21.15.40:0/1084012239 wait complete. 2023-07-14T19:15:23.674 INFO:tasks.cephadm:Adding mds.a on smithi040 2023-07-14T19:15:23.675 INFO:tasks.cephadm:Adding mds.d on smithi040 2023-07-14T19:15:23.675 INFO:tasks.cephadm:Adding mds.g on smithi040 2023-07-14T19:15:23.675 INFO:tasks.cephadm:Adding mds.j on smithi040 2023-07-14T19:15:23.675 INFO:tasks.cephadm:Adding mds.b on smithi130 2023-07-14T19:15:23.676 INFO:tasks.cephadm:Adding mds.e on smithi130 2023-07-14T19:15:23.676 INFO:tasks.cephadm:Adding mds.h on smithi130 2023-07-14T19:15:23.676 INFO:tasks.cephadm:Adding mds.k on smithi130 2023-07-14T19:15:23.676 INFO:tasks.cephadm:Adding mds.c on smithi191 2023-07-14T19:15:23.676 INFO:tasks.cephadm:Adding mds.f on smithi191 2023-07-14T19:15:23.677 INFO:tasks.cephadm:Adding mds.i on smithi191 2023-07-14T19:15:23.677 INFO:tasks.cephadm:Adding mds.l on smithi191 2023-07-14T19:15:23.677 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid adc52f04-2279-11ee-9b33-001a4aab830c -- ceph orch apply mds all '12;smithi040=a;smithi040=d;smithi040=g;smithi040=j;smithi130=b;smithi130=e;smithi130=h;smithi130=k;smithi191=c;smithi191=f;smithi191=i;smithi191=l' 2023-07-14T19:15:23.971 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:15:23 smithi040 ceph-mon[96518]: from='client.? 172.21.15.40:0/1084012239' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-07-14T19:15:23.971 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:15:23 smithi191 ceph-mon[98311]: from='client.? 172.21.15.40:0/1084012239' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-07-14T19:15:23.992 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:15:23 smithi130 ceph-mon[97832]: from='client.? 172.21.15.40:0/1084012239' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-07-14T19:15:24.245 INFO:teuthology.orchestra.run.smithi191.stderr:Inferring config /var/lib/ceph/adc52f04-2279-11ee-9b33-001a4aab830c/mon.c/config 2023-07-14T19:15:24.722 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:15:24 smithi191 ceph-mon[98311]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T19:15:24.742 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:15:24 smithi130 ceph-mon[97832]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T19:15:24.971 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:15:24 smithi040 ceph-mon[96518]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T19:15:25.549 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:15:25.545+0000 7f7f52ec9700 1 -- 172.21.15.191:0/3159187725 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7f4c0fee30 msgr2=0x7f7f4c0ff2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:15:25.549 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:15:25.545+0000 7f7f52ec9700 1 --2- 172.21.15.191:0/3159187725 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7f4c0fee30 0x7f7f4c0ff2a0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f7f38009990 tx=0x7f7f38009ca0 comp rx=0 tx=0).stop 2023-07-14T19:15:25.549 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:15:25.545+0000 7f7f52ec9700 1 -- 172.21.15.191:0/3159187725 shutdown_connections 2023-07-14T19:15:25.550 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:15:25.545+0000 7f7f52ec9700 1 --2- 172.21.15.191:0/3159187725 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7f4c0ff7e0 0x7f7f4c107090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:15:25.550 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:15:25.545+0000 7f7f52ec9700 1 --2- 172.21.15.191:0/3159187725 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7f4c1001d0 0x7f7f4c1005e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:15:25.550 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:15:25.545+0000 7f7f52ec9700 1 --2- 172.21.15.191:0/3159187725 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7f4c0fee30 0x7f7f4c0ff2a0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:15:25.550 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:15:25.545+0000 7f7f52ec9700 1 -- 172.21.15.191:0/3159187725 >> 172.21.15.191:0/3159187725 conn(0x7f7f4c0f9990 msgr2=0x7f7f4c0fbdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:15:25.550 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:15:25.545+0000 7f7f52ec9700 1 -- 172.21.15.191:0/3159187725 shutdown_connections 2023-07-14T19:15:25.550 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:15:25.545+0000 7f7f52ec9700 1 -- 172.21.15.191:0/3159187725 wait complete. 2023-07-14T19:15:25.550 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:15:25.546+0000 7f7f52ec9700 1 Processor -- start 2023-07-14T19:15:25.551 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:15:25.546+0000 7f7f52ec9700 1 -- start start 2023-07-14T19:15:25.551 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:15:25.546+0000 7f7f52ec9700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7f4c0fee30 0x7f7f4c0fe880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:15:25.551 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:15:25.546+0000 7f7f52ec9700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7f4c0ff7e0 0x7f7f4c19cb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:15:25.551 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:15:25.546+0000 7f7f52ec9700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7f4c1001d0 0x7f7f4c19f070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:15:25.551 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:15:25.546+0000 7f7f52ec9700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f4c19f720 con 0x7f7f4c0ff7e0 2023-07-14T19:15:25.551 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:15:25.546+0000 7f7f52ec9700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f4c19f890 con 0x7f7f4c1001d0 2023-07-14T19:15:25.551 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:15:25.546+0000 7f7f52ec9700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f4c19fa50 con 0x7f7f4c0fee30 2023-07-14T19:15:25.551 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:15:25.546+0000 7f7f50c65700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7f4c0fee30 0x7f7f4c0fe880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:15:25.552 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:15:25.546+0000 7f7f50c65700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7f4c0fee30 0x7f7f4c0fe880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.191:59750/0 (socket says 172.21.15.191:59750) 2023-07-14T19:15:25.552 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:15:25.546+0000 7f7f50c65700 1 -- 172.21.15.191:0/3294200774 learned_addr learned my addr 172.21.15.191:0/3294200774 (peer_addr_for_me v2:172.21.15.191:0/0) 2023-07-14T19:15:25.552 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:15:25.546+0000 7f7f51466700 1 --2- 172.21.15.191:0/3294200774 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7f4c1001d0 0x7f7f4c19f070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:15:25.552 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:15:25.547+0000 7f7f4bfff700 1 --2- 172.21.15.191:0/3294200774 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7f4c0ff7e0 0x7f7f4c19cb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:15:25.552 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:15:25.547+0000 7f7f50c65700 1 -- 172.21.15.191:0/3294200774 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7f4c1001d0 msgr2=0x7f7f4c19f070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:15:25.552 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:15:25.547+0000 7f7f50c65700 1 --2- 172.21.15.191:0/3294200774 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7f4c1001d0 0x7f7f4c19f070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:15:25.552 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:15:25.547+0000 7f7f50c65700 1 -- 172.21.15.191:0/3294200774 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7f4c0ff7e0 msgr2=0x7f7f4c19cb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:15:25.553 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:15:25.547+0000 7f7f50c65700 1 --2- 172.21.15.191:0/3294200774 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7f4c0ff7e0 0x7f7f4c19cb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:15:25.553 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:15:25.547+0000 7f7f50c65700 1 -- 172.21.15.191:0/3294200774 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f380095f0 con 0x7f7f4c0fee30 2023-07-14T19:15:25.553 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:15:25.547+0000 7f7f51466700 1 --2- 172.21.15.191:0/3294200774 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7f4c1001d0 0x7f7f4c19f070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T19:15:25.553 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:15:25.547+0000 7f7f50c65700 1 --2- 172.21.15.191:0/3294200774 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7f4c0fee30 0x7f7f4c0fe880 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f7f4000e970 tx=0x7f7f4000ed30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:15:25.553 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:15:25.547+0000 7f7f49ffb700 1 -- 172.21.15.191:0/3294200774 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f40009790 con 0x7f7f4c0fee30 2023-07-14T19:15:25.553 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:15:25.547+0000 7f7f49ffb700 1 -- 172.21.15.191:0/3294200774 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f7f400044e0 con 0x7f7f4c0fee30 2023-07-14T19:15:25.553 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:15:25.547+0000 7f7f52ec9700 1 -- 172.21.15.191:0/3294200774 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7f4c19fd30 con 0x7f7f4c0fee30 2023-07-14T19:15:25.554 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:15:25.547+0000 7f7f49ffb700 1 -- 172.21.15.191:0/3294200774 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f40003e30 con 0x7f7f4c0fee30 2023-07-14T19:15:25.554 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:15:25.547+0000 7f7f52ec9700 1 -- 172.21.15.191:0/3294200774 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7f4c1a01d0 con 0x7f7f4c0fee30 2023-07-14T19:15:25.554 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:15:25.548+0000 7f7f52ec9700 1 -- 172.21.15.191:0/3294200774 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7f4c106200 con 0x7f7f4c0fee30 2023-07-14T19:15:25.555 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:15:25.549+0000 7f7f49ffb700 1 -- 172.21.15.191:0/3294200774 <== mon.1 v2:172.21.15.191:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f7f40003f90 con 0x7f7f4c0fee30 2023-07-14T19:15:25.555 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:15:25.550+0000 7f7f49ffb700 1 --2- 172.21.15.191:0/3294200774 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f7f3c0a1150 0x7f7f3c0a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:15:25.555 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:15:25.550+0000 7f7f49ffb700 1 -- 172.21.15.191:0/3294200774 <== mon.1 v2:172.21.15.191:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f7f40014070 con 0x7f7f4c0fee30 2023-07-14T19:15:25.555 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:15:25.550+0000 7f7f4bfff700 1 --2- 172.21.15.191:0/3294200774 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f7f3c0a1150 0x7f7f3c0a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:15:25.555 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:15:25.550+0000 7f7f4bfff700 1 --2- 172.21.15.191:0/3294200774 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f7f3c0a1150 0x7f7f3c0a3600 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f7f380098e0 tx=0x7f7f38000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:15:25.556 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:15:25.554+0000 7f7f49ffb700 1 -- 172.21.15.191:0/3294200774 <== mon.1 v2:172.21.15.191:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f7f40096390 con 0x7f7f4c0fee30 2023-07-14T19:15:25.751 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:15:25.749+0000 7f7f52ec9700 1 -- 172.21.15.191:0/3294200774 --> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] -- mgr_command(tid 0: {"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi040=a;smithi040=d;smithi040=g;smithi040=j;smithi130=b;smithi130=e;smithi130=h;smithi130=k;smithi191=c;smithi191=f;smithi191=i;smithi191=l", "target": ["mon-mgr", ""]}) v1 -- 0x7f7f4c060c80 con 0x7f7f3c0a1150 2023-07-14T19:15:25.760 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:15:25.758+0000 7f7f49ffb700 1 -- 172.21.15.191:0/3294200774 <== mgr.14152 v2:172.21.15.40:6800/1693027118 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+28 (secure 0 0 0) 0x7f7f38027030 con 0x7f7f3c0a1150 2023-07-14T19:15:25.760 INFO:teuthology.orchestra.run.smithi191.stdout:Scheduled mds.all update... 2023-07-14T19:15:25.762 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:15:25.760+0000 7f7f52ec9700 1 -- 172.21.15.191:0/3294200774 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f7f3c0a1150 msgr2=0x7f7f3c0a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:15:25.762 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:15:25.760+0000 7f7f52ec9700 1 --2- 172.21.15.191:0/3294200774 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f7f3c0a1150 0x7f7f3c0a3600 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f7f380098e0 tx=0x7f7f38000bc0 comp rx=0 tx=0).stop 2023-07-14T19:15:25.763 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:15:25.760+0000 7f7f52ec9700 1 -- 172.21.15.191:0/3294200774 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7f4c0fee30 msgr2=0x7f7f4c0fe880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:15:25.763 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:15:25.760+0000 7f7f52ec9700 1 --2- 172.21.15.191:0/3294200774 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7f4c0fee30 0x7f7f4c0fe880 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f7f4000e970 tx=0x7f7f4000ed30 comp rx=0 tx=0).stop 2023-07-14T19:15:25.763 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:15:25.760+0000 7f7f52ec9700 1 -- 172.21.15.191:0/3294200774 shutdown_connections 2023-07-14T19:15:25.763 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:15:25.760+0000 7f7f52ec9700 1 --2- 172.21.15.191:0/3294200774 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f7f3c0a1150 0x7f7f3c0a3600 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:15:25.763 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:15:25.760+0000 7f7f52ec9700 1 --2- 172.21.15.191:0/3294200774 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7f4c0fee30 0x7f7f4c0fe880 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:15:25.763 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:15:25.760+0000 7f7f52ec9700 1 --2- 172.21.15.191:0/3294200774 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7f4c1001d0 0x7f7f4c19f070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:15:25.763 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:15:25.760+0000 7f7f52ec9700 1 --2- 172.21.15.191:0/3294200774 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7f4c0ff7e0 0x7f7f4c19cb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:15:25.764 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:15:25.760+0000 7f7f52ec9700 1 -- 172.21.15.191:0/3294200774 >> 172.21.15.191:0/3294200774 conn(0x7f7f4c0f9990 msgr2=0x7f7f4c103340 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:15:25.764 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:15:25.760+0000 7f7f52ec9700 1 -- 172.21.15.191:0/3294200774 shutdown_connections 2023-07-14T19:15:25.764 INFO:teuthology.orchestra.run.smithi191.stderr:2023-07-14T19:15:25.760+0000 7f7f52ec9700 1 -- 172.21.15.191:0/3294200774 wait complete. 2023-07-14T19:15:26.190 DEBUG:teuthology.orchestra.run.smithi040:mds.a> sudo journalctl -f -n 0 -u ceph-adc52f04-2279-11ee-9b33-001a4aab830c@mds.a.service 2023-07-14T19:15:26.193 DEBUG:teuthology.orchestra.run.smithi040:mds.d> sudo journalctl -f -n 0 -u ceph-adc52f04-2279-11ee-9b33-001a4aab830c@mds.d.service 2023-07-14T19:15:26.231 INFO:journalctl@ceph.mds.a.smithi040.stdout:-- Logs begin at Fri 2023-07-14 18:50:26 UTC. -- 2023-07-14T19:15:26.232 DEBUG:teuthology.orchestra.run.smithi040:mds.g> sudo journalctl -f -n 0 -u ceph-adc52f04-2279-11ee-9b33-001a4aab830c@mds.g.service 2023-07-14T19:15:26.262 INFO:journalctl@ceph.mds.d.smithi040.stdout:-- Logs begin at Fri 2023-07-14 18:50:26 UTC. -- 2023-07-14T19:15:26.263 DEBUG:teuthology.orchestra.run.smithi040:mds.j> sudo journalctl -f -n 0 -u ceph-adc52f04-2279-11ee-9b33-001a4aab830c@mds.j.service 2023-07-14T19:15:26.292 INFO:journalctl@ceph.mds.g.smithi040.stdout:-- Logs begin at Fri 2023-07-14 18:50:26 UTC. -- 2023-07-14T19:15:26.293 DEBUG:teuthology.orchestra.run.smithi130:mds.b> sudo journalctl -f -n 0 -u ceph-adc52f04-2279-11ee-9b33-001a4aab830c@mds.b.service 2023-07-14T19:15:26.296 DEBUG:teuthology.orchestra.run.smithi130:mds.e> sudo journalctl -f -n 0 -u ceph-adc52f04-2279-11ee-9b33-001a4aab830c@mds.e.service 2023-07-14T19:15:26.320 INFO:journalctl@ceph.mds.j.smithi040.stdout:-- Logs begin at Fri 2023-07-14 18:50:26 UTC. -- 2023-07-14T19:15:26.332 INFO:journalctl@ceph.mds.b.smithi130.stdout:-- Logs begin at Fri 2023-07-14 18:50:32 UTC. -- 2023-07-14T19:15:26.334 DEBUG:teuthology.orchestra.run.smithi130:mds.h> sudo journalctl -f -n 0 -u ceph-adc52f04-2279-11ee-9b33-001a4aab830c@mds.h.service 2023-07-14T19:15:26.363 INFO:journalctl@ceph.mds.e.smithi130.stdout:-- Logs begin at Fri 2023-07-14 18:50:32 UTC. -- 2023-07-14T19:15:26.364 DEBUG:teuthology.orchestra.run.smithi130:mds.k> sudo journalctl -f -n 0 -u ceph-adc52f04-2279-11ee-9b33-001a4aab830c@mds.k.service 2023-07-14T19:15:26.392 INFO:journalctl@ceph.mds.h.smithi130.stdout:-- Logs begin at Fri 2023-07-14 18:50:32 UTC. -- 2023-07-14T19:15:26.393 DEBUG:teuthology.orchestra.run.smithi191:mds.c> sudo journalctl -f -n 0 -u ceph-adc52f04-2279-11ee-9b33-001a4aab830c@mds.c.service 2023-07-14T19:15:26.396 DEBUG:teuthology.orchestra.run.smithi191:mds.f> sudo journalctl -f -n 0 -u ceph-adc52f04-2279-11ee-9b33-001a4aab830c@mds.f.service 2023-07-14T19:15:26.419 INFO:journalctl@ceph.mds.k.smithi130.stdout:-- Logs begin at Fri 2023-07-14 18:50:32 UTC. -- 2023-07-14T19:15:26.424 INFO:journalctl@ceph.mds.c.smithi191.stdout:-- Logs begin at Fri 2023-07-14 18:50:54 UTC. -- 2023-07-14T19:15:26.425 DEBUG:teuthology.orchestra.run.smithi191:mds.i> sudo journalctl -f -n 0 -u ceph-adc52f04-2279-11ee-9b33-001a4aab830c@mds.i.service 2023-07-14T19:15:26.451 INFO:journalctl@ceph.mds.f.smithi191.stdout:-- Logs begin at Fri 2023-07-14 18:50:54 UTC. -- 2023-07-14T19:15:26.452 DEBUG:teuthology.orchestra.run.smithi191:mds.l> sudo journalctl -f -n 0 -u ceph-adc52f04-2279-11ee-9b33-001a4aab830c@mds.l.service 2023-07-14T19:15:26.477 INFO:journalctl@ceph.mds.i.smithi191.stdout:-- Logs begin at Fri 2023-07-14 18:50:54 UTC. -- 2023-07-14T19:15:26.478 INFO:tasks.cephadm:Setting up CephFS filesystem(s)... 2023-07-14T19:15:26.479 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-07-14T19:15:26.502 INFO:journalctl@ceph.mds.l.smithi191.stdout:-- Logs begin at Fri 2023-07-14 18:50:54 UTC. -- 2023-07-14T19:15:26.650 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:26.649+0000 7f130d684700 1 -- 172.21.15.40:0/3845593618 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f13080ff750 msgr2=0x7f1308107020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:15:26.651 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:26.649+0000 7f130d684700 1 --2- 172.21.15.40:0/3845593618 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f13080ff750 0x7f1308107020 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f130000b0a0 tx=0x7f130000b3b0 comp rx=0 tx=0).stop 2023-07-14T19:15:26.651 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:26.649+0000 7f130d684700 1 -- 172.21.15.40:0/3845593618 shutdown_connections 2023-07-14T19:15:26.651 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:26.649+0000 7f130d684700 1 --2- 172.21.15.40:0/3845593618 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f13080ff750 0x7f1308107020 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:15:26.651 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:26.649+0000 7f130d684700 1 --2- 172.21.15.40:0/3845593618 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1308100090 0x7f13081004a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:15:26.651 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:26.649+0000 7f130d684700 1 --2- 172.21.15.40:0/3845593618 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f13080fedc0 0x7f13080ff210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:15:26.652 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:26.649+0000 7f130d684700 1 -- 172.21.15.40:0/3845593618 >> 172.21.15.40:0/3845593618 conn(0x7f13080f98a0 msgr2=0x7f13080fbcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:15:26.652 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:26.649+0000 7f130d684700 1 -- 172.21.15.40:0/3845593618 shutdown_connections 2023-07-14T19:15:26.652 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:26.649+0000 7f130d684700 1 -- 172.21.15.40:0/3845593618 wait complete. 2023-07-14T19:15:26.652 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:26.650+0000 7f130d684700 1 Processor -- start 2023-07-14T19:15:26.654 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:26.650+0000 7f130d684700 1 -- start start 2023-07-14T19:15:26.654 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:26.650+0000 7f130d684700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f13080fedc0 0x7f13080fe840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:15:26.654 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:26.650+0000 7f130d684700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f13080ff750 0x7f130819a770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:15:26.654 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:26.650+0000 7f130d684700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1308100090 0x7f130819cdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:15:26.654 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:26.650+0000 7f130d684700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f130819d590 con 0x7f13080ff750 2023-07-14T19:15:26.654 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:26.650+0000 7f130d684700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f130819d700 con 0x7f1308100090 2023-07-14T19:15:26.655 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:26.650+0000 7f130d684700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f130819d8c0 con 0x7f13080fedc0 2023-07-14T19:15:26.655 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:26.650+0000 7f13067fc700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f13080ff750 0x7f130819a770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:15:26.655 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:26.650+0000 7f13067fc700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f13080ff750 0x7f130819a770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:60764/0 (socket says 172.21.15.40:60764) 2023-07-14T19:15:26.655 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:26.650+0000 7f13067fc700 1 -- 172.21.15.40:0/2319755287 learned_addr learned my addr 172.21.15.40:0/2319755287 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-07-14T19:15:26.655 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:26.650+0000 7f1306ffd700 1 --2- 172.21.15.40:0/2319755287 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f13080fedc0 0x7f13080fe840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:15:26.655 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:26.650+0000 7f13077fe700 1 --2- 172.21.15.40:0/2319755287 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1308100090 0x7f130819cdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:15:26.655 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:26.651+0000 7f13067fc700 1 -- 172.21.15.40:0/2319755287 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f13080fedc0 msgr2=0x7f13080fe840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:15:26.656 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:26.651+0000 7f13067fc700 1 --2- 172.21.15.40:0/2319755287 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f13080fedc0 0x7f13080fe840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:15:26.656 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:26.651+0000 7f13067fc700 1 -- 172.21.15.40:0/2319755287 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1308100090 msgr2=0x7f130819cdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:15:26.656 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:26.651+0000 7f13067fc700 1 --2- 172.21.15.40:0/2319755287 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1308100090 0x7f130819cdc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:15:26.656 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:26.651+0000 7f13067fc700 1 -- 172.21.15.40:0/2319755287 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1300009d00 con 0x7f13080ff750 2023-07-14T19:15:26.656 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:26.651+0000 7f13067fc700 1 --2- 172.21.15.40:0/2319755287 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f13080ff750 0x7f130819a770 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f12f800e9a0 tx=0x7f12f800ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:15:26.656 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:26.651+0000 7f12fffff700 1 -- 172.21.15.40:0/2319755287 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12f8009790 con 0x7f13080ff750 2023-07-14T19:15:26.657 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:26.651+0000 7f12fffff700 1 -- 172.21.15.40:0/2319755287 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f12f8010430 con 0x7f13080ff750 2023-07-14T19:15:26.657 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:26.651+0000 7f12fffff700 1 -- 172.21.15.40:0/2319755287 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12f8018680 con 0x7f13080ff750 2023-07-14T19:15:26.657 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:26.651+0000 7f130d684700 1 -- 172.21.15.40:0/2319755287 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f130819dba0 con 0x7f13080ff750 2023-07-14T19:15:26.657 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:26.651+0000 7f130d684700 1 -- 172.21.15.40:0/2319755287 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f130819e0b0 con 0x7f13080ff750 2023-07-14T19:15:26.657 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:26.652+0000 7f130d684700 1 -- 172.21.15.40:0/2319755287 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f13081060d0 con 0x7f13080ff750 2023-07-14T19:15:26.657 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:26.653+0000 7f12fffff700 1 -- 172.21.15.40:0/2319755287 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f12f80105a0 con 0x7f13080ff750 2023-07-14T19:15:26.658 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:26.654+0000 7f12fffff700 1 --2- 172.21.15.40:0/2319755287 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f12f40a1220 0x7f12f40a36d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:15:26.658 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:26.654+0000 7f1306ffd700 1 --2- 172.21.15.40:0/2319755287 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f12f40a1220 0x7f12f40a36d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:15:26.658 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:26.654+0000 7f12fffff700 1 -- 172.21.15.40:0/2319755287 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f12f8014070 con 0x7f13080ff750 2023-07-14T19:15:26.658 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:26.654+0000 7f1306ffd700 1 --2- 172.21.15.40:0/2319755287 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f12f40a1220 0x7f12f40a36d0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f1308067520 tx=0x7f12f0009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:15:26.659 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:26.658+0000 7f12fffff700 1 -- 172.21.15.40:0/2319755287 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f12f8051780 con 0x7f13080ff750 2023-07-14T19:15:26.866 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:26.866+0000 7f130d684700 1 -- 172.21.15.40:0/2319755287 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f130802c8e0 con 0x7f13080ff750 2023-07-14T19:15:26.868 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:26.867+0000 7f12fffff700 1 -- 172.21.15.40:0/2319755287 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v77) v1 ==== 74+0+18424 (secure 0 0 0) 0x7f12f80961b0 con 0x7f13080ff750 2023-07-14T19:15:26.868 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-07-14T19:15:26.869 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":77,"fsid":"adc52f04-2279-11ee-9b33-001a4aab830c","created":"2023-07-14T19:08:56.893730+0000","modified":"2023-07-14T19:15:17.644846+0000","last_up_change":"2023-07-14T19:15:16.632979+0000","last_in_change":"2023-07-14T19:14:58.961702+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-07-14T19:12:03.271333+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"0d79356a-95d2-49ad-857b-2092ae9f7e13","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":3599703125},{"type":"v1","addr":"172.21.15.40:6803","nonce":3599703125}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":3599703125},{"type":"v1","addr":"172.21.15.40:6805","nonce":3599703125}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":3599703125},{"type":"v1","addr":"172.21.15.40:6809","nonce":3599703125}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":3599703125},{"type":"v1","addr":"172.21.15.40:6807","nonce":3599703125}]},"public_addr":"172.21.15.40:6803/3599703125","cluster_addr":"172.21.15.40:6805/3599703125","heartbeat_back_addr":"172.21.15.40:6809/3599703125","heartbeat_front_addr":"172.21.15.40:6807/3599703125","state":["exists","up"]},{"osd":1,"uuid":"40f8a3d1-a5e4-473c-b9f2-7600ec22be0d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6800","nonce":2512760702},{"type":"v1","addr":"172.21.15.130:6801","nonce":2512760702}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6802","nonce":2512760702},{"type":"v1","addr":"172.21.15.130:6803","nonce":2512760702}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6806","nonce":2512760702},{"type":"v1","addr":"172.21.15.130:6807","nonce":2512760702}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6804","nonce":2512760702},{"type":"v1","addr":"172.21.15.130:6805","nonce":2512760702}]},"public_addr":"172.21.15.130:6801/2512760702","cluster_addr":"172.21.15.130:6803/2512760702","heartbeat_back_addr":"172.21.15.130:6807/2512760702","heartbeat_front_addr":"172.21.15.130:6805/2512760702","state":["exists","up"]},{"osd":2,"uuid":"5a157891-cb7a-4ea6-9239-f48a7c67c003","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6800","nonce":589569135},{"type":"v1","addr":"172.21.15.191:6801","nonce":589569135}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6802","nonce":589569135},{"type":"v1","addr":"172.21.15.191:6803","nonce":589569135}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6806","nonce":589569135},{"type":"v1","addr":"172.21.15.191:6807","nonce":589569135}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6804","nonce":589569135},{"type":"v1","addr":"172.21.15.191:6805","nonce":589569135}]},"public_addr":"172.21.15.191:6801/589569135","cluster_addr":"172.21.15.191:6803/589569135","heartbeat_back_addr":"172.21.15.191:6807/589569135","heartbeat_front_addr":"172.21.15.191:6805/589569135","state":["exists","up"]},{"osd":3,"uuid":"8020e7dc-ac40-487f-857a-8da40bccb459","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":996287860},{"type":"v1","addr":"172.21.15.40:6811","nonce":996287860}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6812","nonce":996287860},{"type":"v1","addr":"172.21.15.40:6813","nonce":996287860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6816","nonce":996287860},{"type":"v1","addr":"172.21.15.40:6817","nonce":996287860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":996287860},{"type":"v1","addr":"172.21.15.40:6815","nonce":996287860}]},"public_addr":"172.21.15.40:6811/996287860","cluster_addr":"172.21.15.40:6813/996287860","heartbeat_back_addr":"172.21.15.40:6817/996287860","heartbeat_front_addr":"172.21.15.40:6815/996287860","state":["exists","up"]},{"osd":4,"uuid":"c5501b35-f62d-482d-8d22-115c88f0298e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6808","nonce":3507448720},{"type":"v1","addr":"172.21.15.130:6809","nonce":3507448720}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6810","nonce":3507448720},{"type":"v1","addr":"172.21.15.130:6811","nonce":3507448720}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6814","nonce":3507448720},{"type":"v1","addr":"172.21.15.130:6815","nonce":3507448720}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6812","nonce":3507448720},{"type":"v1","addr":"172.21.15.130:6813","nonce":3507448720}]},"public_addr":"172.21.15.130:6809/3507448720","cluster_addr":"172.21.15.130:6811/3507448720","heartbeat_back_addr":"172.21.15.130:6815/3507448720","heartbeat_front_addr":"172.21.15.130:6813/3507448720","state":["exists","up"]},{"osd":5,"uuid":"7770b6d6-5874-493a-986a-84db482cee42","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6808","nonce":1781944663},{"type":"v1","addr":"172.21.15.191:6809","nonce":1781944663}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6810","nonce":1781944663},{"type":"v1","addr":"172.21.15.191:6811","nonce":1781944663}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6814","nonce":1781944663},{"type":"v1","addr":"172.21.15.191:6815","nonce":1781944663}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6812","nonce":1781944663},{"type":"v1","addr":"172.21.15.191:6813","nonce":1781944663}]},"public_addr":"172.21.15.191:6809/1781944663","cluster_addr":"172.21.15.191:6811/1781944663","heartbeat_back_addr":"172.21.15.191:6815/1781944663","heartbeat_front_addr":"172.21.15.191:6813/1781944663","state":["exists","up"]},{"osd":6,"uuid":"ef65ad00-dfee-4300-9961-dfc8573aab06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6818","nonce":693186773},{"type":"v1","addr":"172.21.15.40:6819","nonce":693186773}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6820","nonce":693186773},{"type":"v1","addr":"172.21.15.40:6821","nonce":693186773}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6824","nonce":693186773},{"type":"v1","addr":"172.21.15.40:6825","nonce":693186773}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6822","nonce":693186773},{"type":"v1","addr":"172.21.15.40:6823","nonce":693186773}]},"public_addr":"172.21.15.40:6819/693186773","cluster_addr":"172.21.15.40:6821/693186773","heartbeat_back_addr":"172.21.15.40:6825/693186773","heartbeat_front_addr":"172.21.15.40:6823/693186773","state":["exists","up"]},{"osd":7,"uuid":"1700fbfa-cbfc-4f76-97be-727a3b100740","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":50,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6816","nonce":2384019528},{"type":"v1","addr":"172.21.15.130:6817","nonce":2384019528}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6818","nonce":2384019528},{"type":"v1","addr":"172.21.15.130:6819","nonce":2384019528}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6822","nonce":2384019528},{"type":"v1","addr":"172.21.15.130:6823","nonce":2384019528}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6820","nonce":2384019528},{"type":"v1","addr":"172.21.15.130:6821","nonce":2384019528}]},"public_addr":"172.21.15.130:6817/2384019528","cluster_addr":"172.21.15.130:6819/2384019528","heartbeat_back_addr":"172.21.15.130:6823/2384019528","heartbeat_front_addr":"172.21.15.130:6821/2384019528","state":["exists","up"]},{"osd":8,"uuid":"b76d4033-c1e2-457b-8571-c8a49fe443f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":57,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6816","nonce":2386818417},{"type":"v1","addr":"172.21.15.191:6817","nonce":2386818417}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6818","nonce":2386818417},{"type":"v1","addr":"172.21.15.191:6819","nonce":2386818417}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6822","nonce":2386818417},{"type":"v1","addr":"172.21.15.191:6823","nonce":2386818417}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6820","nonce":2386818417},{"type":"v1","addr":"172.21.15.191:6821","nonce":2386818417}]},"public_addr":"172.21.15.191:6817/2386818417","cluster_addr":"172.21.15.191:6819/2386818417","heartbeat_back_addr":"172.21.15.191:6823/2386818417","heartbeat_front_addr":"172.21.15.191:6821/2386818417","state":["exists","up"]},{"osd":9,"uuid":"df31bf2f-e30e-41b6-b2b9-0e26a07a88bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":64,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6826","nonce":2228380263},{"type":"v1","addr":"172.21.15.40:6827","nonce":2228380263}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6828","nonce":2228380263},{"type":"v1","addr":"172.21.15.40:6829","nonce":2228380263}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":2228380263},{"type":"v1","addr":"172.21.15.40:6833","nonce":2228380263}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6830","nonce":2228380263},{"type":"v1","addr":"172.21.15.40:6831","nonce":2228380263}]},"public_addr":"172.21.15.40:6827/2228380263","cluster_addr":"172.21.15.40:6829/2228380263","heartbeat_back_addr":"172.21.15.40:6833/2228380263","heartbeat_front_addr":"172.21.15.40:6831/2228380263","state":["exists","up"]},{"osd":10,"uuid":"4a93d621-49ee-4ad7-9e49-750614faf912","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6824","nonce":2417067798},{"type":"v1","addr":"172.21.15.130:6825","nonce":2417067798}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6826","nonce":2417067798},{"type":"v1","addr":"172.21.15.130:6827","nonce":2417067798}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6830","nonce":2417067798},{"type":"v1","addr":"172.21.15.130:6831","nonce":2417067798}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6828","nonce":2417067798},{"type":"v1","addr":"172.21.15.130:6829","nonce":2417067798}]},"public_addr":"172.21.15.130:6825/2417067798","cluster_addr":"172.21.15.130:6827/2417067798","heartbeat_back_addr":"172.21.15.130:6831/2417067798","heartbeat_front_addr":"172.21.15.130:6829/2417067798","state":["exists","up"]},{"osd":11,"uuid":"f43a2a6f-82f3-42ac-b77c-4f1b34da6918","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":76,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6824","nonce":2886059446},{"type":"v1","addr":"172.21.15.191:6825","nonce":2886059446}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6826","nonce":2886059446},{"type":"v1","addr":"172.21.15.191:6827","nonce":2886059446}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6830","nonce":2886059446},{"type":"v1","addr":"172.21.15.191:6831","nonce":2886059446}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6828","nonce":2886059446},{"type":"v1","addr":"172.21.15.191:6829","nonce":2886059446}]},"public_addr":"172.21.15.191:6825/2886059446","cluster_addr":"172.21.15.191:6827/2886059446","heartbeat_back_addr":"172.21.15.191:6831/2886059446","heartbeat_front_addr":"172.21.15.191:6829/2886059446","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:11:18.320495+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:11:39.901199+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:12:00.869042+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:12:23.292742+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:12:44.587019+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:13:06.494893+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:13:28.378374+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:13:50.033491+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:14:10.937062+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:14:32.358498+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:14:53.666584+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:15:14.837566+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.40:6801/3698757264":"2023-07-15T19:09:50.732458+0000","172.21.15.40:6800/3698757264":"2023-07-15T19:09:50.732458+0000","172.21.15.40:6800/2569015667":"2023-07-15T19:09:24.441543+0000","172.21.15.40:0/3653961003":"2023-07-15T19:09:24.441543+0000","172.21.15.40:0/3568861680":"2023-07-15T19:09:24.441543+0000","172.21.15.40:0/3249773172":"2023-07-15T19:09:50.732458+0000","172.21.15.40:0/4084215480":"2023-07-15T19:09:24.441543+0000","172.21.15.40:0/1163941760":"2023-07-15T19:09:50.732458+0000","172.21.15.40:6801/2569015667":"2023-07-15T19:09:24.441543+0000","172.21.15.40:0/2326639960":"2023-07-15T19:09:50.732458+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-07-14T19:15:26.871 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:26.869+0000 7f130d684700 1 -- 172.21.15.40:0/2319755287 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f12f40a1220 msgr2=0x7f12f40a36d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:15:26.871 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:26.869+0000 7f130d684700 1 --2- 172.21.15.40:0/2319755287 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f12f40a1220 0x7f12f40a36d0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f1308067520 tx=0x7f12f0009260 comp rx=0 tx=0).stop 2023-07-14T19:15:26.871 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:26.869+0000 7f130d684700 1 -- 172.21.15.40:0/2319755287 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f13080ff750 msgr2=0x7f130819a770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:15:26.872 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:26.869+0000 7f130d684700 1 --2- 172.21.15.40:0/2319755287 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f13080ff750 0x7f130819a770 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f12f800e9a0 tx=0x7f12f800ecb0 comp rx=0 tx=0).stop 2023-07-14T19:15:26.872 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:26.869+0000 7f130d684700 1 -- 172.21.15.40:0/2319755287 shutdown_connections 2023-07-14T19:15:26.872 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:26.869+0000 7f130d684700 1 --2- 172.21.15.40:0/2319755287 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f12f40a1220 0x7f12f40a36d0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:15:26.872 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:26.869+0000 7f130d684700 1 --2- 172.21.15.40:0/2319755287 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f13080fedc0 0x7f13080fe840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:15:26.873 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:26.869+0000 7f130d684700 1 --2- 172.21.15.40:0/2319755287 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1308100090 0x7f130819cdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:15:26.873 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:26.869+0000 7f130d684700 1 --2- 172.21.15.40:0/2319755287 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f13080ff750 0x7f130819a770 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:15:26.873 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:26.869+0000 7f130d684700 1 -- 172.21.15.40:0/2319755287 >> 172.21.15.40:0/2319755287 conn(0x7f13080f98a0 msgr2=0x7f1308103210 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:15:26.874 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:26.869+0000 7f130d684700 1 -- 172.21.15.40:0/2319755287 shutdown_connections 2023-07-14T19:15:26.874 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:26.869+0000 7f130d684700 1 -- 172.21.15.40:0/2319755287 wait complete. 2023-07-14T19:15:26.883 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-07-14T19:12:03.271333+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-07-14T19:15:26.883 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-07-14T19:15:27.011 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:27.010+0000 7f9edc924700 1 -- 172.21.15.40:0/3737309981 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9ed80ffc10 msgr2=0x7f9ed8100020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:15:27.012 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:27.010+0000 7f9edc924700 1 --2- 172.21.15.40:0/3737309981 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9ed80ffc10 0x7f9ed8100020 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f9ec8009990 tx=0x7f9ec8009ca0 comp rx=0 tx=0).stop 2023-07-14T19:15:27.012 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:27.010+0000 7f9edc924700 1 -- 172.21.15.40:0/3737309981 shutdown_connections 2023-07-14T19:15:27.012 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:27.010+0000 7f9edc924700 1 --2- 172.21.15.40:0/3737309981 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9ed8101790 0x7f9ed8108050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:15:27.012 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:27.010+0000 7f9edc924700 1 --2- 172.21.15.40:0/3737309981 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9ed80ffc10 0x7f9ed8100020 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:15:27.012 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:27.010+0000 7f9edc924700 1 --2- 172.21.15.40:0/3737309981 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9ed8100e00 0x7f9ed8101250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:15:27.012 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:27.010+0000 7f9edc924700 1 -- 172.21.15.40:0/3737309981 >> 172.21.15.40:0/3737309981 conn(0x7f9ed80fb250 msgr2=0x7f9ed80fd660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:15:27.012 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:27.010+0000 7f9edc924700 1 -- 172.21.15.40:0/3737309981 shutdown_connections 2023-07-14T19:15:27.013 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:27.010+0000 7f9edc924700 1 -- 172.21.15.40:0/3737309981 wait complete. 2023-07-14T19:15:27.014 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:27.011+0000 7f9edc924700 1 Processor -- start 2023-07-14T19:15:27.014 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:27.011+0000 7f9edc924700 1 -- start start 2023-07-14T19:15:27.014 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:27.011+0000 7f9edc924700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9ed80ffc10 0x7f9ed81959b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:15:27.015 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:27.011+0000 7f9edc924700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9ed8100e00 0x7f9ed8195ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:15:27.015 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:27.011+0000 7f9edc924700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9ed8101790 0x7f9ed819cf10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:15:27.015 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:27.011+0000 7f9edc924700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ed819d4e0 con 0x7f9ed80ffc10 2023-07-14T19:15:27.015 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:27.011+0000 7f9edc924700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ed819d620 con 0x7f9ed8101790 2023-07-14T19:15:27.015 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:27.011+0000 7f9edc924700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ed819d7b0 con 0x7f9ed8100e00 2023-07-14T19:15:27.015 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:27.011+0000 7f9ed659c700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9ed80ffc10 0x7f9ed81959b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:15:27.015 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:27.011+0000 7f9ed659c700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9ed80ffc10 0x7f9ed81959b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:60776/0 (socket says 172.21.15.40:60776) 2023-07-14T19:15:27.016 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:27.011+0000 7f9ed659c700 1 -- 172.21.15.40:0/2676909995 learned_addr learned my addr 172.21.15.40:0/2676909995 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-07-14T19:15:27.016 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:27.011+0000 7f9ed5d9b700 1 --2- 172.21.15.40:0/2676909995 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9ed8100e00 0x7f9ed8195ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:15:27.016 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:27.012+0000 7f9ed6d9d700 1 --2- 172.21.15.40:0/2676909995 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9ed8101790 0x7f9ed819cf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:15:27.016 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:27.012+0000 7f9ed659c700 1 -- 172.21.15.40:0/2676909995 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9ed8100e00 msgr2=0x7f9ed8195ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:15:27.016 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:27.012+0000 7f9ed659c700 1 --2- 172.21.15.40:0/2676909995 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9ed8100e00 0x7f9ed8195ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:15:27.016 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:27.012+0000 7f9ed659c700 1 -- 172.21.15.40:0/2676909995 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9ed8101790 msgr2=0x7f9ed819cf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:15:27.016 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:27.012+0000 7f9ed659c700 1 --2- 172.21.15.40:0/2676909995 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9ed8101790 0x7f9ed819cf10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:15:27.017 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:27.012+0000 7f9ed659c700 1 -- 172.21.15.40:0/2676909995 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ec80095f0 con 0x7f9ed80ffc10 2023-07-14T19:15:27.017 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:27.012+0000 7f9ed659c700 1 --2- 172.21.15.40:0/2676909995 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9ed80ffc10 0x7f9ed81959b0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f9ec80059d0 tx=0x7f9ec8005740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:15:27.017 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:27.012+0000 7f9ecf7fe700 1 -- 172.21.15.40:0/2676909995 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ec801d070 con 0x7f9ed80ffc10 2023-07-14T19:15:27.017 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:27.012+0000 7f9ecf7fe700 1 -- 172.21.15.40:0/2676909995 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f9ec8004020 con 0x7f9ed80ffc10 2023-07-14T19:15:27.017 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:27.012+0000 7f9ecf7fe700 1 -- 172.21.15.40:0/2676909995 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ec8022d00 con 0x7f9ed80ffc10 2023-07-14T19:15:27.017 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:27.012+0000 7f9edc924700 1 -- 172.21.15.40:0/2676909995 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9ed819da30 con 0x7f9ed80ffc10 2023-07-14T19:15:27.017 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:27.012+0000 7f9edc924700 1 -- 172.21.15.40:0/2676909995 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ed819de50 con 0x7f9ed80ffc10 2023-07-14T19:15:27.017 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:27.013+0000 7f9edc924700 1 -- 172.21.15.40:0/2676909995 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9ed8107250 con 0x7f9ed80ffc10 2023-07-14T19:15:27.018 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:27.014+0000 7f9ecf7fe700 1 -- 172.21.15.40:0/2676909995 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f9ec8022470 con 0x7f9ed80ffc10 2023-07-14T19:15:27.018 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:27.015+0000 7f9ecf7fe700 1 --2- 172.21.15.40:0/2676909995 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f9ec40a11d0 0x7f9ec40a3680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:15:27.018 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:27.015+0000 7f9ed5d9b700 1 --2- 172.21.15.40:0/2676909995 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f9ec40a11d0 0x7f9ec40a3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:15:27.019 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:27.015+0000 7f9ecf7fe700 1 -- 172.21.15.40:0/2676909995 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f9ec80cdf80 con 0x7f9ed80ffc10 2023-07-14T19:15:27.019 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:27.015+0000 7f9ed5d9b700 1 --2- 172.21.15.40:0/2676909995 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f9ec40a11d0 0x7f9ec40a3680 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f9ec0007c60 tx=0x7f9ec00073d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:15:27.019 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:27.018+0000 7f9ecf7fe700 1 -- 172.21.15.40:0/2676909995 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f9ec8097b60 con 0x7f9ed80ffc10 2023-07-14T19:15:27.160 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:15:26 smithi191 ceph-mon[98311]: pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T19:15:27.160 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:15:26 smithi191 ceph-mon[98311]: from='client.24412 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi040=a;smithi040=d;smithi040=g;smithi040=j;smithi130=b;smithi130=e;smithi130=h;smithi130=k;smithi191=c;smithi191=f;smithi191=i;smithi191=l", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T19:15:27.160 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:15:26 smithi191 ceph-mon[98311]: Saving service mds.all spec with placement smithi040=a;smithi040=d;smithi040=g;smithi040=j;smithi130=b;smithi130=e;smithi130=h;smithi130=k;smithi191=c;smithi191=f;smithi191=i;smithi191=l;count:12 2023-07-14T19:15:27.160 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:15:26 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:15:27.161 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:15:26 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T19:15:27.161 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:15:26 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:15:27.161 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:15:26 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T19:15:27.161 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:15:26 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:15:27.161 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:15:26 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-14T19:15:27.161 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:15:26 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-14T19:15:27.161 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:15:26 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:15:27.217 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:27.216+0000 7f9edc924700 1 -- 172.21.15.40:0/2676909995 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f9ed802ca70 con 0x7f9ed80ffc10 2023-07-14T19:15:27.217 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:27.216+0000 7f9ecf7fe700 1 -- 172.21.15.40:0/2676909995 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v77) v1 ==== 93+0+10 (secure 0 0 0) 0x7f9ec8022e60 con 0x7f9ed80ffc10 2023-07-14T19:15:27.217 INFO:teuthology.orchestra.run.smithi040.stdout:pg_num: 1 2023-07-14T19:15:27.219 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:27.218+0000 7f9edc924700 1 -- 172.21.15.40:0/2676909995 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f9ec40a11d0 msgr2=0x7f9ec40a3680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:15:27.220 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:27.218+0000 7f9edc924700 1 --2- 172.21.15.40:0/2676909995 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f9ec40a11d0 0x7f9ec40a3680 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f9ec0007c60 tx=0x7f9ec00073d0 comp rx=0 tx=0).stop 2023-07-14T19:15:27.220 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:27.218+0000 7f9edc924700 1 -- 172.21.15.40:0/2676909995 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9ed80ffc10 msgr2=0x7f9ed81959b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:15:27.220 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:27.218+0000 7f9edc924700 1 --2- 172.21.15.40:0/2676909995 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9ed80ffc10 0x7f9ed81959b0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f9ec80059d0 tx=0x7f9ec8005740 comp rx=0 tx=0).stop 2023-07-14T19:15:27.220 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:27.218+0000 7f9edc924700 1 -- 172.21.15.40:0/2676909995 shutdown_connections 2023-07-14T19:15:27.220 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:27.218+0000 7f9edc924700 1 --2- 172.21.15.40:0/2676909995 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f9ec40a11d0 0x7f9ec40a3680 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:15:27.220 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:27.218+0000 7f9edc924700 1 --2- 172.21.15.40:0/2676909995 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9ed8101790 0x7f9ed819cf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:15:27.221 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:27.218+0000 7f9edc924700 1 --2- 172.21.15.40:0/2676909995 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9ed80ffc10 0x7f9ed81959b0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:15:27.221 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:27.218+0000 7f9edc924700 1 --2- 172.21.15.40:0/2676909995 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9ed8100e00 0x7f9ed8195ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:15:27.221 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:27.218+0000 7f9edc924700 1 -- 172.21.15.40:0/2676909995 >> 172.21.15.40:0/2676909995 conn(0x7f9ed80fb250 msgr2=0x7f9ed8104390 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:15:27.221 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:27.218+0000 7f9edc924700 1 -- 172.21.15.40:0/2676909995 shutdown_connections 2023-07-14T19:15:27.221 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:27.218+0000 7f9edc924700 1 -- 172.21.15.40:0/2676909995 wait complete. 2023-07-14T19:15:27.222 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:15:26 smithi040 ceph-mon[96518]: pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T19:15:27.222 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:15:26 smithi040 ceph-mon[96518]: from='client.24412 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi040=a;smithi040=d;smithi040=g;smithi040=j;smithi130=b;smithi130=e;smithi130=h;smithi130=k;smithi191=c;smithi191=f;smithi191=i;smithi191=l", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T19:15:27.222 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:15:26 smithi040 ceph-mon[96518]: Saving service mds.all spec with placement smithi040=a;smithi040=d;smithi040=g;smithi040=j;smithi130=b;smithi130=e;smithi130=h;smithi130=k;smithi191=c;smithi191=f;smithi191=i;smithi191=l;count:12 2023-07-14T19:15:27.222 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:15:26 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:15:27.222 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:15:26 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T19:15:27.222 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:15:26 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:15:27.222 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:15:26 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T19:15:27.223 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:15:26 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:15:27.223 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:15:26 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-14T19:15:27.223 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:15:26 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-14T19:15:27.223 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:15:26 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:15:27.232 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-07-14T19:15:27.242 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:15:26 smithi130 ceph-mon[97832]: pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T19:15:27.243 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:15:26 smithi130 ceph-mon[97832]: from='client.24412 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi040=a;smithi040=d;smithi040=g;smithi040=j;smithi130=b;smithi130=e;smithi130=h;smithi130=k;smithi191=c;smithi191=f;smithi191=i;smithi191=l", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T19:15:27.243 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:15:26 smithi130 ceph-mon[97832]: Saving service mds.all spec with placement smithi040=a;smithi040=d;smithi040=g;smithi040=j;smithi130=b;smithi130=e;smithi130=h;smithi130=k;smithi191=c;smithi191=f;smithi191=i;smithi191=l;count:12 2023-07-14T19:15:27.243 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:15:26 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:15:27.243 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:15:26 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T19:15:27.243 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:15:26 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:15:27.243 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:15:26 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T19:15:27.244 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:15:26 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:15:27.244 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:15:26 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-14T19:15:27.244 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:15:26 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-14T19:15:27.244 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:15:26 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:15:27.359 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:27.358+0000 7f1545d9e700 1 -- 172.21.15.40:0/2064567983 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f15401006e0 msgr2=0x7f1540102ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:15:27.360 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:27.358+0000 7f1545d9e700 1 --2- 172.21.15.40:0/2064567983 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f15401006e0 0x7f1540102ac0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f1530009990 tx=0x7f1530009ca0 comp rx=0 tx=0).stop 2023-07-14T19:15:27.360 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:27.358+0000 7f1545d9e700 1 -- 172.21.15.40:0/2064567983 shutdown_connections 2023-07-14T19:15:27.360 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:27.358+0000 7f1545d9e700 1 --2- 172.21.15.40:0/2064567983 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1540105920 0x7f1540107d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:15:27.360 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:27.358+0000 7f1545d9e700 1 --2- 172.21.15.40:0/2064567983 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f15401006e0 0x7f1540102ac0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:15:27.360 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:27.358+0000 7f1545d9e700 1 --2- 172.21.15.40:0/2064567983 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1540103000 0x7f15401053e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:15:27.361 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:27.358+0000 7f1545d9e700 1 -- 172.21.15.40:0/2064567983 >> 172.21.15.40:0/2064567983 conn(0x7f15400fa100 msgr2=0x7f15400fc550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:15:27.361 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:27.358+0000 7f1545d9e700 1 -- 172.21.15.40:0/2064567983 shutdown_connections 2023-07-14T19:15:27.361 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:27.358+0000 7f1545d9e700 1 -- 172.21.15.40:0/2064567983 wait complete. 2023-07-14T19:15:27.361 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:27.359+0000 7f1545d9e700 1 Processor -- start 2023-07-14T19:15:27.363 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:27.359+0000 7f1545d9e700 1 -- start start 2023-07-14T19:15:27.363 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:27.359+0000 7f1545d9e700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f15401006e0 0x7f15401959a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:15:27.363 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:27.359+0000 7f1545d9e700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1540103000 0x7f1540195ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:15:27.363 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:27.359+0000 7f1545d9e700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1540105920 0x7f154019cf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:15:27.364 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:27.359+0000 7f1545d9e700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f154019d4d0 con 0x7f15401006e0 2023-07-14T19:15:27.364 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:27.359+0000 7f1545d9e700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f154019d610 con 0x7f1540105920 2023-07-14T19:15:27.364 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:27.359+0000 7f1545d9e700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f154019d7a0 con 0x7f1540103000 2023-07-14T19:15:27.364 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:27.359+0000 7f153f7fe700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f15401006e0 0x7f15401959a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:15:27.364 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:27.359+0000 7f153f7fe700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f15401006e0 0x7f15401959a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:60790/0 (socket says 172.21.15.40:60790) 2023-07-14T19:15:27.364 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:27.359+0000 7f153f7fe700 1 -- 172.21.15.40:0/3648634055 learned_addr learned my addr 172.21.15.40:0/3648634055 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-07-14T19:15:27.365 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:27.359+0000 7f153effd700 1 --2- 172.21.15.40:0/3648634055 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1540103000 0x7f1540195ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:15:27.365 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:27.359+0000 7f153ffff700 1 --2- 172.21.15.40:0/3648634055 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1540105920 0x7f154019cf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:15:27.365 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:27.360+0000 7f153f7fe700 1 -- 172.21.15.40:0/3648634055 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1540103000 msgr2=0x7f1540195ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:15:27.365 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:27.360+0000 7f153f7fe700 1 --2- 172.21.15.40:0/3648634055 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1540103000 0x7f1540195ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:15:27.365 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:27.360+0000 7f153f7fe700 1 -- 172.21.15.40:0/3648634055 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1540105920 msgr2=0x7f154019cf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:15:27.365 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:27.360+0000 7f153f7fe700 1 --2- 172.21.15.40:0/3648634055 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1540105920 0x7f154019cf00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:15:27.366 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:27.360+0000 7f153f7fe700 1 -- 172.21.15.40:0/3648634055 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15300095f0 con 0x7f15401006e0 2023-07-14T19:15:27.366 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:27.360+0000 7f153f7fe700 1 --2- 172.21.15.40:0/3648634055 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f15401006e0 0x7f15401959a0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f1530000c00 tx=0x7f153000f710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:15:27.366 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:27.360+0000 7f153cff9700 1 -- 172.21.15.40:0/3648634055 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f153001d070 con 0x7f15401006e0 2023-07-14T19:15:27.366 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:27.360+0000 7f153cff9700 1 -- 172.21.15.40:0/3648634055 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f153000fb50 con 0x7f15401006e0 2023-07-14T19:15:27.366 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:27.360+0000 7f153cff9700 1 -- 172.21.15.40:0/3648634055 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15300173f0 con 0x7f15401006e0 2023-07-14T19:15:27.366 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:27.360+0000 7f1545d9e700 1 -- 172.21.15.40:0/3648634055 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f154019da20 con 0x7f15401006e0 2023-07-14T19:15:27.366 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:27.360+0000 7f1545d9e700 1 -- 172.21.15.40:0/3648634055 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f154019de80 con 0x7f15401006e0 2023-07-14T19:15:27.367 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:27.361+0000 7f1545d9e700 1 -- 172.21.15.40:0/3648634055 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f15400fbc70 con 0x7f15401006e0 2023-07-14T19:15:27.367 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:27.362+0000 7f153cff9700 1 -- 172.21.15.40:0/3648634055 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f153000fcc0 con 0x7f15401006e0 2023-07-14T19:15:27.367 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:27.363+0000 7f153cff9700 1 --2- 172.21.15.40:0/3648634055 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f15280a11a0 0x7f15280a3650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:15:27.368 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:27.363+0000 7f153cff9700 1 -- 172.21.15.40:0/3648634055 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f15300cdc80 con 0x7f15401006e0 2023-07-14T19:15:27.368 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:27.363+0000 7f153effd700 1 --2- 172.21.15.40:0/3648634055 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f15280a11a0 0x7f15280a3650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:15:27.368 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:27.363+0000 7f153effd700 1 --2- 172.21.15.40:0/3648634055 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f15280a11a0 0x7f15280a3650 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f1534005950 tx=0x7f153400b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:15:27.368 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:27.366+0000 7f153cff9700 1 -- 172.21.15.40:0/3648634055 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f1530097860 con 0x7f15401006e0 2023-07-14T19:15:27.613 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:27.612+0000 7f1545d9e700 1 -- 172.21.15.40:0/3648634055 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f154002c8e0 con 0x7f15401006e0 2023-07-14T19:15:27.613 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:27.612+0000 7f153cff9700 1 -- 172.21.15.40:0/3648634055 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 1 v1) v1 ==== 93+0+501 (secure 0 0 0) 0x7f153009ae80 con 0x7f15401006e0 2023-07-14T19:15:27.613 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-07-14T19:15:27.613 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":1,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[]} 2023-07-14T19:15:27.615 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:27.614+0000 7f1545d9e700 1 -- 172.21.15.40:0/3648634055 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f15280a11a0 msgr2=0x7f15280a3650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:15:27.616 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:27.614+0000 7f1545d9e700 1 --2- 172.21.15.40:0/3648634055 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f15280a11a0 0x7f15280a3650 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f1534005950 tx=0x7f153400b410 comp rx=0 tx=0).stop 2023-07-14T19:15:27.616 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:27.614+0000 7f1545d9e700 1 -- 172.21.15.40:0/3648634055 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f15401006e0 msgr2=0x7f15401959a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:15:27.616 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:27.614+0000 7f1545d9e700 1 --2- 172.21.15.40:0/3648634055 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f15401006e0 0x7f15401959a0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f1530000c00 tx=0x7f153000f710 comp rx=0 tx=0).stop 2023-07-14T19:15:27.616 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:27.614+0000 7f1545d9e700 1 -- 172.21.15.40:0/3648634055 shutdown_connections 2023-07-14T19:15:27.616 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:27.614+0000 7f1545d9e700 1 --2- 172.21.15.40:0/3648634055 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f15280a11a0 0x7f15280a3650 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:15:27.616 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:27.614+0000 7f1545d9e700 1 --2- 172.21.15.40:0/3648634055 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1540105920 0x7f154019cf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:15:27.617 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:27.614+0000 7f1545d9e700 1 --2- 172.21.15.40:0/3648634055 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f15401006e0 0x7f15401959a0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:15:27.617 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:27.614+0000 7f1545d9e700 1 --2- 172.21.15.40:0/3648634055 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1540103000 0x7f1540195ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:15:27.617 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:27.614+0000 7f1545d9e700 1 -- 172.21.15.40:0/3648634055 >> 172.21.15.40:0/3648634055 conn(0x7f15400fa100 msgr2=0x7f1540101360 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:15:27.617 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:27.614+0000 7f1545d9e700 1 -- 172.21.15.40:0/3648634055 shutdown_connections 2023-07-14T19:15:27.617 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:27.615+0000 7f1545d9e700 1 -- 172.21.15.40:0/3648634055 wait complete. 2023-07-14T19:15:27.618 INFO:teuthology.orchestra.run.smithi040.stderr:dumped fsmap epoch 1 2023-07-14T19:15:27.722 INFO:journalctl@ceph.mds.l.smithi191.stdout:Jul 14 19:15:27 smithi191 systemd[1]: Starting Ceph mds.l for adc52f04-2279-11ee-9b33-001a4aab830c... 2023-07-14T19:15:27.971 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:15:27 smithi040 ceph-mon[96518]: Deploying daemon mds.l on smithi191 2023-07-14T19:15:27.971 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:15:27 smithi040 ceph-mon[96518]: from='client.? 172.21.15.40:0/2319755287' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T19:15:27.971 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:15:27 smithi040 ceph-mon[96518]: from='client.? 172.21.15.40:0/2676909995' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-07-14T19:15:27.971 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:15:27 smithi040 ceph-mon[96518]: from='client.? 172.21.15.40:0/3648634055' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T19:15:28.003 INFO:journalctl@ceph.mds.l.smithi191.stdout:Jul 14 19:15:27 smithi191 podman[123505]: 2023-07-14T19:15:28.004 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:15:27 smithi191 ceph-mon[98311]: Deploying daemon mds.l on smithi191 2023-07-14T19:15:28.004 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:15:27 smithi191 ceph-mon[98311]: from='client.? 172.21.15.40:0/2319755287' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T19:15:28.005 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:15:27 smithi191 ceph-mon[98311]: from='client.? 172.21.15.40:0/2676909995' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-07-14T19:15:28.005 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:15:27 smithi191 ceph-mon[98311]: from='client.? 172.21.15.40:0/3648634055' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T19:15:28.242 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:15:27 smithi130 ceph-mon[97832]: Deploying daemon mds.l on smithi191 2023-07-14T19:15:28.242 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:15:27 smithi130 ceph-mon[97832]: from='client.? 172.21.15.40:0/2319755287' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T19:15:28.242 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:15:27 smithi130 ceph-mon[97832]: from='client.? 172.21.15.40:0/2676909995' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-07-14T19:15:28.243 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:15:27 smithi130 ceph-mon[97832]: from='client.? 172.21.15.40:0/3648634055' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T19:15:28.472 INFO:journalctl@ceph.mds.l.smithi191.stdout:Jul 14 19:15:28 smithi191 ceph-mds[123546]: -- 172.21.15.191:0/673717570 <== mon.1 v2:172.21.15.191:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55ba8277b1e0 con 0x55ba82750c00 2023-07-14T19:15:28.472 INFO:journalctl@ceph.mds.l.smithi191.stdout:Jul 14 19:15:28 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-mds-l[123521]: starting mds.l at 2023-07-14T19:15:28.472 INFO:journalctl@ceph.mds.l.smithi191.stdout:Jul 14 19:15:28 smithi191 bash[123505]: aba4b62e9f98efb6c110367e6e7d1512e522a20d4f17b895e378ac0f5495da93 2023-07-14T19:15:28.472 INFO:journalctl@ceph.mds.l.smithi191.stdout:Jul 14 19:15:28 smithi191 systemd[1]: Started Ceph mds.l for adc52f04-2279-11ee-9b33-001a4aab830c. 2023-07-14T19:15:28.993 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:15:28 smithi130 ceph-mon[97832]: pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T19:15:28.993 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:15:28 smithi130 ceph-mon[97832]: mds.? [v2:172.21.15.191:6832/31509862,v1:172.21.15.191:6833/31509862] up:boot 2023-07-14T19:15:28.993 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:15:28 smithi130 ceph-mon[97832]: fsmap 1 up:standby 2023-07-14T19:15:28.994 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:15:28 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2023-07-14T19:15:28.994 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:15:28 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:15:28.994 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:15:28 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:15:28.995 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:15:28 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:15:28.995 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:15:28 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-14T19:15:28.995 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:15:28 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-14T19:15:28.995 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:15:28 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:15:29.221 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:15:28 smithi040 ceph-mon[96518]: pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T19:15:29.221 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:15:28 smithi040 ceph-mon[96518]: mds.? [v2:172.21.15.191:6832/31509862,v1:172.21.15.191:6833/31509862] up:boot 2023-07-14T19:15:29.222 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:15:28 smithi040 ceph-mon[96518]: fsmap 1 up:standby 2023-07-14T19:15:29.222 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:15:28 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2023-07-14T19:15:29.222 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:15:28 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:15:29.222 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:15:28 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:15:29.222 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:15:28 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:15:29.222 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:15:28 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-14T19:15:29.222 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:15:28 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-14T19:15:29.223 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:15:28 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:15:29.223 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:15:28 smithi191 ceph-mon[98311]: pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T19:15:29.223 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:15:28 smithi191 ceph-mon[98311]: mds.? [v2:172.21.15.191:6832/31509862,v1:172.21.15.191:6833/31509862] up:boot 2023-07-14T19:15:29.223 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:15:28 smithi191 ceph-mon[98311]: fsmap 1 up:standby 2023-07-14T19:15:29.223 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:15:28 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2023-07-14T19:15:29.224 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:15:28 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:15:29.224 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:15:28 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:15:29.224 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:15:28 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:15:29.224 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:15:28 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-14T19:15:29.224 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:15:28 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-14T19:15:29.224 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:15:28 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:15:29.629 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-07-14T19:15:29.742 INFO:journalctl@ceph.mds.k.smithi130.stdout:Jul 14 19:15:29 smithi130 systemd[1]: Starting Ceph mds.k for adc52f04-2279-11ee-9b33-001a4aab830c... 2023-07-14T19:15:29.770 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:29.769+0000 7f1ea0825700 1 -- 172.21.15.40:0/523532930 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1e9c102fa0 msgr2=0x7f1e9c105380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:15:29.771 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:29.769+0000 7f1ea0825700 1 --2- 172.21.15.40:0/523532930 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1e9c102fa0 0x7f1e9c105380 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f1e90009990 tx=0x7f1e90009ca0 comp rx=0 tx=0).stop 2023-07-14T19:15:29.771 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:29.770+0000 7f1ea0825700 1 -- 172.21.15.40:0/523532930 shutdown_connections 2023-07-14T19:15:29.771 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:29.770+0000 7f1ea0825700 1 --2- 172.21.15.40:0/523532930 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1e9c100680 0x7f1e9c102a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:15:29.771 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:29.770+0000 7f1ea0825700 1 --2- 172.21.15.40:0/523532930 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1e9c1058c0 0x7f1e9c107d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:15:29.771 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:29.770+0000 7f1ea0825700 1 --2- 172.21.15.40:0/523532930 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1e9c102fa0 0x7f1e9c105380 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:15:29.771 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:29.770+0000 7f1ea0825700 1 -- 172.21.15.40:0/523532930 >> 172.21.15.40:0/523532930 conn(0x7f1e9c0fa100 msgr2=0x7f1e9c0fc550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:15:29.772 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:29.770+0000 7f1ea0825700 1 -- 172.21.15.40:0/523532930 shutdown_connections 2023-07-14T19:15:29.772 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:29.770+0000 7f1ea0825700 1 -- 172.21.15.40:0/523532930 wait complete. 2023-07-14T19:15:29.774 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:29.770+0000 7f1ea0825700 1 Processor -- start 2023-07-14T19:15:29.774 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:29.770+0000 7f1ea0825700 1 -- start start 2023-07-14T19:15:29.774 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:29.771+0000 7f1ea0825700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1e9c100680 0x7f1e9c1958d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:15:29.774 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:29.771+0000 7f1ea0825700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1e9c102fa0 0x7f1e9c195e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:15:29.774 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:29.771+0000 7f1ea0825700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1e9c1058c0 0x7f1e9c19ce80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:15:29.774 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:29.771+0000 7f1ea0825700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e9c19d450 con 0x7f1e9c102fa0 2023-07-14T19:15:29.775 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:29.771+0000 7f1ea0825700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e9c19d590 con 0x7f1e9c1058c0 2023-07-14T19:15:29.775 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:29.771+0000 7f1ea0825700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e9c19d720 con 0x7f1e9c100680 2023-07-14T19:15:29.775 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:29.771+0000 7f1e99d9b700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1e9c102fa0 0x7f1e9c195e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:15:29.775 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:29.771+0000 7f1e99d9b700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1e9c102fa0 0x7f1e9c195e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:60806/0 (socket says 172.21.15.40:60806) 2023-07-14T19:15:29.775 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:29.771+0000 7f1e99d9b700 1 -- 172.21.15.40:0/3690939942 learned_addr learned my addr 172.21.15.40:0/3690939942 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-07-14T19:15:29.775 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:29.771+0000 7f1e9ad9d700 1 --2- 172.21.15.40:0/3690939942 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1e9c1058c0 0x7f1e9c19ce80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:15:29.775 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:29.771+0000 7f1e9a59c700 1 --2- 172.21.15.40:0/3690939942 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1e9c100680 0x7f1e9c1958d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:15:29.775 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:29.771+0000 7f1e99d9b700 1 -- 172.21.15.40:0/3690939942 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1e9c100680 msgr2=0x7f1e9c1958d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:15:29.776 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:29.771+0000 7f1e99d9b700 1 --2- 172.21.15.40:0/3690939942 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1e9c100680 0x7f1e9c1958d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:15:29.776 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:29.771+0000 7f1e99d9b700 1 -- 172.21.15.40:0/3690939942 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1e9c1058c0 msgr2=0x7f1e9c19ce80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:15:29.776 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:29.771+0000 7f1e99d9b700 1 --2- 172.21.15.40:0/3690939942 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1e9c1058c0 0x7f1e9c19ce80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:15:29.776 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:29.771+0000 7f1e99d9b700 1 -- 172.21.15.40:0/3690939942 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e900095f0 con 0x7f1e9c102fa0 2023-07-14T19:15:29.776 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:29.771+0000 7f1e9ad9d700 1 --2- 172.21.15.40:0/3690939942 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1e9c1058c0 0x7f1e9c19ce80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-14T19:15:29.776 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:29.771+0000 7f1e99d9b700 1 --2- 172.21.15.40:0/3690939942 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1e9c102fa0 0x7f1e9c195e10 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f1e900059d0 tx=0x7f1e90004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:15:29.777 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:29.774+0000 7f1e8b7fe700 1 -- 172.21.15.40:0/3690939942 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e9001d070 con 0x7f1e9c102fa0 2023-07-14T19:15:29.777 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:29.774+0000 7f1ea0825700 1 -- 172.21.15.40:0/3690939942 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1e9c19d9a0 con 0x7f1e9c102fa0 2023-07-14T19:15:29.778 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:29.774+0000 7f1e8b7fe700 1 -- 172.21.15.40:0/3690939942 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f1e9000fb90 con 0x7f1e9c102fa0 2023-07-14T19:15:29.778 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:29.774+0000 7f1e8b7fe700 1 -- 172.21.15.40:0/3690939942 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e900173f0 con 0x7f1e9c102fa0 2023-07-14T19:15:29.778 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:29.774+0000 7f1ea0825700 1 -- 172.21.15.40:0/3690939942 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1e9c19de20 con 0x7f1e9c102fa0 2023-07-14T19:15:29.778 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:29.775+0000 7f1ea0825700 1 -- 172.21.15.40:0/3690939942 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1e9c10b3c0 con 0x7f1e9c102fa0 2023-07-14T19:15:29.778 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:29.776+0000 7f1e8b7fe700 1 -- 172.21.15.40:0/3690939942 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f1e9000fd00 con 0x7f1e9c102fa0 2023-07-14T19:15:29.778 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:29.776+0000 7f1e8b7fe700 1 --2- 172.21.15.40:0/3690939942 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f1e840a11a0 0x7f1e840a3650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:15:29.778 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:29.776+0000 7f1e9a59c700 1 --2- 172.21.15.40:0/3690939942 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f1e840a11a0 0x7f1e840a3650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:15:29.779 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:29.776+0000 7f1e8b7fe700 1 -- 172.21.15.40:0/3690939942 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f1e900cd840 con 0x7f1e9c102fa0 2023-07-14T19:15:29.779 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:29.777+0000 7f1e9a59c700 1 --2- 172.21.15.40:0/3690939942 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f1e840a11a0 0x7f1e840a3650 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f1e8c009c60 tx=0x7f1e8c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:15:29.781 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:29.780+0000 7f1e8b7fe700 1 -- 172.21.15.40:0/3690939942 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f1e9005c780 con 0x7f1e9c102fa0 2023-07-14T19:15:29.971 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:15:29 smithi040 ceph-mon[96518]: Deploying daemon mds.k on smithi130 2023-07-14T19:15:30.025 INFO:journalctl@ceph.mds.k.smithi130.stdout:Jul 14 19:15:29 smithi130 podman[121802]: 2023-07-14T19:15:30.025 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:15:29 smithi130 ceph-mon[97832]: Deploying daemon mds.k on smithi130 2023-07-14T19:15:30.026 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:30.025+0000 7f1ea0825700 1 -- 172.21.15.40:0/3690939942 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1e9c19e460 con 0x7f1e9c102fa0 2023-07-14T19:15:30.026 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:30.025+0000 7f1e8b7fe700 1 -- 172.21.15.40:0/3690939942 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 2 v2) v1 ==== 93+0+1254 (secure 0 0 0) 0x7f1e900974c0 con 0x7f1e9c102fa0 2023-07-14T19:15:30.026 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-07-14T19:15:30.026 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":2,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24433,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6833/31509862","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":31509862},{"type":"v1","addr":"172.21.15.191:6833","nonce":31509862}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[]} 2023-07-14T19:15:30.028 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:30.027+0000 7f1ea0825700 1 -- 172.21.15.40:0/3690939942 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f1e840a11a0 msgr2=0x7f1e840a3650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:15:30.028 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:30.027+0000 7f1ea0825700 1 --2- 172.21.15.40:0/3690939942 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f1e840a11a0 0x7f1e840a3650 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f1e8c009c60 tx=0x7f1e8c009260 comp rx=0 tx=0).stop 2023-07-14T19:15:30.028 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:30.027+0000 7f1ea0825700 1 -- 172.21.15.40:0/3690939942 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1e9c102fa0 msgr2=0x7f1e9c195e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:15:30.029 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:30.027+0000 7f1ea0825700 1 --2- 172.21.15.40:0/3690939942 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1e9c102fa0 0x7f1e9c195e10 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f1e900059d0 tx=0x7f1e90004730 comp rx=0 tx=0).stop 2023-07-14T19:15:30.029 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:30.027+0000 7f1ea0825700 1 -- 172.21.15.40:0/3690939942 shutdown_connections 2023-07-14T19:15:30.029 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:30.027+0000 7f1ea0825700 1 --2- 172.21.15.40:0/3690939942 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f1e840a11a0 0x7f1e840a3650 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:15:30.029 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:30.027+0000 7f1ea0825700 1 --2- 172.21.15.40:0/3690939942 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1e9c100680 0x7f1e9c1958d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:15:30.029 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:30.027+0000 7f1ea0825700 1 --2- 172.21.15.40:0/3690939942 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1e9c1058c0 0x7f1e9c19ce80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:15:30.029 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:30.027+0000 7f1ea0825700 1 --2- 172.21.15.40:0/3690939942 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1e9c102fa0 0x7f1e9c195e10 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:15:30.029 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:30.027+0000 7f1ea0825700 1 -- 172.21.15.40:0/3690939942 >> 172.21.15.40:0/3690939942 conn(0x7f1e9c0fa100 msgr2=0x7f1e9c101ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:15:30.030 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:30.027+0000 7f1ea0825700 1 -- 172.21.15.40:0/3690939942 shutdown_connections 2023-07-14T19:15:30.030 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:30.027+0000 7f1ea0825700 1 -- 172.21.15.40:0/3690939942 wait complete. 2023-07-14T19:15:30.030 INFO:teuthology.orchestra.run.smithi040.stderr:dumped fsmap epoch 2 2023-07-14T19:15:30.222 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:15:29 smithi191 ceph-mon[98311]: Deploying daemon mds.k on smithi130 2023-07-14T19:15:30.492 INFO:journalctl@ceph.mds.k.smithi130.stdout:Jul 14 19:15:30 smithi130 bash[121802]: fb7eb8e6ffbb497bd8f7a32ba96e9e57912cd22ffc704ca78360a438082cab37 2023-07-14T19:15:30.492 INFO:journalctl@ceph.mds.k.smithi130.stdout:Jul 14 19:15:30 smithi130 ceph-mds[121842]: -- 172.21.15.130:0/111678147 <== mon.2 v2:172.21.15.130:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x564ff6e851e0 con 0x564ff6e5a800 2023-07-14T19:15:30.493 INFO:journalctl@ceph.mds.k.smithi130.stdout:Jul 14 19:15:30 smithi130 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-mds-k[121818]: starting mds.k at 2023-07-14T19:15:30.493 INFO:journalctl@ceph.mds.k.smithi130.stdout:Jul 14 19:15:30 smithi130 systemd[1]: Started Ceph mds.k for adc52f04-2279-11ee-9b33-001a4aab830c. 2023-07-14T19:15:31.129 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:15:30 smithi040 ceph-mon[96518]: pgmap v236: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T19:15:31.130 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:15:30 smithi040 ceph-mon[96518]: from='client.? 172.21.15.40:0/3690939942' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T19:15:31.130 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:15:30 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:15:31.130 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:15:30 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:15:31.130 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:15:30 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:15:31.130 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:15:30 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-14T19:15:31.130 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:15:30 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-14T19:15:31.130 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:15:30 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:15:31.222 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:15:30 smithi191 ceph-mon[98311]: pgmap v236: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T19:15:31.222 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:15:30 smithi191 ceph-mon[98311]: from='client.? 172.21.15.40:0/3690939942' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T19:15:31.222 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:15:30 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:15:31.222 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:15:30 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:15:31.223 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:15:30 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:15:31.223 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:15:30 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-14T19:15:31.223 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:15:30 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-14T19:15:31.223 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:15:30 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:15:31.242 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:15:30 smithi130 ceph-mon[97832]: pgmap v236: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T19:15:31.242 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:15:30 smithi130 ceph-mon[97832]: from='client.? 172.21.15.40:0/3690939942' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T19:15:31.243 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:15:30 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:15:31.243 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:15:30 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:15:31.243 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:15:30 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:15:31.243 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:15:30 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-14T19:15:31.243 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:15:30 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-14T19:15:31.243 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:15:30 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:15:32.041 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-07-14T19:15:32.070 INFO:journalctl@ceph.mds.j.smithi040.stdout:Jul 14 19:15:31 smithi040 systemd[1]: Starting Ceph mds.j for adc52f04-2279-11ee-9b33-001a4aab830c... 2023-07-14T19:15:32.070 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:15:31 smithi040 ceph-mon[96518]: Deploying daemon mds.j on smithi040 2023-07-14T19:15:32.070 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:15:31 smithi040 ceph-mon[96518]: mds.? [v2:172.21.15.130:6832/3677452688,v1:172.21.15.130:6833/3677452688] up:boot 2023-07-14T19:15:32.071 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:15:31 smithi040 ceph-mon[96518]: fsmap 2 up:standby 2023-07-14T19:15:32.071 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:15:31 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2023-07-14T19:15:32.180 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:32.176+0000 7fb571719700 1 -- 172.21.15.40:0/2361066343 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb56c0ff6b0 msgr2=0x7fb56c106f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:15:32.180 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:32.176+0000 7fb571719700 1 --2- 172.21.15.40:0/2361066343 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb56c0ff6b0 0x7fb56c106f60 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fb56400b0a0 tx=0x7fb56400b3b0 comp rx=0 tx=0).stop 2023-07-14T19:15:32.180 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:32.176+0000 7fb571719700 1 -- 172.21.15.40:0/2361066343 shutdown_connections 2023-07-14T19:15:32.180 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:32.176+0000 7fb571719700 1 --2- 172.21.15.40:0/2361066343 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb56c1000b0 0x7fb56c1004c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:15:32.181 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:32.176+0000 7fb571719700 1 --2- 172.21.15.40:0/2361066343 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb56c0ff6b0 0x7fb56c106f60 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:15:32.181 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:32.176+0000 7fb571719700 1 --2- 172.21.15.40:0/2361066343 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb56c0fed00 0x7fb56c0ff170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:15:32.181 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:32.176+0000 7fb571719700 1 -- 172.21.15.40:0/2361066343 >> 172.21.15.40:0/2361066343 conn(0x7fb56c0f98a0 msgr2=0x7fb56c0fbcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:15:32.181 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:32.176+0000 7fb571719700 1 -- 172.21.15.40:0/2361066343 shutdown_connections 2023-07-14T19:15:32.181 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:32.176+0000 7fb571719700 1 -- 172.21.15.40:0/2361066343 wait complete. 2023-07-14T19:15:32.181 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:32.177+0000 7fb571719700 1 Processor -- start 2023-07-14T19:15:32.181 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:32.177+0000 7fb571719700 1 -- start start 2023-07-14T19:15:32.182 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:32.177+0000 7fb571719700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb56c0fed00 0x7fb56c0fe760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:15:32.182 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:32.177+0000 7fb571719700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb56c0ff6b0 0x7fb56c19a670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:15:32.183 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:32.177+0000 7fb571719700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb56c1000b0 0x7fb56c19ccb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:15:32.183 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:32.177+0000 7fb571719700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb56c19d450 con 0x7fb56c0fed00 2023-07-14T19:15:32.183 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:32.177+0000 7fb571719700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb56c19d5c0 con 0x7fb56c0ff6b0 2023-07-14T19:15:32.183 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:32.177+0000 7fb571719700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb56c19d780 con 0x7fb56c1000b0 2023-07-14T19:15:32.183 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:32.177+0000 7fb56affd700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb56c0fed00 0x7fb56c0fe760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:15:32.183 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:32.177+0000 7fb56affd700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb56c0fed00 0x7fb56c0fe760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:60818/0 (socket says 172.21.15.40:60818) 2023-07-14T19:15:32.184 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:32.177+0000 7fb56affd700 1 -- 172.21.15.40:0/662845428 learned_addr learned my addr 172.21.15.40:0/662845428 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-07-14T19:15:32.184 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:32.178+0000 7fb56b7fe700 1 --2- 172.21.15.40:0/662845428 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb56c1000b0 0x7fb56c19ccb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:15:32.184 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:32.178+0000 7fb563fff700 1 --2- 172.21.15.40:0/662845428 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb56c0ff6b0 0x7fb56c19a670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:15:32.184 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:32.178+0000 7fb56affd700 1 -- 172.21.15.40:0/662845428 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb56c1000b0 msgr2=0x7fb56c19ccb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:15:32.184 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:32.178+0000 7fb56affd700 1 --2- 172.21.15.40:0/662845428 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb56c1000b0 0x7fb56c19ccb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:15:32.184 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:32.178+0000 7fb56affd700 1 -- 172.21.15.40:0/662845428 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb56c0ff6b0 msgr2=0x7fb56c19a670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:15:32.185 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:32.178+0000 7fb56affd700 1 --2- 172.21.15.40:0/662845428 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb56c0ff6b0 0x7fb56c19a670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:15:32.185 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:32.178+0000 7fb56affd700 1 -- 172.21.15.40:0/662845428 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb564009d00 con 0x7fb56c0fed00 2023-07-14T19:15:32.185 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:32.178+0000 7fb56b7fe700 1 --2- 172.21.15.40:0/662845428 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb56c1000b0 0x7fb56c19ccb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T19:15:32.185 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:32.178+0000 7fb56affd700 1 --2- 172.21.15.40:0/662845428 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb56c0fed00 0x7fb56c0fe760 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fb55c00d950 tx=0x7fb55c00dd10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:15:32.185 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:32.178+0000 7fb568ff9700 1 -- 172.21.15.40:0/662845428 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb55c009b40 con 0x7fb56c0fed00 2023-07-14T19:15:32.185 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:32.178+0000 7fb568ff9700 1 -- 172.21.15.40:0/662845428 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb55c009ca0 con 0x7fb56c0fed00 2023-07-14T19:15:32.185 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:32.178+0000 7fb571719700 1 -- 172.21.15.40:0/662845428 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb56c19da60 con 0x7fb56c0fed00 2023-07-14T19:15:32.185 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:32.178+0000 7fb568ff9700 1 -- 172.21.15.40:0/662845428 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb55c00f620 con 0x7fb56c0fed00 2023-07-14T19:15:32.186 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:32.178+0000 7fb571719700 1 -- 172.21.15.40:0/662845428 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb56c19df70 con 0x7fb56c0fed00 2023-07-14T19:15:32.186 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:32.179+0000 7fb571719700 1 -- 172.21.15.40:0/662845428 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb56c1058c0 con 0x7fb56c0fed00 2023-07-14T19:15:32.187 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:32.181+0000 7fb568ff9700 1 -- 172.21.15.40:0/662845428 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7fb55c010460 con 0x7fb56c0fed00 2023-07-14T19:15:32.187 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:32.181+0000 7fb568ff9700 1 --2- 172.21.15.40:0/662845428 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7fb5580a11a0 0x7fb5580a3650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:15:32.187 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:32.181+0000 7fb568ff9700 1 -- 172.21.15.40:0/662845428 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fb55c0cc7f0 con 0x7fb56c0fed00 2023-07-14T19:15:32.187 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:32.181+0000 7fb563fff700 1 --2- 172.21.15.40:0/662845428 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7fb5580a11a0 0x7fb5580a3650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:15:32.188 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:32.182+0000 7fb563fff700 1 --2- 172.21.15.40:0/662845428 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7fb5580a11a0 0x7fb5580a3650 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fb554005ea0 tx=0x7fb554005e30 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:15:32.188 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:32.186+0000 7fb568ff9700 1 -- 172.21.15.40:0/662845428 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fb55c0963e0 con 0x7fb56c0fed00 2023-07-14T19:15:32.222 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:15:31 smithi191 ceph-mon[98311]: Deploying daemon mds.j on smithi040 2023-07-14T19:15:32.222 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:15:31 smithi191 ceph-mon[98311]: mds.? [v2:172.21.15.130:6832/3677452688,v1:172.21.15.130:6833/3677452688] up:boot 2023-07-14T19:15:32.222 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:15:31 smithi191 ceph-mon[98311]: fsmap 2 up:standby 2023-07-14T19:15:32.222 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:15:31 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2023-07-14T19:15:32.242 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:15:31 smithi130 ceph-mon[97832]: Deploying daemon mds.j on smithi040 2023-07-14T19:15:32.243 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:15:31 smithi130 ceph-mon[97832]: mds.? [v2:172.21.15.130:6832/3677452688,v1:172.21.15.130:6833/3677452688] up:boot 2023-07-14T19:15:32.243 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:15:31 smithi130 ceph-mon[97832]: fsmap 2 up:standby 2023-07-14T19:15:32.243 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:15:31 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2023-07-14T19:15:32.430 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:32.429+0000 7fb571719700 1 -- 172.21.15.40:0/662845428 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb56c02c8e0 con 0x7fb56c0fed00 2023-07-14T19:15:32.430 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:32.429+0000 7fb568ff9700 1 -- 172.21.15.40:0/662845428 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) v1 ==== 93+0+2014 (secure 0 0 0) 0x7fb55c0207f0 con 0x7fb56c0fed00 2023-07-14T19:15:32.430 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-07-14T19:15:32.431 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":3,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24433,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6833/31509862","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":31509862},{"type":"v1","addr":"172.21.15.191:6833","nonce":31509862}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24452,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.130:6833/3677452688","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6832","nonce":3677452688},{"type":"v1","addr":"172.21.15.130:6833","nonce":3677452688}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[]} 2023-07-14T19:15:32.432 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:32.431+0000 7fb571719700 1 -- 172.21.15.40:0/662845428 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7fb5580a11a0 msgr2=0x7fb5580a3650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:15:32.432 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:32.431+0000 7fb571719700 1 --2- 172.21.15.40:0/662845428 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7fb5580a11a0 0x7fb5580a3650 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fb554005ea0 tx=0x7fb554005e30 comp rx=0 tx=0).stop 2023-07-14T19:15:32.433 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:32.431+0000 7fb571719700 1 -- 172.21.15.40:0/662845428 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb56c0fed00 msgr2=0x7fb56c0fe760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:15:32.433 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:32.431+0000 7fb571719700 1 --2- 172.21.15.40:0/662845428 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb56c0fed00 0x7fb56c0fe760 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fb55c00d950 tx=0x7fb55c00dd10 comp rx=0 tx=0).stop 2023-07-14T19:15:32.433 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:32.431+0000 7fb571719700 1 -- 172.21.15.40:0/662845428 shutdown_connections 2023-07-14T19:15:32.433 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:32.431+0000 7fb571719700 1 --2- 172.21.15.40:0/662845428 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7fb5580a11a0 0x7fb5580a3650 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:15:32.433 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:32.431+0000 7fb571719700 1 --2- 172.21.15.40:0/662845428 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb56c1000b0 0x7fb56c19ccb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:15:32.433 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:32.431+0000 7fb571719700 1 --2- 172.21.15.40:0/662845428 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb56c0fed00 0x7fb56c0fe760 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:15:32.433 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:32.431+0000 7fb571719700 1 --2- 172.21.15.40:0/662845428 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb56c0ff6b0 0x7fb56c19a670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:15:32.434 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:32.431+0000 7fb571719700 1 -- 172.21.15.40:0/662845428 >> 172.21.15.40:0/662845428 conn(0x7fb56c0f98a0 msgr2=0x7fb56c0fb990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:15:32.434 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:32.431+0000 7fb571719700 1 -- 172.21.15.40:0/662845428 shutdown_connections 2023-07-14T19:15:32.434 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:32.431+0000 7fb571719700 1 -- 172.21.15.40:0/662845428 wait complete. 2023-07-14T19:15:32.434 INFO:teuthology.orchestra.run.smithi040.stderr:dumped fsmap epoch 3 2023-07-14T19:15:32.443 INFO:journalctl@ceph.mds.j.smithi040.stdout:Jul 14 19:15:32 smithi040 podman[129817]: 2023-07-14T19:15:32.721 INFO:journalctl@ceph.mds.j.smithi040.stdout:Jul 14 19:15:32 smithi040 ceph-mds[129903]: -- 172.21.15.40:0/2430245372 <== mon.0 v2:172.21.15.40:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5621c03e31e0 con 0x5621c03b8800 2023-07-14T19:15:32.721 INFO:journalctl@ceph.mds.j.smithi040.stdout:Jul 14 19:15:32 smithi040 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-mds-j[129876]: starting mds.j at 2023-07-14T19:15:32.721 INFO:journalctl@ceph.mds.j.smithi040.stdout:Jul 14 19:15:32 smithi040 bash[129817]: 0561434c66d8edd1add239cdde16f81212ed99978601d86dcbabf00314ae3720 2023-07-14T19:15:32.721 INFO:journalctl@ceph.mds.j.smithi040.stdout:Jul 14 19:15:32 smithi040 systemd[1]: Started Ceph mds.j for adc52f04-2279-11ee-9b33-001a4aab830c. 2023-07-14T19:15:33.221 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:15:32 smithi040 ceph-mon[96518]: pgmap v237: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T19:15:33.221 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:15:32 smithi040 ceph-mon[96518]: from='client.? 172.21.15.40:0/662845428' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T19:15:33.221 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:15:32 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:15:33.221 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:15:32 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:15:33.222 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:15:32 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:15:33.222 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:15:32 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-14T19:15:33.222 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:15:32 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-14T19:15:33.222 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:15:32 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:15:33.222 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:15:32 smithi191 ceph-mon[98311]: pgmap v237: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T19:15:33.223 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:15:32 smithi191 ceph-mon[98311]: from='client.? 172.21.15.40:0/662845428' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T19:15:33.223 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:15:32 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:15:33.223 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:15:32 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:15:33.223 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:15:32 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:15:33.223 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:15:32 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-14T19:15:33.223 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:15:32 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-14T19:15:33.223 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:15:32 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:15:33.242 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:15:32 smithi130 ceph-mon[97832]: pgmap v237: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T19:15:33.243 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:15:32 smithi130 ceph-mon[97832]: from='client.? 172.21.15.40:0/662845428' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T19:15:33.243 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:15:32 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:15:33.243 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:15:32 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:15:33.243 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:15:32 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:15:33.243 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:15:32 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-14T19:15:33.243 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:15:32 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-14T19:15:33.243 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:15:32 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:15:33.818 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:15:33 smithi040 ceph-mon[96518]: Deploying daemon mds.d on smithi040 2023-07-14T19:15:33.819 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:15:33 smithi040 ceph-mon[96518]: mds.? [v2:172.21.15.40:6834/501143718,v1:172.21.15.40:6835/501143718] up:boot 2023-07-14T19:15:33.819 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:15:33 smithi040 ceph-mon[96518]: fsmap 3 up:standby 2023-07-14T19:15:33.819 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:15:33 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2023-07-14T19:15:34.123 INFO:journalctl@ceph.mds.d.smithi040.stdout:Jul 14 19:15:34 smithi040 systemd[1]: Starting Ceph mds.d for adc52f04-2279-11ee-9b33-001a4aab830c... 2023-07-14T19:15:34.222 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:15:33 smithi191 ceph-mon[98311]: Deploying daemon mds.d on smithi040 2023-07-14T19:15:34.222 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:15:33 smithi191 ceph-mon[98311]: mds.? [v2:172.21.15.40:6834/501143718,v1:172.21.15.40:6835/501143718] up:boot 2023-07-14T19:15:34.222 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:15:33 smithi191 ceph-mon[98311]: fsmap 3 up:standby 2023-07-14T19:15:34.222 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:15:33 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2023-07-14T19:15:34.242 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:15:33 smithi130 ceph-mon[97832]: Deploying daemon mds.d on smithi040 2023-07-14T19:15:34.243 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:15:33 smithi130 ceph-mon[97832]: mds.? [v2:172.21.15.40:6834/501143718,v1:172.21.15.40:6835/501143718] up:boot 2023-07-14T19:15:34.243 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:15:33 smithi130 ceph-mon[97832]: fsmap 3 up:standby 2023-07-14T19:15:34.243 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:15:33 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2023-07-14T19:15:34.445 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-07-14T19:15:34.581 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:34.579+0000 7f776fe05700 1 -- 172.21.15.40:0/455641594 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7768105900 msgr2=0x7f7768107d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:15:34.581 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:34.579+0000 7f776fe05700 1 --2- 172.21.15.40:0/455641594 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7768105900 0x7f7768107d60 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f776000b0a0 tx=0x7f776000b3b0 comp rx=0 tx=0).stop 2023-07-14T19:15:34.581 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:34.579+0000 7f776fe05700 1 -- 172.21.15.40:0/455641594 shutdown_connections 2023-07-14T19:15:34.581 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:34.580+0000 7f776fe05700 1 --2- 172.21.15.40:0/455641594 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f77681006c0 0x7f7768102aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:15:34.581 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:34.580+0000 7f776fe05700 1 --2- 172.21.15.40:0/455641594 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7768105900 0x7f7768107d60 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:15:34.582 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:34.580+0000 7f776fe05700 1 --2- 172.21.15.40:0/455641594 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7768102fe0 0x7f77681053c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:15:34.582 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:34.580+0000 7f776fe05700 1 -- 172.21.15.40:0/455641594 >> 172.21.15.40:0/455641594 conn(0x7f77680fa160 msgr2=0x7f77680fc590 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:15:34.582 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:34.580+0000 7f776fe05700 1 -- 172.21.15.40:0/455641594 shutdown_connections 2023-07-14T19:15:34.582 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:34.580+0000 7f776fe05700 1 -- 172.21.15.40:0/455641594 wait complete. 2023-07-14T19:15:34.582 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:34.580+0000 7f776fe05700 1 Processor -- start 2023-07-14T19:15:34.582 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:34.580+0000 7f776fe05700 1 -- start start 2023-07-14T19:15:34.584 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:34.580+0000 7f776fe05700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f77681006c0 0x7f7768100030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:15:34.584 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:34.580+0000 7f776fe05700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7768102fe0 0x7f77680fe670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:15:34.585 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:34.580+0000 7f776fe05700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7768105900 0x7f77680febb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:15:34.585 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:34.580+0000 7f776fe05700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77680ff350 con 0x7f7768105900 2023-07-14T19:15:34.585 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:34.580+0000 7f776fe05700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77680ff4c0 con 0x7f77681006c0 2023-07-14T19:15:34.585 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:34.580+0000 7f776fe05700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f776819f980 con 0x7f7768102fe0 2023-07-14T19:15:34.585 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:34.581+0000 7f776e3a2700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7768105900 0x7f77680febb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:15:34.585 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:34.581+0000 7f776e3a2700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7768105900 0x7f77680febb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:60850/0 (socket says 172.21.15.40:60850) 2023-07-14T19:15:34.585 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:34.581+0000 7f776e3a2700 1 -- 172.21.15.40:0/1849743656 learned_addr learned my addr 172.21.15.40:0/1849743656 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-07-14T19:15:34.586 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:34.581+0000 7f776d3a0700 1 --2- 172.21.15.40:0/1849743656 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7768102fe0 0x7f77680fe670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:15:34.586 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:34.581+0000 7f776dba1700 1 --2- 172.21.15.40:0/1849743656 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f77681006c0 0x7f7768100030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:15:34.586 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:34.581+0000 7f776e3a2700 1 -- 172.21.15.40:0/1849743656 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7768102fe0 msgr2=0x7f77680fe670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:15:34.586 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:34.581+0000 7f776e3a2700 1 --2- 172.21.15.40:0/1849743656 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7768102fe0 0x7f77680fe670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:15:34.586 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:34.581+0000 7f776e3a2700 1 -- 172.21.15.40:0/1849743656 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f77681006c0 msgr2=0x7f7768100030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:15:34.586 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:34.581+0000 7f776e3a2700 1 --2- 172.21.15.40:0/1849743656 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f77681006c0 0x7f7768100030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:15:34.586 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:34.581+0000 7f776e3a2700 1 -- 172.21.15.40:0/1849743656 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7760009d00 con 0x7f7768105900 2023-07-14T19:15:34.587 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:34.581+0000 7f776dba1700 1 --2- 172.21.15.40:0/1849743656 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f77681006c0 0x7f7768100030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T19:15:34.587 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:34.581+0000 7f776e3a2700 1 --2- 172.21.15.40:0/1849743656 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7768105900 0x7f77680febb0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f77600062a0 tx=0x7f7760004db0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:15:34.587 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:34.581+0000 7f775effd700 1 -- 172.21.15.40:0/1849743656 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f776000e040 con 0x7f7768105900 2023-07-14T19:15:34.587 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:34.581+0000 7f775effd700 1 -- 172.21.15.40:0/1849743656 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f776001db50 con 0x7f7768105900 2023-07-14T19:15:34.587 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:34.581+0000 7f776fe05700 1 -- 172.21.15.40:0/1849743656 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f776819fc00 con 0x7f7768105900 2023-07-14T19:15:34.587 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:34.581+0000 7f775effd700 1 -- 172.21.15.40:0/1849743656 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f776001bc40 con 0x7f7768105900 2023-07-14T19:15:34.587 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:34.581+0000 7f776fe05700 1 -- 172.21.15.40:0/1849743656 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f77681a00d0 con 0x7f7768105900 2023-07-14T19:15:34.587 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:34.582+0000 7f776fe05700 1 -- 172.21.15.40:0/1849743656 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f77680fbcd0 con 0x7f7768105900 2023-07-14T19:15:34.588 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:34.583+0000 7f775effd700 1 -- 172.21.15.40:0/1849743656 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f776001d690 con 0x7f7768105900 2023-07-14T19:15:34.588 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:34.584+0000 7f775effd700 1 --2- 172.21.15.40:0/1849743656 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f77540a1220 0x7f77540a36d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:15:34.589 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:34.584+0000 7f776dba1700 1 --2- 172.21.15.40:0/1849743656 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f77540a1220 0x7f77540a36d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:15:34.589 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:34.584+0000 7f775effd700 1 -- 172.21.15.40:0/1849743656 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f77600ce720 con 0x7f7768105900 2023-07-14T19:15:34.589 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:34.584+0000 7f776dba1700 1 --2- 172.21.15.40:0/1849743656 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f77540a1220 0x7f77540a36d0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f7758005fd0 tx=0x7f7758005e30 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:15:34.589 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:34.587+0000 7f775effd700 1 -- 172.21.15.40:0/1849743656 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f77600983a0 con 0x7f7768105900 2023-07-14T19:15:34.721 INFO:journalctl@ceph.mds.d.smithi040.stdout:Jul 14 19:15:34 smithi040 podman[130213]: 2023-07-14T19:15:34.844 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:34.842+0000 7f776fe05700 1 -- 172.21.15.40:0/1849743656 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f776802c8e0 con 0x7f7768105900 2023-07-14T19:15:34.844 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:34.843+0000 7f775effd700 1 -- 172.21.15.40:0/1849743656 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 5 v5) v1 ==== 93+0+3525 (secure 0 0 0) 0x7f7760021de0 con 0x7f7768105900 2023-07-14T19:15:34.845 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-07-14T19:15:34.845 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":5,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14523,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6835/501143718","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6834","nonce":501143718},{"type":"v1","addr":"172.21.15.40:6835","nonce":501143718}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14535,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6837/2634476117","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6836","nonce":2634476117},{"type":"v1","addr":"172.21.15.40:6837","nonce":2634476117}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":24433,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6833/31509862","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":31509862},{"type":"v1","addr":"172.21.15.191:6833","nonce":31509862}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24452,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.130:6833/3677452688","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6832","nonce":3677452688},{"type":"v1","addr":"172.21.15.130:6833","nonce":3677452688}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[]} 2023-07-14T19:15:34.847 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:34.845+0000 7f776fe05700 1 -- 172.21.15.40:0/1849743656 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f77540a1220 msgr2=0x7f77540a36d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:15:34.847 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:34.845+0000 7f776fe05700 1 --2- 172.21.15.40:0/1849743656 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f77540a1220 0x7f77540a36d0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f7758005fd0 tx=0x7f7758005e30 comp rx=0 tx=0).stop 2023-07-14T19:15:34.847 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:34.845+0000 7f776fe05700 1 -- 172.21.15.40:0/1849743656 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7768105900 msgr2=0x7f77680febb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:15:34.847 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:34.845+0000 7f776fe05700 1 --2- 172.21.15.40:0/1849743656 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7768105900 0x7f77680febb0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f77600062a0 tx=0x7f7760004db0 comp rx=0 tx=0).stop 2023-07-14T19:15:34.848 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:34.845+0000 7f776fe05700 1 -- 172.21.15.40:0/1849743656 shutdown_connections 2023-07-14T19:15:34.848 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:34.845+0000 7f776fe05700 1 --2- 172.21.15.40:0/1849743656 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f77540a1220 0x7f77540a36d0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:15:34.848 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:34.845+0000 7f776fe05700 1 --2- 172.21.15.40:0/1849743656 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7768105900 0x7f77680febb0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:15:34.848 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:34.845+0000 7f776fe05700 1 --2- 172.21.15.40:0/1849743656 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f77681006c0 0x7f7768100030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:15:34.848 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:34.845+0000 7f776fe05700 1 --2- 172.21.15.40:0/1849743656 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7768102fe0 0x7f77680fe670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:15:34.848 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:34.845+0000 7f776fe05700 1 -- 172.21.15.40:0/1849743656 >> 172.21.15.40:0/1849743656 conn(0x7f77680fa160 msgr2=0x7f7768101320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:15:34.849 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:34.846+0000 7f776fe05700 1 -- 172.21.15.40:0/1849743656 shutdown_connections 2023-07-14T19:15:34.849 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:34.846+0000 7f776fe05700 1 -- 172.21.15.40:0/1849743656 wait complete. 2023-07-14T19:15:34.849 INFO:teuthology.orchestra.run.smithi040.stderr:dumped fsmap epoch 5 2023-07-14T19:15:35.221 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:15:34 smithi040 ceph-mon[96518]: pgmap v238: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T19:15:35.221 INFO:journalctl@ceph.mds.d.smithi040.stdout:Jul 14 19:15:34 smithi040 ceph-mds[130296]: -- 172.21.15.40:0/3827793670 <== mon.0 v2:172.21.15.40:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x561fa9f991e0 con 0x561fa9f6e800 2023-07-14T19:15:35.221 INFO:journalctl@ceph.mds.d.smithi040.stdout:Jul 14 19:15:34 smithi040 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-mds-d[130252]: starting mds.d at 2023-07-14T19:15:35.222 INFO:journalctl@ceph.mds.d.smithi040.stdout:Jul 14 19:15:34 smithi040 bash[130213]: dfe3597584c7c96060b9a6af0716b03f21172a1cf7b14a7864997952678d8dc5 2023-07-14T19:15:35.222 INFO:journalctl@ceph.mds.d.smithi040.stdout:Jul 14 19:15:34 smithi040 systemd[1]: Started Ceph mds.d for adc52f04-2279-11ee-9b33-001a4aab830c. 2023-07-14T19:15:35.222 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:15:34 smithi191 ceph-mon[98311]: pgmap v238: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T19:15:35.242 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:15:34 smithi130 ceph-mon[97832]: pgmap v238: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T19:15:36.221 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:15:35 smithi040 ceph-mon[96518]: pgmap v239: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T19:15:36.221 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:15:35 smithi040 ceph-mon[96518]: mds.? [v2:172.21.15.40:6836/2634476117,v1:172.21.15.40:6837/2634476117] up:boot 2023-07-14T19:15:36.221 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:15:35 smithi040 ceph-mon[96518]: fsmap 4 up:standby 2023-07-14T19:15:36.221 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:15:35 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2023-07-14T19:15:36.222 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:15:35 smithi040 ceph-mon[96518]: from='client.? 172.21.15.40:0/1849743656' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T19:15:36.222 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:15:35 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:15:36.222 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:15:35 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:15:36.222 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:15:35 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:15:36.222 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:15:35 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-14T19:15:36.222 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:15:35 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-14T19:15:36.222 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:15:35 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:15:36.223 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:15:35 smithi040 ceph-mon[96518]: Deploying daemon mds.e on smithi130 2023-07-14T19:15:36.223 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:15:35 smithi191 ceph-mon[98311]: pgmap v239: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T19:15:36.223 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:15:35 smithi191 ceph-mon[98311]: mds.? [v2:172.21.15.40:6836/2634476117,v1:172.21.15.40:6837/2634476117] up:boot 2023-07-14T19:15:36.223 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:15:35 smithi191 ceph-mon[98311]: fsmap 4 up:standby 2023-07-14T19:15:36.223 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:15:35 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2023-07-14T19:15:36.224 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:15:35 smithi191 ceph-mon[98311]: from='client.? 172.21.15.40:0/1849743656' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T19:15:36.224 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:15:35 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:15:36.224 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:15:35 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:15:36.224 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:15:35 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:15:36.224 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:15:35 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-14T19:15:36.224 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:15:35 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-14T19:15:36.224 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:15:35 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:15:36.225 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:15:35 smithi191 ceph-mon[98311]: Deploying daemon mds.e on smithi130 2023-07-14T19:15:36.225 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:15:35 smithi130 ceph-mon[97832]: pgmap v239: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T19:15:36.225 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:15:35 smithi130 ceph-mon[97832]: mds.? [v2:172.21.15.40:6836/2634476117,v1:172.21.15.40:6837/2634476117] up:boot 2023-07-14T19:15:36.226 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:15:35 smithi130 ceph-mon[97832]: fsmap 4 up:standby 2023-07-14T19:15:36.226 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:15:35 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2023-07-14T19:15:36.226 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:15:35 smithi130 ceph-mon[97832]: from='client.? 172.21.15.40:0/1849743656' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T19:15:36.226 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:15:35 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:15:36.226 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:15:35 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:15:36.226 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:15:35 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:15:36.226 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:15:35 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-14T19:15:36.227 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:15:35 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-14T19:15:36.227 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:15:35 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:15:36.227 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:15:35 smithi130 ceph-mon[97832]: Deploying daemon mds.e on smithi130 2023-07-14T19:15:36.492 INFO:journalctl@ceph.mds.e.smithi130.stdout:Jul 14 19:15:36 smithi130 systemd[1]: Starting Ceph mds.e for adc52f04-2279-11ee-9b33-001a4aab830c... 2023-07-14T19:15:36.860 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-07-14T19:15:37.006 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:37.002+0000 7f4665bc1700 1 -- 172.21.15.40:0/212889630 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f46600ffcf0 msgr2=0x7f4660100140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:15:37.007 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:37.002+0000 7f4665bc1700 1 --2- 172.21.15.40:0/212889630 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f46600ffcf0 0x7f4660100140 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f4648009990 tx=0x7f4648009ca0 comp rx=0 tx=0).stop 2023-07-14T19:15:37.007 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:37.002+0000 7f4665bc1700 1 -- 172.21.15.40:0/212889630 shutdown_connections 2023-07-14T19:15:37.007 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:37.002+0000 7f4665bc1700 1 --2- 172.21.15.40:0/212889630 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4660100680 0x7f4660106f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:15:37.007 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:37.002+0000 7f4665bc1700 1 --2- 172.21.15.40:0/212889630 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f46600feb00 0x7f46600fef10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:15:37.008 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:37.002+0000 7f4665bc1700 1 --2- 172.21.15.40:0/212889630 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f46600ffcf0 0x7f4660100140 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:15:37.008 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:37.002+0000 7f4665bc1700 1 -- 172.21.15.40:0/212889630 >> 172.21.15.40:0/212889630 conn(0x7f46600fa100 msgr2=0x7f46600fc550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:15:37.008 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:37.002+0000 7f4665bc1700 1 -- 172.21.15.40:0/212889630 shutdown_connections 2023-07-14T19:15:37.009 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:37.002+0000 7f4665bc1700 1 -- 172.21.15.40:0/212889630 wait complete. 2023-07-14T19:15:37.009 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:37.003+0000 7f4665bc1700 1 Processor -- start 2023-07-14T19:15:37.009 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:37.003+0000 7f4665bc1700 1 -- start start 2023-07-14T19:15:37.009 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:37.003+0000 7f4665bc1700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f46600feb00 0x7f46601947d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:15:37.009 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:37.003+0000 7f4665bc1700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f46600ffcf0 0x7f4660194d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:15:37.009 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:37.003+0000 7f4665bc1700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4660100680 0x7f466019bd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:15:37.010 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:37.003+0000 7f4665bc1700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f466019c350 con 0x7f46600ffcf0 2023-07-14T19:15:37.010 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:37.003+0000 7f4665bc1700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f466019c490 con 0x7f46600feb00 2023-07-14T19:15:37.010 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:37.003+0000 7f4665bc1700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f466019c620 con 0x7f4660100680 2023-07-14T19:15:37.010 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:37.003+0000 7f4657fff700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f46600ffcf0 0x7f4660194d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:15:37.010 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:37.003+0000 7f4657fff700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f46600ffcf0 0x7f4660194d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:60870/0 (socket says 172.21.15.40:60870) 2023-07-14T19:15:37.010 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:37.003+0000 7f4657fff700 1 -- 172.21.15.40:0/3629472437 learned_addr learned my addr 172.21.15.40:0/3629472437 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-07-14T19:15:37.010 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:37.003+0000 7f465f7fe700 1 --2- 172.21.15.40:0/3629472437 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f46600feb00 0x7f46601947d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:15:37.011 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:37.003+0000 7f465ffff700 1 --2- 172.21.15.40:0/3629472437 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4660100680 0x7f466019bd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:15:37.011 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:37.004+0000 7f4657fff700 1 -- 172.21.15.40:0/3629472437 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4660100680 msgr2=0x7f466019bd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:15:37.011 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:37.004+0000 7f4657fff700 1 --2- 172.21.15.40:0/3629472437 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4660100680 0x7f466019bd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:15:37.011 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:37.004+0000 7f4657fff700 1 -- 172.21.15.40:0/3629472437 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f46600feb00 msgr2=0x7f46601947d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:15:37.011 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:37.004+0000 7f4657fff700 1 --2- 172.21.15.40:0/3629472437 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f46600feb00 0x7f46601947d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:15:37.011 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:37.004+0000 7f4657fff700 1 -- 172.21.15.40:0/3629472437 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46480095f0 con 0x7f46600ffcf0 2023-07-14T19:15:37.011 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:37.004+0000 7f465f7fe700 1 --2- 172.21.15.40:0/3629472437 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f46600feb00 0x7f46601947d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-14T19:15:37.011 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:37.004+0000 7f4657fff700 1 --2- 172.21.15.40:0/3629472437 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f46600ffcf0 0x7f4660194d10 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f464800b620 tx=0x7f4648004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:15:37.012 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:37.004+0000 7f465d7fa700 1 -- 172.21.15.40:0/3629472437 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f464801f070 con 0x7f46600ffcf0 2023-07-14T19:15:37.012 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:37.004+0000 7f465d7fa700 1 -- 172.21.15.40:0/3629472437 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f4648005030 con 0x7f46600ffcf0 2023-07-14T19:15:37.012 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:37.004+0000 7f465d7fa700 1 -- 172.21.15.40:0/3629472437 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4648004bf0 con 0x7f46600ffcf0 2023-07-14T19:15:37.012 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:37.004+0000 7f4665bc1700 1 -- 172.21.15.40:0/3629472437 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f46600677a0 con 0x7f46600ffcf0 2023-07-14T19:15:37.012 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:37.004+0000 7f4665bc1700 1 -- 172.21.15.40:0/3629472437 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4660067ba0 con 0x7f46600ffcf0 2023-07-14T19:15:37.012 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:37.005+0000 7f4665bc1700 1 -- 172.21.15.40:0/3629472437 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f46600685f0 con 0x7f46600ffcf0 2023-07-14T19:15:37.013 INFO:journalctl@ceph.mds.e.smithi130.stdout:Jul 14 19:15:36 smithi130 podman[122144]: 2023-07-14T19:15:37.014 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:37.006+0000 7f465d7fa700 1 -- 172.21.15.40:0/3629472437 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f4648023070 con 0x7f46600ffcf0 2023-07-14T19:15:37.014 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:37.007+0000 7f465d7fa700 1 --2- 172.21.15.40:0/3629472437 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f464c0a11a0 0x7f464c0a3650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:15:37.014 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:37.007+0000 7f465d7fa700 1 -- 172.21.15.40:0/3629472437 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f46480cfda0 con 0x7f46600ffcf0 2023-07-14T19:15:37.014 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:37.007+0000 7f465f7fe700 1 --2- 172.21.15.40:0/3629472437 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f464c0a11a0 0x7f464c0a3650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:15:37.014 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:37.007+0000 7f465f7fe700 1 --2- 172.21.15.40:0/3629472437 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f464c0a11a0 0x7f464c0a3650 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f46600ffb50 tx=0x7f4650009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:15:37.015 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:37.010+0000 7f465d7fa700 1 -- 172.21.15.40:0/3629472437 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f4648099a20 con 0x7f46600ffcf0 2023-07-14T19:15:37.258 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:37.257+0000 7f4665bc1700 1 -- 172.21.15.40:0/3629472437 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4660105a20 con 0x7f46600ffcf0 2023-07-14T19:15:37.258 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:37.257+0000 7f465d7fa700 1 -- 172.21.15.40:0/3629472437 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 5 v5) v1 ==== 93+0+3525 (secure 0 0 0) 0x7f4648033de0 con 0x7f46600ffcf0 2023-07-14T19:15:37.259 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-07-14T19:15:37.259 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":5,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14523,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6835/501143718","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6834","nonce":501143718},{"type":"v1","addr":"172.21.15.40:6835","nonce":501143718}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14535,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6837/2634476117","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6836","nonce":2634476117},{"type":"v1","addr":"172.21.15.40:6837","nonce":2634476117}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":24433,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6833/31509862","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":31509862},{"type":"v1","addr":"172.21.15.191:6833","nonce":31509862}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24452,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.130:6833/3677452688","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6832","nonce":3677452688},{"type":"v1","addr":"172.21.15.130:6833","nonce":3677452688}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[]} 2023-07-14T19:15:37.262 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:37.259+0000 7f4665bc1700 1 -- 172.21.15.40:0/3629472437 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f464c0a11a0 msgr2=0x7f464c0a3650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:15:37.262 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:37.259+0000 7f4665bc1700 1 --2- 172.21.15.40:0/3629472437 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f464c0a11a0 0x7f464c0a3650 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f46600ffb50 tx=0x7f4650009260 comp rx=0 tx=0).stop 2023-07-14T19:15:37.262 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:37.259+0000 7f4665bc1700 1 -- 172.21.15.40:0/3629472437 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f46600ffcf0 msgr2=0x7f4660194d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:15:37.262 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:37.259+0000 7f4665bc1700 1 --2- 172.21.15.40:0/3629472437 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f46600ffcf0 0x7f4660194d10 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f464800b620 tx=0x7f4648004730 comp rx=0 tx=0).stop 2023-07-14T19:15:37.263 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:37.259+0000 7f4665bc1700 1 -- 172.21.15.40:0/3629472437 shutdown_connections 2023-07-14T19:15:37.263 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:37.259+0000 7f4665bc1700 1 --2- 172.21.15.40:0/3629472437 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f464c0a11a0 0x7f464c0a3650 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:15:37.263 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:37.259+0000 7f4665bc1700 1 --2- 172.21.15.40:0/3629472437 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4660100680 0x7f466019bd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:15:37.263 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:37.259+0000 7f4665bc1700 1 --2- 172.21.15.40:0/3629472437 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f46600feb00 0x7f46601947d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:15:37.263 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:37.259+0000 7f4665bc1700 1 --2- 172.21.15.40:0/3629472437 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f46600ffcf0 0x7f4660194d10 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:15:37.263 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:37.259+0000 7f4665bc1700 1 -- 172.21.15.40:0/3629472437 >> 172.21.15.40:0/3629472437 conn(0x7f46600fa100 msgr2=0x7f4660103180 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:15:37.264 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:37.259+0000 7f4665bc1700 1 -- 172.21.15.40:0/3629472437 shutdown_connections 2023-07-14T19:15:37.264 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:37.259+0000 7f4665bc1700 1 -- 172.21.15.40:0/3629472437 wait complete. 2023-07-14T19:15:37.264 INFO:teuthology.orchestra.run.smithi040.stderr:dumped fsmap epoch 5 2023-07-14T19:15:37.492 INFO:journalctl@ceph.mds.e.smithi130.stdout:Jul 14 19:15:37 smithi130 ceph-mds[122184]: -- 172.21.15.130:0/2789153448 <== mon.2 v2:172.21.15.130:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5555a4c331e0 con 0x5555a4c08800 2023-07-14T19:15:37.493 INFO:journalctl@ceph.mds.e.smithi130.stdout:Jul 14 19:15:37 smithi130 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-mds-e[122160]: starting mds.e at 2023-07-14T19:15:37.493 INFO:journalctl@ceph.mds.e.smithi130.stdout:Jul 14 19:15:37 smithi130 bash[122144]: 696262f80d9818a5c9227db6e4ccdb4b678802b4bd64c42477899fa2cf2804c6 2023-07-14T19:15:37.493 INFO:journalctl@ceph.mds.e.smithi130.stdout:Jul 14 19:15:37 smithi130 systemd[1]: Started Ceph mds.e for adc52f04-2279-11ee-9b33-001a4aab830c. 2023-07-14T19:15:38.266 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:15:38 smithi191 ceph-mon[98311]: pgmap v240: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T19:15:38.267 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:15:38 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:15:38.267 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:15:38 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:15:38.267 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:15:38 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:15:38.268 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:15:38 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-14T19:15:38.268 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:15:38 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-14T19:15:38.268 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:15:38 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:15:38.268 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:15:38 smithi191 ceph-mon[98311]: Deploying daemon mds.f on smithi191 2023-07-14T19:15:38.269 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:15:38 smithi191 ceph-mon[98311]: from='client.? 172.21.15.40:0/3629472437' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T19:15:38.269 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:15:38 smithi191 ceph-mon[98311]: mds.? [v2:172.21.15.130:6834/3088490808,v1:172.21.15.130:6835/3088490808] up:boot 2023-07-14T19:15:38.269 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:15:38 smithi191 ceph-mon[98311]: fsmap 5 up:standby 2023-07-14T19:15:38.269 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:15:38 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2023-07-14T19:15:38.471 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:15:38 smithi040 ceph-mon[96518]: pgmap v240: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T19:15:38.471 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:15:38 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:15:38.471 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:15:38 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:15:38.472 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:15:38 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:15:38.472 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:15:38 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-14T19:15:38.472 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:15:38 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-14T19:15:38.472 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:15:38 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:15:38.472 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:15:38 smithi040 ceph-mon[96518]: Deploying daemon mds.f on smithi191 2023-07-14T19:15:38.472 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:15:38 smithi040 ceph-mon[96518]: from='client.? 172.21.15.40:0/3629472437' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T19:15:38.472 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:15:38 smithi040 ceph-mon[96518]: mds.? [v2:172.21.15.130:6834/3088490808,v1:172.21.15.130:6835/3088490808] up:boot 2023-07-14T19:15:38.473 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:15:38 smithi040 ceph-mon[96518]: fsmap 5 up:standby 2023-07-14T19:15:38.473 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:15:38 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2023-07-14T19:15:38.492 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:15:38 smithi130 ceph-mon[97832]: pgmap v240: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T19:15:38.492 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:15:38 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:15:38.493 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:15:38 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:15:38.493 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:15:38 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:15:38.493 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:15:38 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-14T19:15:38.493 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:15:38 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-14T19:15:38.493 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:15:38 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:15:38.493 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:15:38 smithi130 ceph-mon[97832]: Deploying daemon mds.f on smithi191 2023-07-14T19:15:38.493 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:15:38 smithi130 ceph-mon[97832]: from='client.? 172.21.15.40:0/3629472437' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T19:15:38.494 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:15:38 smithi130 ceph-mon[97832]: mds.? [v2:172.21.15.130:6834/3088490808,v1:172.21.15.130:6835/3088490808] up:boot 2023-07-14T19:15:38.494 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:15:38 smithi130 ceph-mon[97832]: fsmap 5 up:standby 2023-07-14T19:15:38.494 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:15:38 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2023-07-14T19:15:38.855 INFO:journalctl@ceph.mds.f.smithi191.stdout:Jul 14 19:15:38 smithi191 systemd[1]: Starting Ceph mds.f for adc52f04-2279-11ee-9b33-001a4aab830c... 2023-07-14T19:15:39.114 INFO:journalctl@ceph.mds.f.smithi191.stdout:Jul 14 19:15:38 smithi191 podman[123847]: 2023-07-14T19:15:39.273 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-07-14T19:15:39.410 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:39.409+0000 7f1ed0a4b700 1 -- 172.21.15.40:0/1746075263 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1ecc103000 msgr2=0x7f1ecc1053e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:15:39.411 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:39.409+0000 7f1ed0a4b700 1 --2- 172.21.15.40:0/1746075263 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1ecc103000 0x7f1ecc1053e0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f1ec0009990 tx=0x7f1ec0009ca0 comp rx=0 tx=0).stop 2023-07-14T19:15:39.411 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:39.409+0000 7f1ed0a4b700 1 -- 172.21.15.40:0/1746075263 shutdown_connections 2023-07-14T19:15:39.411 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:39.409+0000 7f1ed0a4b700 1 --2- 172.21.15.40:0/1746075263 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1ecc105920 0x7f1ecc107d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:15:39.411 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:39.409+0000 7f1ed0a4b700 1 --2- 172.21.15.40:0/1746075263 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1ecc1006e0 0x7f1ecc102ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:15:39.411 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:39.409+0000 7f1ed0a4b700 1 --2- 172.21.15.40:0/1746075263 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1ecc103000 0x7f1ecc1053e0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:15:39.411 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:39.409+0000 7f1ed0a4b700 1 -- 172.21.15.40:0/1746075263 >> 172.21.15.40:0/1746075263 conn(0x7f1ecc0fa100 msgr2=0x7f1ecc0fc550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:15:39.411 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:39.409+0000 7f1ed0a4b700 1 -- 172.21.15.40:0/1746075263 shutdown_connections 2023-07-14T19:15:39.412 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:39.409+0000 7f1ed0a4b700 1 -- 172.21.15.40:0/1746075263 wait complete. 2023-07-14T19:15:39.413 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:39.410+0000 7f1ed0a4b700 1 Processor -- start 2023-07-14T19:15:39.414 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:39.410+0000 7f1ed0a4b700 1 -- start start 2023-07-14T19:15:39.414 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:39.410+0000 7f1ed0a4b700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1ecc1006e0 0x7f1ecc195900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:15:39.414 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:39.410+0000 7f1ed0a4b700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1ecc103000 0x7f1ecc195e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:15:39.414 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:39.410+0000 7f1ed0a4b700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1ecc105920 0x7f1ecc19ceb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:15:39.414 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:39.410+0000 7f1ed0a4b700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ecc19d480 con 0x7f1ecc1006e0 2023-07-14T19:15:39.414 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:39.410+0000 7f1ed0a4b700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ecc19d5c0 con 0x7f1ecc105920 2023-07-14T19:15:39.414 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:39.410+0000 7f1ed0a4b700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ecc19d750 con 0x7f1ecc103000 2023-07-14T19:15:39.415 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:39.411+0000 7f1eca59c700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1ecc1006e0 0x7f1ecc195900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:15:39.415 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:39.411+0000 7f1eca59c700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1ecc1006e0 0x7f1ecc195900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:60882/0 (socket says 172.21.15.40:60882) 2023-07-14T19:15:39.415 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:39.411+0000 7f1eca59c700 1 -- 172.21.15.40:0/2404836294 learned_addr learned my addr 172.21.15.40:0/2404836294 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-07-14T19:15:39.415 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:39.411+0000 7f1ecad9d700 1 --2- 172.21.15.40:0/2404836294 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1ecc105920 0x7f1ecc19ceb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:15:39.415 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:39.411+0000 7f1ec9d9b700 1 --2- 172.21.15.40:0/2404836294 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1ecc103000 0x7f1ecc195e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:15:39.415 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:39.411+0000 7f1eca59c700 1 -- 172.21.15.40:0/2404836294 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1ecc103000 msgr2=0x7f1ecc195e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:15:39.415 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:39.411+0000 7f1eca59c700 1 --2- 172.21.15.40:0/2404836294 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1ecc103000 0x7f1ecc195e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:15:39.416 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:39.411+0000 7f1eca59c700 1 -- 172.21.15.40:0/2404836294 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1ecc105920 msgr2=0x7f1ecc19ceb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:15:39.416 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:39.411+0000 7f1eca59c700 1 --2- 172.21.15.40:0/2404836294 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1ecc105920 0x7f1ecc19ceb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:15:39.416 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:39.411+0000 7f1eca59c700 1 -- 172.21.15.40:0/2404836294 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ec00095f0 con 0x7f1ecc1006e0 2023-07-14T19:15:39.416 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:39.411+0000 7f1eca59c700 1 --2- 172.21.15.40:0/2404836294 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1ecc1006e0 0x7f1ecc195900 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f1ebc00d950 tx=0x7f1ebc00dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:15:39.416 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:39.411+0000 7f1ebb7fe700 1 -- 172.21.15.40:0/2404836294 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ebc00f890 con 0x7f1ecc1006e0 2023-07-14T19:15:39.416 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:39.411+0000 7f1ed0a4b700 1 -- 172.21.15.40:0/2404836294 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ecc19da30 con 0x7f1ecc1006e0 2023-07-14T19:15:39.416 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:39.411+0000 7f1ebb7fe700 1 -- 172.21.15.40:0/2404836294 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f1ebc010460 con 0x7f1ecc1006e0 2023-07-14T19:15:39.417 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:39.411+0000 7f1ebb7fe700 1 -- 172.21.15.40:0/2404836294 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ebc00e610 con 0x7f1ecc1006e0 2023-07-14T19:15:39.417 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:39.411+0000 7f1ed0a4b700 1 -- 172.21.15.40:0/2404836294 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ecc19df40 con 0x7f1ecc1006e0 2023-07-14T19:15:39.417 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:39.412+0000 7f1ed0a4b700 1 -- 172.21.15.40:0/2404836294 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1ecc0fbc70 con 0x7f1ecc1006e0 2023-07-14T19:15:39.418 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:39.413+0000 7f1ebb7fe700 1 -- 172.21.15.40:0/2404836294 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f1ebc0108d0 con 0x7f1ecc1006e0 2023-07-14T19:15:39.418 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:39.414+0000 7f1ebb7fe700 1 --2- 172.21.15.40:0/2404836294 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f1eb40a11a0 0x7f1eb40a3650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:15:39.418 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:39.414+0000 7f1ec9d9b700 1 --2- 172.21.15.40:0/2404836294 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f1eb40a11a0 0x7f1eb40a3650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:15:39.418 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:39.414+0000 7f1ebb7fe700 1 -- 172.21.15.40:0/2404836294 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f1ebc0cc2d0 con 0x7f1ecc1006e0 2023-07-14T19:15:39.418 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:39.415+0000 7f1ec9d9b700 1 --2- 172.21.15.40:0/2404836294 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f1eb40a11a0 0x7f1eb40a3650 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f1ec000b2e0 tx=0x7f1ec0005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:15:39.419 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:39.418+0000 7f1ebb7fe700 1 -- 172.21.15.40:0/2404836294 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f1ebc095e20 con 0x7f1ecc1006e0 2023-07-14T19:15:39.472 INFO:journalctl@ceph.mds.f.smithi191.stdout:Jul 14 19:15:39 smithi191 bash[123847]: 496a0d1d6338156b36bfe54bc7df3b3101402b7056e9c65e866da1392138f87f 2023-07-14T19:15:39.472 INFO:journalctl@ceph.mds.f.smithi191.stdout:Jul 14 19:15:39 smithi191 ceph-mds[123887]: -- 172.21.15.191:0/4077064436 <== mon.1 v2:172.21.15.191:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55be464c91e0 con 0x55be4649f000 2023-07-14T19:15:39.472 INFO:journalctl@ceph.mds.f.smithi191.stdout:Jul 14 19:15:39 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-mds-f[123863]: starting mds.f at 2023-07-14T19:15:39.472 INFO:journalctl@ceph.mds.f.smithi191.stdout:Jul 14 19:15:39 smithi191 systemd[1]: Started Ceph mds.f for adc52f04-2279-11ee-9b33-001a4aab830c. 2023-07-14T19:15:39.661 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:39.661+0000 7f1ed0a4b700 1 -- 172.21.15.40:0/2404836294 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1ecc02c8e0 con 0x7f1ecc1006e0 2023-07-14T19:15:39.662 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:39.661+0000 7f1ebb7fe700 1 -- 172.21.15.40:0/2404836294 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 6 v6) v1 ==== 93+0+4285 (secure 0 0 0) 0x7f1ebc099440 con 0x7f1ecc1006e0 2023-07-14T19:15:39.663 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-07-14T19:15:39.663 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":6,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14523,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6835/501143718","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6834","nonce":501143718},{"type":"v1","addr":"172.21.15.40:6835","nonce":501143718}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14535,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6837/2634476117","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6836","nonce":2634476117},{"type":"v1","addr":"172.21.15.40:6837","nonce":2634476117}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":24433,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6833/31509862","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":31509862},{"type":"v1","addr":"172.21.15.191:6833","nonce":31509862}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24452,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.130:6833/3677452688","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6832","nonce":3677452688},{"type":"v1","addr":"172.21.15.130:6833","nonce":3677452688}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24482,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.130:6835/3088490808","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6834","nonce":3088490808},{"type":"v1","addr":"172.21.15.130:6835","nonce":3088490808}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6}],"filesystems":[]} 2023-07-14T19:15:39.664 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:39.663+0000 7f1ed0a4b700 1 -- 172.21.15.40:0/2404836294 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f1eb40a11a0 msgr2=0x7f1eb40a3650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:15:39.665 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:39.663+0000 7f1ed0a4b700 1 --2- 172.21.15.40:0/2404836294 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f1eb40a11a0 0x7f1eb40a3650 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f1ec000b2e0 tx=0x7f1ec0005fb0 comp rx=0 tx=0).stop 2023-07-14T19:15:39.665 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:39.663+0000 7f1ed0a4b700 1 -- 172.21.15.40:0/2404836294 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1ecc1006e0 msgr2=0x7f1ecc195900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:15:39.665 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:39.663+0000 7f1ed0a4b700 1 --2- 172.21.15.40:0/2404836294 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1ecc1006e0 0x7f1ecc195900 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f1ebc00d950 tx=0x7f1ebc00dc60 comp rx=0 tx=0).stop 2023-07-14T19:15:39.665 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:39.663+0000 7f1ed0a4b700 1 -- 172.21.15.40:0/2404836294 shutdown_connections 2023-07-14T19:15:39.666 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:39.663+0000 7f1ed0a4b700 1 --2- 172.21.15.40:0/2404836294 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f1eb40a11a0 0x7f1eb40a3650 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:15:39.666 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:39.663+0000 7f1ed0a4b700 1 --2- 172.21.15.40:0/2404836294 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1ecc105920 0x7f1ecc19ceb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:15:39.666 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:39.663+0000 7f1ed0a4b700 1 --2- 172.21.15.40:0/2404836294 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1ecc1006e0 0x7f1ecc195900 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:15:39.666 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:39.663+0000 7f1ed0a4b700 1 --2- 172.21.15.40:0/2404836294 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1ecc103000 0x7f1ecc195e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:15:39.667 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:39.663+0000 7f1ed0a4b700 1 -- 172.21.15.40:0/2404836294 >> 172.21.15.40:0/2404836294 conn(0x7f1ecc0fa100 msgr2=0x7f1ecc101340 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:15:39.667 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:39.663+0000 7f1ed0a4b700 1 -- 172.21.15.40:0/2404836294 shutdown_connections 2023-07-14T19:15:39.667 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:39.663+0000 7f1ed0a4b700 1 -- 172.21.15.40:0/2404836294 wait complete. 2023-07-14T19:15:39.667 INFO:teuthology.orchestra.run.smithi040.stderr:dumped fsmap epoch 6 2023-07-14T19:15:40.466 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:15:40 smithi191 ceph-mon[98311]: pgmap v241: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T19:15:40.466 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:15:40 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:15:40.467 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:15:40 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:15:40.467 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:15:40 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:15:40.467 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:15:40 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-14T19:15:40.468 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:15:40 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-14T19:15:40.468 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:15:40 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:15:40.468 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:15:40 smithi191 ceph-mon[98311]: Deploying daemon mds.c on smithi191 2023-07-14T19:15:40.468 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:15:40 smithi191 ceph-mon[98311]: from='client.? 172.21.15.40:0/2404836294' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T19:15:40.469 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:15:40 smithi191 ceph-mon[98311]: mds.? [v2:172.21.15.191:6834/3712674634,v1:172.21.15.191:6835/3712674634] up:boot 2023-07-14T19:15:40.469 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:15:40 smithi191 ceph-mon[98311]: fsmap 6 up:standby 2023-07-14T19:15:40.469 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:15:40 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2023-07-14T19:15:40.471 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:15:40 smithi040 ceph-mon[96518]: pgmap v241: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T19:15:40.471 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:15:40 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:15:40.471 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:15:40 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:15:40.472 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:15:40 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:15:40.472 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:15:40 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-14T19:15:40.472 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:15:40 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-14T19:15:40.473 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:15:40 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:15:40.473 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:15:40 smithi040 ceph-mon[96518]: Deploying daemon mds.c on smithi191 2023-07-14T19:15:40.474 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:15:40 smithi040 ceph-mon[96518]: from='client.? 172.21.15.40:0/2404836294' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T19:15:40.474 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:15:40 smithi040 ceph-mon[96518]: mds.? [v2:172.21.15.191:6834/3712674634,v1:172.21.15.191:6835/3712674634] up:boot 2023-07-14T19:15:40.474 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:15:40 smithi040 ceph-mon[96518]: fsmap 6 up:standby 2023-07-14T19:15:40.475 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:15:40 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2023-07-14T19:15:40.492 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:15:40 smithi130 ceph-mon[97832]: pgmap v241: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T19:15:40.493 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:15:40 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:15:40.493 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:15:40 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:15:40.493 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:15:40 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:15:40.493 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:15:40 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-14T19:15:40.494 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:15:40 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-14T19:15:40.494 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:15:40 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:15:40.494 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:15:40 smithi130 ceph-mon[97832]: Deploying daemon mds.c on smithi191 2023-07-14T19:15:40.494 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:15:40 smithi130 ceph-mon[97832]: from='client.? 172.21.15.40:0/2404836294' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T19:15:40.494 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:15:40 smithi130 ceph-mon[97832]: mds.? [v2:172.21.15.191:6834/3712674634,v1:172.21.15.191:6835/3712674634] up:boot 2023-07-14T19:15:40.494 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:15:40 smithi130 ceph-mon[97832]: fsmap 6 up:standby 2023-07-14T19:15:40.495 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:15:40 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2023-07-14T19:15:40.722 INFO:journalctl@ceph.mds.c.smithi191.stdout:Jul 14 19:15:40 smithi191 systemd[1]: Starting Ceph mds.c for adc52f04-2279-11ee-9b33-001a4aab830c... 2023-07-14T19:15:41.272 INFO:journalctl@ceph.mds.c.smithi191.stdout:Jul 14 19:15:41 smithi191 podman[124188]: 2023-07-14T19:15:41.677 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-07-14T19:15:41.722 INFO:journalctl@ceph.mds.c.smithi191.stdout:Jul 14 19:15:41 smithi191 ceph-mds[124228]: -- 172.21.15.191:0/3482912125 <== mon.1 v2:172.21.15.191:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55d7687571e0 con 0x55d76872d000 2023-07-14T19:15:41.722 INFO:journalctl@ceph.mds.c.smithi191.stdout:Jul 14 19:15:41 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-mds-c[124204]: starting mds.c at 2023-07-14T19:15:41.722 INFO:journalctl@ceph.mds.c.smithi191.stdout:Jul 14 19:15:41 smithi191 bash[124188]: f77b1956565c19dff5b2f7f549fa4312b79eb1d75c5673fbada20ed12cf36bb2 2023-07-14T19:15:41.722 INFO:journalctl@ceph.mds.c.smithi191.stdout:Jul 14 19:15:41 smithi191 systemd[1]: Started Ceph mds.c for adc52f04-2279-11ee-9b33-001a4aab830c. 2023-07-14T19:15:41.817 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:41.816+0000 7f7ca03cd700 1 -- 172.21.15.40:0/1102758571 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7c98100680 msgr2=0x7f7c98106f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:15:41.817 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:41.816+0000 7f7ca03cd700 1 --2- 172.21.15.40:0/1102758571 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7c98100680 0x7f7c98106f40 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f7c9000b0a0 tx=0x7f7c9000b3b0 comp rx=0 tx=0).stop 2023-07-14T19:15:41.818 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:41.816+0000 7f7ca03cd700 1 -- 172.21.15.40:0/1102758571 shutdown_connections 2023-07-14T19:15:41.818 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:41.816+0000 7f7ca03cd700 1 --2- 172.21.15.40:0/1102758571 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7c980feb00 0x7f7c980fef10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:15:41.818 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:41.816+0000 7f7ca03cd700 1 --2- 172.21.15.40:0/1102758571 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7c98100680 0x7f7c98106f40 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:15:41.818 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:41.816+0000 7f7ca03cd700 1 --2- 172.21.15.40:0/1102758571 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7c980ffcf0 0x7f7c98100140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:15:41.818 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:41.816+0000 7f7ca03cd700 1 -- 172.21.15.40:0/1102758571 >> 172.21.15.40:0/1102758571 conn(0x7f7c980fa100 msgr2=0x7f7c980fc550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:15:41.818 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:41.816+0000 7f7ca03cd700 1 -- 172.21.15.40:0/1102758571 shutdown_connections 2023-07-14T19:15:41.819 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:41.816+0000 7f7ca03cd700 1 -- 172.21.15.40:0/1102758571 wait complete. 2023-07-14T19:15:41.821 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:41.817+0000 7f7ca03cd700 1 Processor -- start 2023-07-14T19:15:41.821 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:41.817+0000 7f7ca03cd700 1 -- start start 2023-07-14T19:15:41.821 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:41.817+0000 7f7ca03cd700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7c980feb00 0x7f7c981948b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:15:41.821 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:41.817+0000 7f7ca03cd700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7c980ffcf0 0x7f7c98194df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:15:41.821 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:41.817+0000 7f7ca03cd700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7c98100680 0x7f7c9819be60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:15:41.822 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:41.817+0000 7f7ca03cd700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c9819c430 con 0x7f7c980ffcf0 2023-07-14T19:15:41.822 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:41.817+0000 7f7ca03cd700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c9819c570 con 0x7f7c98100680 2023-07-14T19:15:41.822 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:41.817+0000 7f7ca03cd700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c9819c700 con 0x7f7c980feb00 2023-07-14T19:15:41.822 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:41.817+0000 7f7c9d968700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7c980ffcf0 0x7f7c98194df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:15:41.822 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:41.817+0000 7f7c9d968700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7c980ffcf0 0x7f7c98194df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:60898/0 (socket says 172.21.15.40:60898) 2023-07-14T19:15:41.822 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:41.817+0000 7f7c9d968700 1 -- 172.21.15.40:0/2264633554 learned_addr learned my addr 172.21.15.40:0/2264633554 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-07-14T19:15:41.823 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:41.817+0000 7f7c9e96a700 1 --2- 172.21.15.40:0/2264633554 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7c98100680 0x7f7c9819be60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:15:41.823 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:41.818+0000 7f7c9e169700 1 --2- 172.21.15.40:0/2264633554 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7c980feb00 0x7f7c981948b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:15:41.823 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:41.818+0000 7f7c9d968700 1 -- 172.21.15.40:0/2264633554 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7c980feb00 msgr2=0x7f7c981948b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:15:41.823 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:41.818+0000 7f7c9d968700 1 --2- 172.21.15.40:0/2264633554 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7c980feb00 0x7f7c981948b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:15:41.823 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:41.818+0000 7f7c9d968700 1 -- 172.21.15.40:0/2264633554 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7c98100680 msgr2=0x7f7c9819be60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:15:41.824 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:41.818+0000 7f7c9d968700 1 --2- 172.21.15.40:0/2264633554 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7c98100680 0x7f7c9819be60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:15:41.824 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:41.818+0000 7f7c9d968700 1 -- 172.21.15.40:0/2264633554 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c90009d00 con 0x7f7c980ffcf0 2023-07-14T19:15:41.824 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:41.818+0000 7f7c9e96a700 1 --2- 172.21.15.40:0/2264633554 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7c98100680 0x7f7c9819be60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T19:15:41.824 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:41.818+0000 7f7c9e169700 1 --2- 172.21.15.40:0/2264633554 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7c980feb00 0x7f7c981948b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-14T19:15:41.824 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:41.818+0000 7f7c9d968700 1 --2- 172.21.15.40:0/2264633554 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7c980ffcf0 0x7f7c98194df0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f7c8800e9a0 tx=0x7f7c8800ed60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:15:41.824 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:41.818+0000 7f7c8f7fe700 1 -- 172.21.15.40:0/2264633554 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c88009790 con 0x7f7c980ffcf0 2023-07-14T19:15:41.825 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:41.818+0000 7f7c8f7fe700 1 -- 172.21.15.40:0/2264633554 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f7c8800cbd0 con 0x7f7c980ffcf0 2023-07-14T19:15:41.825 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:41.818+0000 7f7ca03cd700 1 -- 172.21.15.40:0/2264633554 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7c9819c9e0 con 0x7f7c980ffcf0 2023-07-14T19:15:41.825 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:41.818+0000 7f7c8f7fe700 1 -- 172.21.15.40:0/2264633554 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c88018680 con 0x7f7c980ffcf0 2023-07-14T19:15:41.825 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:41.818+0000 7f7ca03cd700 1 -- 172.21.15.40:0/2264633554 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7c9819cf40 con 0x7f7c980ffcf0 2023-07-14T19:15:41.825 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:41.819+0000 7f7ca03cd700 1 -- 172.21.15.40:0/2264633554 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7c9810a2e0 con 0x7f7c980ffcf0 2023-07-14T19:15:41.825 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:41.820+0000 7f7c8f7fe700 1 -- 172.21.15.40:0/2264633554 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f7c8800cd40 con 0x7f7c980ffcf0 2023-07-14T19:15:41.826 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:41.821+0000 7f7c8f7fe700 1 --2- 172.21.15.40:0/2264633554 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f7c840a1220 0x7f7c840a36d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:15:41.826 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:41.821+0000 7f7c9e169700 1 --2- 172.21.15.40:0/2264633554 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f7c840a1220 0x7f7c840a36d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:15:41.826 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:41.821+0000 7f7c8f7fe700 1 -- 172.21.15.40:0/2264633554 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f7c88014070 con 0x7f7c980ffcf0 2023-07-14T19:15:41.827 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:41.821+0000 7f7c9e169700 1 --2- 172.21.15.40:0/2264633554 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f7c840a1220 0x7f7c840a36d0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f7c94005fd0 tx=0x7f7c94005f00 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:15:41.827 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:41.824+0000 7f7c8f7fe700 1 -- 172.21.15.40:0/2264633554 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f7c880967e0 con 0x7f7c980ffcf0 2023-07-14T19:15:42.070 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:42.069+0000 7f7ca03cd700 1 -- 172.21.15.40:0/2264633554 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7c9810a4d0 con 0x7f7c980ffcf0 2023-07-14T19:15:42.070 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:42.069+0000 7f7c8f7fe700 1 -- 172.21.15.40:0/2264633554 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) v1 ==== 93+0+5045 (secure 0 0 0) 0x7f7c88099e00 con 0x7f7c980ffcf0 2023-07-14T19:15:42.071 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-07-14T19:15:42.071 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":7,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14523,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6835/501143718","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6834","nonce":501143718},{"type":"v1","addr":"172.21.15.40:6835","nonce":501143718}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14535,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6837/2634476117","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6836","nonce":2634476117},{"type":"v1","addr":"172.21.15.40:6837","nonce":2634476117}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":24433,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6833/31509862","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":31509862},{"type":"v1","addr":"172.21.15.191:6833","nonce":31509862}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24452,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.130:6833/3677452688","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6832","nonce":3677452688},{"type":"v1","addr":"172.21.15.130:6833","nonce":3677452688}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24478,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6835/3712674634","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":3712674634},{"type":"v1","addr":"172.21.15.191:6835","nonce":3712674634}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":7},{"gid":24482,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.130:6835/3088490808","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6834","nonce":3088490808},{"type":"v1","addr":"172.21.15.130:6835","nonce":3088490808}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6}],"filesystems":[]} 2023-07-14T19:15:42.072 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:42.072+0000 7f7ca03cd700 1 -- 172.21.15.40:0/2264633554 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f7c840a1220 msgr2=0x7f7c840a36d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:15:42.072 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:42.072+0000 7f7ca03cd700 1 --2- 172.21.15.40:0/2264633554 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f7c840a1220 0x7f7c840a36d0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f7c94005fd0 tx=0x7f7c94005f00 comp rx=0 tx=0).stop 2023-07-14T19:15:42.073 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:42.072+0000 7f7ca03cd700 1 -- 172.21.15.40:0/2264633554 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7c980ffcf0 msgr2=0x7f7c98194df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:15:42.073 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:42.072+0000 7f7ca03cd700 1 --2- 172.21.15.40:0/2264633554 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7c980ffcf0 0x7f7c98194df0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f7c8800e9a0 tx=0x7f7c8800ed60 comp rx=0 tx=0).stop 2023-07-14T19:15:42.073 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:42.072+0000 7f7ca03cd700 1 -- 172.21.15.40:0/2264633554 shutdown_connections 2023-07-14T19:15:42.073 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:42.072+0000 7f7ca03cd700 1 --2- 172.21.15.40:0/2264633554 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f7c840a1220 0x7f7c840a36d0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:15:42.073 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:42.072+0000 7f7ca03cd700 1 --2- 172.21.15.40:0/2264633554 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7c980feb00 0x7f7c981948b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:15:42.073 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:42.072+0000 7f7ca03cd700 1 --2- 172.21.15.40:0/2264633554 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7c98100680 0x7f7c9819be60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:15:42.073 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:42.072+0000 7f7ca03cd700 1 --2- 172.21.15.40:0/2264633554 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7c980ffcf0 0x7f7c98194df0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:15:42.074 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:42.072+0000 7f7ca03cd700 1 -- 172.21.15.40:0/2264633554 >> 172.21.15.40:0/2264633554 conn(0x7f7c980fa100 msgr2=0x7f7c980fc2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:15:42.074 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:42.072+0000 7f7ca03cd700 1 -- 172.21.15.40:0/2264633554 shutdown_connections 2023-07-14T19:15:42.074 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:42.072+0000 7f7ca03cd700 1 -- 172.21.15.40:0/2264633554 wait complete. 2023-07-14T19:15:42.075 INFO:teuthology.orchestra.run.smithi040.stderr:dumped fsmap epoch 7 2023-07-14T19:15:42.715 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:15:42 smithi130 ceph-mon[97832]: pgmap v242: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T19:15:42.716 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:15:42 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:15:42.716 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:15:42 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:15:42.716 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:15:42 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:15:42.716 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:15:42 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-14T19:15:42.717 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:15:42 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-14T19:15:42.717 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:15:42 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:15:42.717 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:15:42 smithi130 ceph-mon[97832]: Deploying daemon mds.h on smithi130 2023-07-14T19:15:42.717 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:15:42 smithi130 ceph-mon[97832]: from='client.? 172.21.15.40:0/2264633554' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T19:15:42.717 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:15:42 smithi130 ceph-mon[97832]: mds.? [v2:172.21.15.191:6836/2406892975,v1:172.21.15.191:6837/2406892975] up:boot 2023-07-14T19:15:42.717 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:15:42 smithi130 ceph-mon[97832]: fsmap 7 up:standby 2023-07-14T19:15:42.718 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:15:42 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2023-07-14T19:15:42.721 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:15:42 smithi040 ceph-mon[96518]: pgmap v242: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T19:15:42.721 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:15:42 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:15:42.721 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:15:42 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:15:42.721 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:15:42 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:15:42.722 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:15:42 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-14T19:15:42.722 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:15:42 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-14T19:15:42.722 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:15:42 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:15:42.723 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:15:42 smithi040 ceph-mon[96518]: Deploying daemon mds.h on smithi130 2023-07-14T19:15:42.723 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:15:42 smithi040 ceph-mon[96518]: from='client.? 172.21.15.40:0/2264633554' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T19:15:42.723 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:15:42 smithi040 ceph-mon[96518]: mds.? [v2:172.21.15.191:6836/2406892975,v1:172.21.15.191:6837/2406892975] up:boot 2023-07-14T19:15:42.724 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:15:42 smithi040 ceph-mon[96518]: fsmap 7 up:standby 2023-07-14T19:15:42.724 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:15:42 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2023-07-14T19:15:42.725 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:15:42 smithi191 ceph-mon[98311]: pgmap v242: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T19:15:42.725 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:15:42 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:15:42.725 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:15:42 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:15:42.726 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:15:42 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:15:42.726 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:15:42 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-14T19:15:42.726 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:15:42 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-14T19:15:42.726 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:15:42 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:15:42.727 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:15:42 smithi191 ceph-mon[98311]: Deploying daemon mds.h on smithi130 2023-07-14T19:15:42.727 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:15:42 smithi191 ceph-mon[98311]: from='client.? 172.21.15.40:0/2264633554' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T19:15:42.727 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:15:42 smithi191 ceph-mon[98311]: mds.? [v2:172.21.15.191:6836/2406892975,v1:172.21.15.191:6837/2406892975] up:boot 2023-07-14T19:15:42.727 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:15:42 smithi191 ceph-mon[98311]: fsmap 7 up:standby 2023-07-14T19:15:42.728 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:15:42 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2023-07-14T19:15:42.992 INFO:journalctl@ceph.mds.h.smithi130.stdout:Jul 14 19:15:42 smithi130 systemd[1]: Starting Ceph mds.h for adc52f04-2279-11ee-9b33-001a4aab830c... 2023-07-14T19:15:43.521 INFO:journalctl@ceph.mds.h.smithi130.stdout:Jul 14 19:15:43 smithi130 podman[122488]: 2023-07-14T19:15:43.992 INFO:journalctl@ceph.mds.h.smithi130.stdout:Jul 14 19:15:43 smithi130 ceph-mds[122529]: -- 172.21.15.130:0/1132244689 <== mon.2 v2:172.21.15.130:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x557aaffd71e0 con 0x557aaffad000 2023-07-14T19:15:43.992 INFO:journalctl@ceph.mds.h.smithi130.stdout:Jul 14 19:15:43 smithi130 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-mds-h[122505]: starting mds.h at 2023-07-14T19:15:43.992 INFO:journalctl@ceph.mds.h.smithi130.stdout:Jul 14 19:15:43 smithi130 bash[122488]: 798bacc371932127adaaa2a75f8393b1aa365c9616d8260e096839883b2e027e 2023-07-14T19:15:43.993 INFO:journalctl@ceph.mds.h.smithi130.stdout:Jul 14 19:15:43 smithi130 systemd[1]: Started Ceph mds.h for adc52f04-2279-11ee-9b33-001a4aab830c. 2023-07-14T19:15:44.085 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-07-14T19:15:44.226 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:44.225+0000 7f7594c69700 1 -- 172.21.15.40:0/2850340761 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7590100680 msgr2=0x7f7590102a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:15:44.226 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:44.225+0000 7f7594c69700 1 --2- 172.21.15.40:0/2850340761 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7590100680 0x7f7590102a60 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f7580009990 tx=0x7f7580009ca0 comp rx=0 tx=0).stop 2023-07-14T19:15:44.226 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:44.225+0000 7f7594c69700 1 -- 172.21.15.40:0/2850340761 shutdown_connections 2023-07-14T19:15:44.226 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:44.225+0000 7f7594c69700 1 --2- 172.21.15.40:0/2850340761 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f75901058c0 0x7f7590107d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:15:44.227 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:44.225+0000 7f7594c69700 1 --2- 172.21.15.40:0/2850340761 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7590100680 0x7f7590102a60 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:15:44.227 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:44.225+0000 7f7594c69700 1 --2- 172.21.15.40:0/2850340761 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7590102fa0 0x7f7590105380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:15:44.227 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:44.225+0000 7f7594c69700 1 -- 172.21.15.40:0/2850340761 >> 172.21.15.40:0/2850340761 conn(0x7f75900fa100 msgr2=0x7f75900fc550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:15:44.227 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:44.225+0000 7f7594c69700 1 -- 172.21.15.40:0/2850340761 shutdown_connections 2023-07-14T19:15:44.227 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:44.225+0000 7f7594c69700 1 -- 172.21.15.40:0/2850340761 wait complete. 2023-07-14T19:15:44.229 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:44.225+0000 7f7594c69700 1 Processor -- start 2023-07-14T19:15:44.229 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:44.226+0000 7f7594c69700 1 -- start start 2023-07-14T19:15:44.229 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:44.226+0000 7f7594c69700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7590100680 0x7f75900ffff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:15:44.229 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:44.226+0000 7f7594c69700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7590102fa0 0x7f75900fe630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:15:44.229 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:44.226+0000 7f7594c69700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f75901058c0 0x7f75900feb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:15:44.230 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:44.226+0000 7f7594c69700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75900ff310 con 0x7f75901058c0 2023-07-14T19:15:44.230 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:44.226+0000 7f7594c69700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75900ff480 con 0x7f7590100680 2023-07-14T19:15:44.230 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:44.226+0000 7f7594c69700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f759019f8e0 con 0x7f7590102fa0 2023-07-14T19:15:44.230 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:44.226+0000 7f758ed9d700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f75901058c0 0x7f75900feb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:15:44.230 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:44.226+0000 7f758dd9b700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7590102fa0 0x7f75900fe630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:15:44.230 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:44.226+0000 7f758ed9d700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f75901058c0 0x7f75900feb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:60914/0 (socket says 172.21.15.40:60914) 2023-07-14T19:15:44.230 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:44.226+0000 7f758e59c700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7590100680 0x7f75900ffff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:15:44.231 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:44.226+0000 7f758ed9d700 1 -- 172.21.15.40:0/1748535386 learned_addr learned my addr 172.21.15.40:0/1748535386 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-07-14T19:15:44.231 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:44.226+0000 7f758ed9d700 1 -- 172.21.15.40:0/1748535386 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7590102fa0 msgr2=0x7f75900fe630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:15:44.231 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:44.226+0000 7f758ed9d700 1 --2- 172.21.15.40:0/1748535386 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7590102fa0 0x7f75900fe630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:15:44.231 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:44.226+0000 7f758ed9d700 1 -- 172.21.15.40:0/1748535386 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7590100680 msgr2=0x7f75900ffff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:15:44.231 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:44.226+0000 7f758ed9d700 1 --2- 172.21.15.40:0/1748535386 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7590100680 0x7f75900ffff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:15:44.231 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:44.226+0000 7f758ed9d700 1 -- 172.21.15.40:0/1748535386 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75800095f0 con 0x7f75901058c0 2023-07-14T19:15:44.231 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:44.226+0000 7f758ed9d700 1 --2- 172.21.15.40:0/1748535386 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f75901058c0 0x7f75900feb70 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f7588007ee0 tx=0x7f758800d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:15:44.231 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:44.227+0000 7f757f7fe700 1 -- 172.21.15.40:0/1748535386 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7588017070 con 0x7f75901058c0 2023-07-14T19:15:44.232 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:44.227+0000 7f757f7fe700 1 -- 172.21.15.40:0/1748535386 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f758800f040 con 0x7f75901058c0 2023-07-14T19:15:44.232 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:44.227+0000 7f757f7fe700 1 -- 172.21.15.40:0/1748535386 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7588005580 con 0x7f75901058c0 2023-07-14T19:15:44.232 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:44.227+0000 7f7594c69700 1 -- 172.21.15.40:0/1748535386 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f759019fbc0 con 0x7f75901058c0 2023-07-14T19:15:44.232 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:44.227+0000 7f7594c69700 1 -- 172.21.15.40:0/1748535386 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f75901a0080 con 0x7f75901058c0 2023-07-14T19:15:44.232 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:44.228+0000 7f7594c69700 1 -- 172.21.15.40:0/1748535386 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f75900fbc70 con 0x7f75901058c0 2023-07-14T19:15:44.233 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:44.229+0000 7f757f7fe700 1 -- 172.21.15.40:0/1748535386 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f758800db20 con 0x7f75901058c0 2023-07-14T19:15:44.233 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:44.229+0000 7f757f7fe700 1 --2- 172.21.15.40:0/1748535386 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f75780a1220 0x7f75780a36d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:15:44.233 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:44.230+0000 7f758e59c700 1 --2- 172.21.15.40:0/1748535386 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f75780a1220 0x7f75780a36d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:15:44.233 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:44.230+0000 7f757f7fe700 1 -- 172.21.15.40:0/1748535386 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f75880cc170 con 0x7f75901058c0 2023-07-14T19:15:44.233 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:44.230+0000 7f758e59c700 1 --2- 172.21.15.40:0/1748535386 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f75780a1220 0x7f75780a36d0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f758000b5c0 tx=0x7f7580005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:15:44.234 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:44.233+0000 7f757f7fe700 1 -- 172.21.15.40:0/1748535386 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f7588095d60 con 0x7f75901058c0 2023-07-14T19:15:44.482 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:44.482+0000 7f7594c69700 1 -- 172.21.15.40:0/1748535386 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f759002c8e0 con 0x7f75901058c0 2023-07-14T19:15:44.483 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:44.482+0000 7f757f7fe700 1 -- 172.21.15.40:0/1748535386 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) v1 ==== 93+0+6565 (secure 0 0 0) 0x7f7588099380 con 0x7f75901058c0 2023-07-14T19:15:44.483 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-07-14T19:15:44.484 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":9,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14523,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6835/501143718","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6834","nonce":501143718},{"type":"v1","addr":"172.21.15.40:6835","nonce":501143718}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14535,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6837/2634476117","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6836","nonce":2634476117},{"type":"v1","addr":"172.21.15.40:6837","nonce":2634476117}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":24433,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6833/31509862","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":31509862},{"type":"v1","addr":"172.21.15.191:6833","nonce":31509862}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24452,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.130:6833/3677452688","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6832","nonce":3677452688},{"type":"v1","addr":"172.21.15.130:6833","nonce":3677452688}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24478,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6835/3712674634","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":3712674634},{"type":"v1","addr":"172.21.15.191:6835","nonce":3712674634}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":7},{"gid":24482,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.130:6835/3088490808","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6834","nonce":3088490808},{"type":"v1","addr":"172.21.15.130:6835","nonce":3088490808}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6},{"gid":24490,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6837/2406892975","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6836","nonce":2406892975},{"type":"v1","addr":"172.21.15.191:6837","nonce":2406892975}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":8},{"gid":24506,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.130:6837/2604019369","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6836","nonce":2604019369},{"type":"v1","addr":"172.21.15.130:6837","nonce":2604019369}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":9}],"filesystems":[]} 2023-07-14T19:15:44.485 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:44.484+0000 7f7594c69700 1 -- 172.21.15.40:0/1748535386 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f75780a1220 msgr2=0x7f75780a36d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:15:44.486 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:44.484+0000 7f7594c69700 1 --2- 172.21.15.40:0/1748535386 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f75780a1220 0x7f75780a36d0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f758000b5c0 tx=0x7f7580005fb0 comp rx=0 tx=0).stop 2023-07-14T19:15:44.486 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:44.484+0000 7f7594c69700 1 -- 172.21.15.40:0/1748535386 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f75901058c0 msgr2=0x7f75900feb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:15:44.486 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:44.484+0000 7f7594c69700 1 --2- 172.21.15.40:0/1748535386 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f75901058c0 0x7f75900feb70 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f7588007ee0 tx=0x7f758800d3b0 comp rx=0 tx=0).stop 2023-07-14T19:15:44.486 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:44.485+0000 7f7594c69700 1 -- 172.21.15.40:0/1748535386 shutdown_connections 2023-07-14T19:15:44.486 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:44.485+0000 7f7594c69700 1 --2- 172.21.15.40:0/1748535386 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f75780a1220 0x7f75780a36d0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:15:44.486 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:44.485+0000 7f7594c69700 1 --2- 172.21.15.40:0/1748535386 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f75901058c0 0x7f75900feb70 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:15:44.486 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:44.485+0000 7f7594c69700 1 --2- 172.21.15.40:0/1748535386 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7590100680 0x7f75900ffff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:15:44.487 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:44.485+0000 7f7594c69700 1 --2- 172.21.15.40:0/1748535386 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7590102fa0 0x7f75900fe630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:15:44.487 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:44.485+0000 7f7594c69700 1 -- 172.21.15.40:0/1748535386 >> 172.21.15.40:0/1748535386 conn(0x7f75900fa100 msgr2=0x7f75901012e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:15:44.487 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:44.485+0000 7f7594c69700 1 -- 172.21.15.40:0/1748535386 shutdown_connections 2023-07-14T19:15:44.487 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:44.485+0000 7f7594c69700 1 -- 172.21.15.40:0/1748535386 wait complete. 2023-07-14T19:15:44.487 INFO:teuthology.orchestra.run.smithi040.stderr:dumped fsmap epoch 9 2023-07-14T19:15:44.905 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:15:44 smithi191 ceph-mon[98311]: pgmap v243: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T19:15:44.906 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:15:44 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:15:44.906 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:15:44 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:15:44.906 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:15:44 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:15:44.906 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:15:44 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-14T19:15:44.906 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:15:44 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-14T19:15:44.907 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:15:44 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:15:44.907 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:15:44 smithi191 ceph-mon[98311]: Deploying daemon mds.i on smithi191 2023-07-14T19:15:44.907 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:15:44 smithi191 ceph-mon[98311]: mds.? [v2:172.21.15.130:6836/2604019369,v1:172.21.15.130:6837/2604019369] up:boot 2023-07-14T19:15:44.907 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:15:44 smithi191 ceph-mon[98311]: fsmap 8 up:standby 2023-07-14T19:15:44.907 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:15:44 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2023-07-14T19:15:44.907 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:15:44 smithi191 ceph-mon[98311]: from='client.? 172.21.15.40:0/1748535386' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T19:15:44.971 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:15:44 smithi040 ceph-mon[96518]: pgmap v243: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T19:15:44.971 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:15:44 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:15:44.971 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:15:44 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:15:44.972 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:15:44 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:15:44.972 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:15:44 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-14T19:15:44.972 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:15:44 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-14T19:15:44.972 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:15:44 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:15:44.972 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:15:44 smithi040 ceph-mon[96518]: Deploying daemon mds.i on smithi191 2023-07-14T19:15:44.972 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:15:44 smithi040 ceph-mon[96518]: mds.? [v2:172.21.15.130:6836/2604019369,v1:172.21.15.130:6837/2604019369] up:boot 2023-07-14T19:15:44.972 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:15:44 smithi040 ceph-mon[96518]: fsmap 8 up:standby 2023-07-14T19:15:44.973 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:15:44 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2023-07-14T19:15:44.973 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:15:44 smithi040 ceph-mon[96518]: from='client.? 172.21.15.40:0/1748535386' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T19:15:44.992 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:15:44 smithi130 ceph-mon[97832]: pgmap v243: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T19:15:44.992 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:15:44 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:15:44.993 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:15:44 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:15:44.993 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:15:44 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:15:44.993 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:15:44 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-14T19:15:44.993 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:15:44 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-14T19:15:44.993 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:15:44 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:15:44.994 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:15:44 smithi130 ceph-mon[97832]: Deploying daemon mds.i on smithi191 2023-07-14T19:15:44.994 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:15:44 smithi130 ceph-mon[97832]: mds.? [v2:172.21.15.130:6836/2604019369,v1:172.21.15.130:6837/2604019369] up:boot 2023-07-14T19:15:44.994 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:15:44 smithi130 ceph-mon[97832]: fsmap 8 up:standby 2023-07-14T19:15:44.994 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:15:44 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2023-07-14T19:15:44.994 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:15:44 smithi130 ceph-mon[97832]: from='client.? 172.21.15.40:0/1748535386' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T19:15:45.519 INFO:journalctl@ceph.mds.i.smithi191.stdout:Jul 14 19:15:45 smithi191 systemd[1]: Starting Ceph mds.i for adc52f04-2279-11ee-9b33-001a4aab830c... 2023-07-14T19:15:45.863 INFO:journalctl@ceph.mds.i.smithi191.stdout:Jul 14 19:15:45 smithi191 podman[124528]: 2023-07-14T19:15:46.222 INFO:journalctl@ceph.mds.i.smithi191.stdout:Jul 14 19:15:45 smithi191 ceph-mds[124568]: -- 172.21.15.191:0/82251646 <== mon.1 v2:172.21.15.191:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55c547bb91e0 con 0x55c547b8e800 2023-07-14T19:15:46.222 INFO:journalctl@ceph.mds.i.smithi191.stdout:Jul 14 19:15:45 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-mds-i[124544]: starting mds.i at 2023-07-14T19:15:46.222 INFO:journalctl@ceph.mds.i.smithi191.stdout:Jul 14 19:15:45 smithi191 bash[124528]: 3bccc6cb169ae64d3dd1582ba713768ec73de227274e3cff96dd1a7bf1760c29 2023-07-14T19:15:46.222 INFO:journalctl@ceph.mds.i.smithi191.stdout:Jul 14 19:15:45 smithi191 systemd[1]: Started Ceph mds.i for adc52f04-2279-11ee-9b33-001a4aab830c. 2023-07-14T19:15:46.498 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-07-14T19:15:46.628 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:46.627+0000 7f8aa9dda700 1 -- 172.21.15.40:0/473379089 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8aa41017b0 msgr2=0x7f8aa4108070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:15:46.628 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:46.627+0000 7f8aa9dda700 1 --2- 172.21.15.40:0/473379089 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8aa41017b0 0x7f8aa4108070 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f8a9c00b0a0 tx=0x7f8a9c00b3b0 comp rx=0 tx=0).stop 2023-07-14T19:15:46.629 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:46.627+0000 7f8aa9dda700 1 -- 172.21.15.40:0/473379089 shutdown_connections 2023-07-14T19:15:46.629 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:46.627+0000 7f8aa9dda700 1 --2- 172.21.15.40:0/473379089 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8aa41017b0 0x7f8aa4108070 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:15:46.629 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:46.627+0000 7f8aa9dda700 1 --2- 172.21.15.40:0/473379089 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8aa40ffc30 0x7f8aa4100040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:15:46.629 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:46.627+0000 7f8aa9dda700 1 --2- 172.21.15.40:0/473379089 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8aa4100e20 0x7f8aa4101270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:15:46.629 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:46.627+0000 7f8aa9dda700 1 -- 172.21.15.40:0/473379089 >> 172.21.15.40:0/473379089 conn(0x7f8aa40fb210 msgr2=0x7f8aa40fd680 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:15:46.629 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:46.627+0000 7f8aa9dda700 1 -- 172.21.15.40:0/473379089 shutdown_connections 2023-07-14T19:15:46.629 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:46.627+0000 7f8aa9dda700 1 -- 172.21.15.40:0/473379089 wait complete. 2023-07-14T19:15:46.631 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:46.628+0000 7f8aa9dda700 1 Processor -- start 2023-07-14T19:15:46.631 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:46.628+0000 7f8aa9dda700 1 -- start start 2023-07-14T19:15:46.632 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:46.628+0000 7f8aa9dda700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8aa40ffc30 0x7f8aa4195a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:15:46.632 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:46.628+0000 7f8aa9dda700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8aa4100e20 0x7f8aa4195f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:15:46.632 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:46.628+0000 7f8aa9dda700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8aa41017b0 0x7f8aa419cfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:15:46.632 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:46.628+0000 7f8aa9dda700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8aa419d5b0 con 0x7f8aa41017b0 2023-07-14T19:15:46.632 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:46.628+0000 7f8aa9dda700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8aa419d6f0 con 0x7f8aa40ffc30 2023-07-14T19:15:46.632 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:46.628+0000 7f8aa9dda700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8aa419d880 con 0x7f8aa4100e20 2023-07-14T19:15:46.632 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:46.628+0000 7f8aa3fff700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8aa41017b0 0x7f8aa419cfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:15:46.633 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:46.628+0000 7f8aa3fff700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8aa41017b0 0x7f8aa419cfe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:60928/0 (socket says 172.21.15.40:60928) 2023-07-14T19:15:46.633 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:46.628+0000 7f8aa3fff700 1 -- 172.21.15.40:0/3190649182 learned_addr learned my addr 172.21.15.40:0/3190649182 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-07-14T19:15:46.633 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:46.629+0000 7f8aa2ffd700 1 --2- 172.21.15.40:0/3190649182 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8aa4100e20 0x7f8aa4195f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:15:46.633 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:46.629+0000 7f8aa37fe700 1 --2- 172.21.15.40:0/3190649182 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8aa40ffc30 0x7f8aa4195a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:15:46.633 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:46.629+0000 7f8aa3fff700 1 -- 172.21.15.40:0/3190649182 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8aa4100e20 msgr2=0x7f8aa4195f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:15:46.633 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:46.629+0000 7f8aa3fff700 1 --2- 172.21.15.40:0/3190649182 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8aa4100e20 0x7f8aa4195f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:15:46.633 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:46.629+0000 7f8aa3fff700 1 -- 172.21.15.40:0/3190649182 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8aa40ffc30 msgr2=0x7f8aa4195a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:15:46.633 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:46.629+0000 7f8aa3fff700 1 --2- 172.21.15.40:0/3190649182 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8aa40ffc30 0x7f8aa4195a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:15:46.634 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:46.629+0000 7f8aa3fff700 1 -- 172.21.15.40:0/3190649182 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a9c009d00 con 0x7f8aa41017b0 2023-07-14T19:15:46.634 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:46.629+0000 7f8aa3fff700 1 --2- 172.21.15.40:0/3190649182 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8aa41017b0 0x7f8aa419cfe0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f8a9c00d010 tx=0x7f8a9c004c80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:15:46.634 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:46.629+0000 7f8aa0ff9700 1 -- 172.21.15.40:0/3190649182 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a9c00e040 con 0x7f8aa41017b0 2023-07-14T19:15:46.634 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:46.629+0000 7f8aa9dda700 1 -- 172.21.15.40:0/3190649182 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8aa419db00 con 0x7f8aa41017b0 2023-07-14T19:15:46.634 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:46.629+0000 7f8aa0ff9700 1 -- 172.21.15.40:0/3190649182 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f8a9c004e30 con 0x7f8aa41017b0 2023-07-14T19:15:46.634 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:46.629+0000 7f8aa0ff9700 1 -- 172.21.15.40:0/3190649182 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a9c004020 con 0x7f8aa41017b0 2023-07-14T19:15:46.634 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:46.629+0000 7f8aa9dda700 1 -- 172.21.15.40:0/3190649182 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8aa419df80 con 0x7f8aa41017b0 2023-07-14T19:15:46.634 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:46.630+0000 7f8aa9dda700 1 -- 172.21.15.40:0/3190649182 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8aa41070d0 con 0x7f8aa41017b0 2023-07-14T19:15:46.635 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:46.631+0000 7f8aa0ff9700 1 -- 172.21.15.40:0/3190649182 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f8a9c01d690 con 0x7f8aa41017b0 2023-07-14T19:15:46.635 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:46.632+0000 7f8aa0ff9700 1 --2- 172.21.15.40:0/3190649182 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f8a8c0a11a0 0x7f8a8c0a3650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:15:46.636 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:46.632+0000 7f8aa0ff9700 1 -- 172.21.15.40:0/3190649182 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f8a9c0ce370 con 0x7f8aa41017b0 2023-07-14T19:15:46.636 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:46.632+0000 7f8aa37fe700 1 --2- 172.21.15.40:0/3190649182 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f8a8c0a11a0 0x7f8a8c0a3650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:15:46.636 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:46.632+0000 7f8aa37fe700 1 --2- 172.21.15.40:0/3190649182 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f8a8c0a11a0 0x7f8a8c0a3650 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f8aa4100c80 tx=0x7f8a94007400 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:15:46.636 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:46.636+0000 7f8aa0ff9700 1 -- 172.21.15.40:0/3190649182 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f8a9c097f50 con 0x7f8aa41017b0 2023-07-14T19:15:46.783 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:15:46 smithi040 ceph-mon[96518]: pgmap v244: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T19:15:46.783 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:15:46 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:15:46.783 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:15:46 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:15:46.783 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:15:46 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:15:46.783 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:15:46 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-14T19:15:46.783 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:15:46 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-14T19:15:46.784 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:15:46 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:15:46.900 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:46.899+0000 7f8aa9dda700 1 -- 172.21.15.40:0/3190649182 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8aa402c8e0 con 0x7f8aa41017b0 2023-07-14T19:15:46.901 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:46.900+0000 7f8aa0ff9700 1 -- 172.21.15.40:0/3190649182 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+7327 (secure 0 0 0) 0x7f8a9c09b570 con 0x7f8aa41017b0 2023-07-14T19:15:46.901 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-07-14T19:15:46.901 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":10,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14523,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6835/501143718","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6834","nonce":501143718},{"type":"v1","addr":"172.21.15.40:6835","nonce":501143718}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14535,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6837/2634476117","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6836","nonce":2634476117},{"type":"v1","addr":"172.21.15.40:6837","nonce":2634476117}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":24433,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6833/31509862","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":31509862},{"type":"v1","addr":"172.21.15.191:6833","nonce":31509862}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24452,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.130:6833/3677452688","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6832","nonce":3677452688},{"type":"v1","addr":"172.21.15.130:6833","nonce":3677452688}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24478,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6835/3712674634","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":3712674634},{"type":"v1","addr":"172.21.15.191:6835","nonce":3712674634}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":7},{"gid":24482,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.130:6835/3088490808","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6834","nonce":3088490808},{"type":"v1","addr":"172.21.15.130:6835","nonce":3088490808}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6},{"gid":24490,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6837/2406892975","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6836","nonce":2406892975},{"type":"v1","addr":"172.21.15.191:6837","nonce":2406892975}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":8},{"gid":24506,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.130:6837/2604019369","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6836","nonce":2604019369},{"type":"v1","addr":"172.21.15.130:6837","nonce":2604019369}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":9},{"gid":24508,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6839/1977507302","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":1977507302},{"type":"v1","addr":"172.21.15.191:6839","nonce":1977507302}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[]} 2023-07-14T19:15:46.904 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:46.903+0000 7f8aa9dda700 1 -- 172.21.15.40:0/3190649182 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f8a8c0a11a0 msgr2=0x7f8a8c0a3650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:15:46.905 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:46.903+0000 7f8aa9dda700 1 --2- 172.21.15.40:0/3190649182 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f8a8c0a11a0 0x7f8a8c0a3650 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f8aa4100c80 tx=0x7f8a94007400 comp rx=0 tx=0).stop 2023-07-14T19:15:46.905 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:46.903+0000 7f8aa9dda700 1 -- 172.21.15.40:0/3190649182 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8aa41017b0 msgr2=0x7f8aa419cfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:15:46.905 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:46.903+0000 7f8aa9dda700 1 --2- 172.21.15.40:0/3190649182 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8aa41017b0 0x7f8aa419cfe0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f8a9c00d010 tx=0x7f8a9c004c80 comp rx=0 tx=0).stop 2023-07-14T19:15:46.905 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:46.904+0000 7f8aa9dda700 1 -- 172.21.15.40:0/3190649182 shutdown_connections 2023-07-14T19:15:46.905 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:46.904+0000 7f8aa9dda700 1 --2- 172.21.15.40:0/3190649182 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f8a8c0a11a0 0x7f8a8c0a3650 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:15:46.905 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:46.904+0000 7f8aa9dda700 1 --2- 172.21.15.40:0/3190649182 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8aa41017b0 0x7f8aa419cfe0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:15:46.905 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:46.904+0000 7f8aa9dda700 1 --2- 172.21.15.40:0/3190649182 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8aa40ffc30 0x7f8aa4195a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:15:46.906 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:46.904+0000 7f8aa9dda700 1 --2- 172.21.15.40:0/3190649182 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8aa4100e20 0x7f8aa4195f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:15:46.906 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:46.904+0000 7f8aa9dda700 1 -- 172.21.15.40:0/3190649182 >> 172.21.15.40:0/3190649182 conn(0x7f8aa40fb210 msgr2=0x7f8aa4104210 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:15:46.906 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:46.904+0000 7f8aa9dda700 1 -- 172.21.15.40:0/3190649182 shutdown_connections 2023-07-14T19:15:46.906 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:46.904+0000 7f8aa9dda700 1 -- 172.21.15.40:0/3190649182 wait complete. 2023-07-14T19:15:46.906 INFO:teuthology.orchestra.run.smithi040.stderr:dumped fsmap epoch 10 2023-07-14T19:15:46.972 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:15:46 smithi191 ceph-mon[98311]: pgmap v244: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T19:15:46.972 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:15:46 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:15:46.972 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:15:46 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:15:46.972 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:15:46 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:15:46.973 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:15:46 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-14T19:15:46.973 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:15:46 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-14T19:15:46.973 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:15:46 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:15:46.993 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:15:46 smithi130 ceph-mon[97832]: pgmap v244: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T19:15:46.993 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:15:46 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:15:46.993 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:15:46 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:15:46.993 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:15:46 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:15:46.994 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:15:46 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-14T19:15:46.994 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:15:46 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-14T19:15:46.994 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:15:46 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:15:47.971 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:15:47 smithi040 ceph-mon[96518]: Deploying daemon mds.g on smithi040 2023-07-14T19:15:47.972 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:15:47 smithi040 ceph-mon[96518]: mds.? [v2:172.21.15.191:6838/1977507302,v1:172.21.15.191:6839/1977507302] up:boot 2023-07-14T19:15:47.972 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:15:47 smithi040 ceph-mon[96518]: fsmap 9 up:standby 2023-07-14T19:15:47.972 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:15:47 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2023-07-14T19:15:47.972 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:15:47 smithi040 ceph-mon[96518]: from='client.? 172.21.15.40:0/3190649182' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T19:15:47.972 INFO:journalctl@ceph.mds.g.smithi040.stdout:Jul 14 19:15:47 smithi040 systemd[1]: Starting Ceph mds.g for adc52f04-2279-11ee-9b33-001a4aab830c... 2023-07-14T19:15:47.973 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:15:47 smithi191 ceph-mon[98311]: Deploying daemon mds.g on smithi040 2023-07-14T19:15:47.973 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:15:47 smithi191 ceph-mon[98311]: mds.? [v2:172.21.15.191:6838/1977507302,v1:172.21.15.191:6839/1977507302] up:boot 2023-07-14T19:15:47.973 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:15:47 smithi191 ceph-mon[98311]: fsmap 9 up:standby 2023-07-14T19:15:47.973 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:15:47 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2023-07-14T19:15:47.973 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:15:47 smithi191 ceph-mon[98311]: from='client.? 172.21.15.40:0/3190649182' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T19:15:47.992 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:15:47 smithi130 ceph-mon[97832]: Deploying daemon mds.g on smithi040 2023-07-14T19:15:47.992 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:15:47 smithi130 ceph-mon[97832]: mds.? [v2:172.21.15.191:6838/1977507302,v1:172.21.15.191:6839/1977507302] up:boot 2023-07-14T19:15:47.992 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:15:47 smithi130 ceph-mon[97832]: fsmap 9 up:standby 2023-07-14T19:15:47.993 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:15:47 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2023-07-14T19:15:47.993 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:15:47 smithi130 ceph-mon[97832]: from='client.? 172.21.15.40:0/3190649182' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T19:15:48.364 INFO:journalctl@ceph.mds.g.smithi040.stdout:Jul 14 19:15:48 smithi040 podman[130820]: 2023-07-14T19:15:48.721 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:15:48 smithi040 ceph-mon[96518]: pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T19:15:48.721 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:15:48 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:15:48.721 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:15:48 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:15:48.722 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:15:48 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:15:48.722 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:15:48 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-14T19:15:48.722 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:15:48 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-14T19:15:48.722 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:15:48 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:15:48.723 INFO:journalctl@ceph.mds.g.smithi040.stdout:Jul 14 19:15:48 smithi040 ceph-mds[130860]: -- 172.21.15.40:0/1737029356 <== mon.0 v2:172.21.15.40:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55fd5d05f1e0 con 0x55fd5d034c00 2023-07-14T19:15:48.723 INFO:journalctl@ceph.mds.g.smithi040.stdout:Jul 14 19:15:48 smithi040 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-mds-g[130836]: starting mds.g at 2023-07-14T19:15:48.723 INFO:journalctl@ceph.mds.g.smithi040.stdout:Jul 14 19:15:48 smithi040 bash[130820]: 8dba83225dcd0270a30c1a79cf085076e61fa480bc55fb27040698abaa12b799 2023-07-14T19:15:48.723 INFO:journalctl@ceph.mds.g.smithi040.stdout:Jul 14 19:15:48 smithi040 systemd[1]: Started Ceph mds.g for adc52f04-2279-11ee-9b33-001a4aab830c. 2023-07-14T19:15:48.918 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-07-14T19:15:48.972 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:15:48 smithi191 ceph-mon[98311]: pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T19:15:48.972 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:15:48 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:15:48.972 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:15:48 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:15:48.972 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:15:48 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:15:48.973 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:15:48 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-14T19:15:48.973 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:15:48 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-14T19:15:48.973 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:15:48 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:15:48.992 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:15:48 smithi130 ceph-mon[97832]: pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T19:15:48.993 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:15:48 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:15:48.993 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:15:48 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:15:48.993 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:15:48 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:15:48.994 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:15:48 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-14T19:15:48.994 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:15:48 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-14T19:15:48.994 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:15:48 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:15:49.052 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:49.051+0000 7fb321637700 1 -- 172.21.15.40:0/1190021034 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb31c0feb00 msgr2=0x7fb31c0fef10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:15:49.053 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:49.051+0000 7fb321637700 1 --2- 172.21.15.40:0/1190021034 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb31c0feb00 0x7fb31c0fef10 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fb304009990 tx=0x7fb304009ca0 comp rx=0 tx=0).stop 2023-07-14T19:15:49.053 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:49.052+0000 7fb321637700 1 -- 172.21.15.40:0/1190021034 shutdown_connections 2023-07-14T19:15:49.053 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:49.052+0000 7fb321637700 1 --2- 172.21.15.40:0/1190021034 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb31c100680 0x7fb31c106f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:15:49.053 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:49.052+0000 7fb321637700 1 --2- 172.21.15.40:0/1190021034 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb31c0feb00 0x7fb31c0fef10 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:15:49.054 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:49.052+0000 7fb321637700 1 --2- 172.21.15.40:0/1190021034 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb31c0ffcf0 0x7fb31c100140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:15:49.054 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:49.052+0000 7fb321637700 1 -- 172.21.15.40:0/1190021034 >> 172.21.15.40:0/1190021034 conn(0x7fb31c0fa100 msgr2=0x7fb31c0fc550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:15:49.054 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:49.052+0000 7fb321637700 1 -- 172.21.15.40:0/1190021034 shutdown_connections 2023-07-14T19:15:49.054 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:49.052+0000 7fb321637700 1 -- 172.21.15.40:0/1190021034 wait complete. 2023-07-14T19:15:49.054 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:49.052+0000 7fb321637700 1 Processor -- start 2023-07-14T19:15:49.056 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:49.052+0000 7fb321637700 1 -- start start 2023-07-14T19:15:49.056 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:49.052+0000 7fb321637700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb31c0feb00 0x7fb31c0695d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:15:49.056 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:49.053+0000 7fb321637700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb31c0ffcf0 0x7fb31c19a660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:15:49.056 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:49.053+0000 7fb321637700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb31c100680 0x7fb31c19cca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:15:49.056 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:49.053+0000 7fb321637700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb31c19d440 con 0x7fb31c0feb00 2023-07-14T19:15:49.057 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:49.053+0000 7fb321637700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb31c19d5b0 con 0x7fb31c100680 2023-07-14T19:15:49.057 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:49.053+0000 7fb321637700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb31c19d770 con 0x7fb31c0ffcf0 2023-07-14T19:15:49.057 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:49.053+0000 7fb31affd700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb31c0feb00 0x7fb31c0695d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:15:49.057 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:49.053+0000 7fb31affd700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb31c0feb00 0x7fb31c0695d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:60950/0 (socket says 172.21.15.40:60950) 2023-07-14T19:15:49.057 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:49.053+0000 7fb31affd700 1 -- 172.21.15.40:0/1127856995 learned_addr learned my addr 172.21.15.40:0/1127856995 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-07-14T19:15:49.058 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:49.053+0000 7fb31b7fe700 1 --2- 172.21.15.40:0/1127856995 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb31c100680 0x7fb31c19cca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:15:49.058 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:49.053+0000 7fb31a7fc700 1 --2- 172.21.15.40:0/1127856995 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb31c0ffcf0 0x7fb31c19a660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:15:49.058 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:49.053+0000 7fb31affd700 1 -- 172.21.15.40:0/1127856995 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb31c0ffcf0 msgr2=0x7fb31c19a660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:15:49.058 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:49.053+0000 7fb31affd700 1 --2- 172.21.15.40:0/1127856995 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb31c0ffcf0 0x7fb31c19a660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:15:49.058 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:49.053+0000 7fb31affd700 1 -- 172.21.15.40:0/1127856995 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb31c100680 msgr2=0x7fb31c19cca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:15:49.058 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:49.053+0000 7fb31affd700 1 --2- 172.21.15.40:0/1127856995 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb31c100680 0x7fb31c19cca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:15:49.058 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:49.053+0000 7fb31affd700 1 -- 172.21.15.40:0/1127856995 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3040095f0 con 0x7fb31c0feb00 2023-07-14T19:15:49.059 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:49.053+0000 7fb31affd700 1 --2- 172.21.15.40:0/1127856995 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb31c0feb00 0x7fb31c0695d0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fb304000c00 tx=0x7fb3040047b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:15:49.059 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:49.053+0000 7fb313fff700 1 -- 172.21.15.40:0/1127856995 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb30401d070 con 0x7fb31c0feb00 2023-07-14T19:15:49.059 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:49.053+0000 7fb313fff700 1 -- 172.21.15.40:0/1127856995 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb304004060 con 0x7fb31c0feb00 2023-07-14T19:15:49.059 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:49.053+0000 7fb321637700 1 -- 172.21.15.40:0/1127856995 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb31c19d9f0 con 0x7fb31c0feb00 2023-07-14T19:15:49.059 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:49.053+0000 7fb313fff700 1 -- 172.21.15.40:0/1127856995 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb304022d30 con 0x7fb31c0feb00 2023-07-14T19:15:49.059 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:49.053+0000 7fb321637700 1 -- 172.21.15.40:0/1127856995 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb31c19dea0 con 0x7fb31c0feb00 2023-07-14T19:15:49.060 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:49.054+0000 7fb321637700 1 -- 172.21.15.40:0/1127856995 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb31c1060a0 con 0x7fb31c0feb00 2023-07-14T19:15:49.060 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:49.055+0000 7fb313fff700 1 -- 172.21.15.40:0/1127856995 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7fb304003ba0 con 0x7fb31c0feb00 2023-07-14T19:15:49.060 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:49.056+0000 7fb313fff700 1 --2- 172.21.15.40:0/1127856995 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7fb3080a11a0 0x7fb3080a3650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:15:49.061 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:49.056+0000 7fb31a7fc700 1 --2- 172.21.15.40:0/1127856995 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7fb3080a11a0 0x7fb3080a3650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:15:49.061 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:49.056+0000 7fb313fff700 1 -- 172.21.15.40:0/1127856995 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fb3040cd7b0 con 0x7fb31c0feb00 2023-07-14T19:15:49.061 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:49.056+0000 7fb31a7fc700 1 --2- 172.21.15.40:0/1127856995 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7fb3080a11a0 0x7fb3080a3650 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fb30c005950 tx=0x7fb30c0058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:15:49.061 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:49.060+0000 7fb313fff700 1 -- 172.21.15.40:0/1127856995 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fb304097390 con 0x7fb31c0feb00 2023-07-14T19:15:49.312 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:49.312+0000 7fb321637700 1 -- 172.21.15.40:0/1127856995 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb31c02c8e0 con 0x7fb31c0feb00 2023-07-14T19:15:49.313 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:49.312+0000 7fb313fff700 1 -- 172.21.15.40:0/1127856995 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 11 v11) v1 ==== 94+0+8082 (secure 0 0 0) 0x7fb30409a9b0 con 0x7fb31c0feb00 2023-07-14T19:15:49.313 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-07-14T19:15:49.314 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":11,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14523,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6835/501143718","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6834","nonce":501143718},{"type":"v1","addr":"172.21.15.40:6835","nonce":501143718}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14535,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6837/2634476117","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6836","nonce":2634476117},{"type":"v1","addr":"172.21.15.40:6837","nonce":2634476117}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":14586,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6839/959937545","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6838","nonce":959937545},{"type":"v1","addr":"172.21.15.40:6839","nonce":959937545}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":24433,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6833/31509862","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":31509862},{"type":"v1","addr":"172.21.15.191:6833","nonce":31509862}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24452,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.130:6833/3677452688","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6832","nonce":3677452688},{"type":"v1","addr":"172.21.15.130:6833","nonce":3677452688}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24478,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6835/3712674634","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":3712674634},{"type":"v1","addr":"172.21.15.191:6835","nonce":3712674634}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":7},{"gid":24482,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.130:6835/3088490808","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6834","nonce":3088490808},{"type":"v1","addr":"172.21.15.130:6835","nonce":3088490808}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6},{"gid":24490,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6837/2406892975","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6836","nonce":2406892975},{"type":"v1","addr":"172.21.15.191:6837","nonce":2406892975}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":8},{"gid":24506,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.130:6837/2604019369","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6836","nonce":2604019369},{"type":"v1","addr":"172.21.15.130:6837","nonce":2604019369}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":9},{"gid":24508,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6839/1977507302","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":1977507302},{"type":"v1","addr":"172.21.15.191:6839","nonce":1977507302}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[]} 2023-07-14T19:15:49.315 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:49.314+0000 7fb321637700 1 -- 172.21.15.40:0/1127856995 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7fb3080a11a0 msgr2=0x7fb3080a3650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:15:49.315 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:49.314+0000 7fb321637700 1 --2- 172.21.15.40:0/1127856995 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7fb3080a11a0 0x7fb3080a3650 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fb30c005950 tx=0x7fb30c0058e0 comp rx=0 tx=0).stop 2023-07-14T19:15:49.316 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:49.314+0000 7fb321637700 1 -- 172.21.15.40:0/1127856995 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb31c0feb00 msgr2=0x7fb31c0695d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:15:49.316 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:49.314+0000 7fb321637700 1 --2- 172.21.15.40:0/1127856995 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb31c0feb00 0x7fb31c0695d0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fb304000c00 tx=0x7fb3040047b0 comp rx=0 tx=0).stop 2023-07-14T19:15:49.316 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:49.315+0000 7fb321637700 1 -- 172.21.15.40:0/1127856995 shutdown_connections 2023-07-14T19:15:49.316 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:49.315+0000 7fb321637700 1 --2- 172.21.15.40:0/1127856995 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7fb3080a11a0 0x7fb3080a3650 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:15:49.316 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:49.315+0000 7fb321637700 1 --2- 172.21.15.40:0/1127856995 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb31c100680 0x7fb31c19cca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:15:49.316 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:49.315+0000 7fb321637700 1 --2- 172.21.15.40:0/1127856995 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb31c0feb00 0x7fb31c0695d0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:15:49.317 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:49.315+0000 7fb321637700 1 --2- 172.21.15.40:0/1127856995 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb31c0ffcf0 0x7fb31c19a660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:15:49.317 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:49.315+0000 7fb321637700 1 -- 172.21.15.40:0/1127856995 >> 172.21.15.40:0/1127856995 conn(0x7fb31c0fa100 msgr2=0x7fb31c1031e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:15:49.317 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:49.315+0000 7fb321637700 1 -- 172.21.15.40:0/1127856995 shutdown_connections 2023-07-14T19:15:49.317 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:49.315+0000 7fb321637700 1 -- 172.21.15.40:0/1127856995 wait complete. 2023-07-14T19:15:49.317 INFO:teuthology.orchestra.run.smithi040.stderr:dumped fsmap epoch 11 2023-07-14T19:15:49.791 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:15:49 smithi040 ceph-mon[96518]: Deploying daemon mds.a on smithi040 2023-07-14T19:15:49.792 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:15:49 smithi040 ceph-mon[96518]: mds.? [v2:172.21.15.40:6838/959937545,v1:172.21.15.40:6839/959937545] up:boot 2023-07-14T19:15:49.792 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:15:49 smithi040 ceph-mon[96518]: fsmap 10 up:standby 2023-07-14T19:15:49.792 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:15:49 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2023-07-14T19:15:49.793 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:15:49 smithi040 ceph-mon[96518]: from='client.? 172.21.15.40:0/1127856995' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T19:15:49.972 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:15:49 smithi191 ceph-mon[98311]: Deploying daemon mds.a on smithi040 2023-07-14T19:15:49.972 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:15:49 smithi191 ceph-mon[98311]: mds.? [v2:172.21.15.40:6838/959937545,v1:172.21.15.40:6839/959937545] up:boot 2023-07-14T19:15:49.972 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:15:49 smithi191 ceph-mon[98311]: fsmap 10 up:standby 2023-07-14T19:15:49.973 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:15:49 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2023-07-14T19:15:49.973 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:15:49 smithi191 ceph-mon[98311]: from='client.? 172.21.15.40:0/1127856995' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T19:15:49.992 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:15:49 smithi130 ceph-mon[97832]: Deploying daemon mds.a on smithi040 2023-07-14T19:15:49.993 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:15:49 smithi130 ceph-mon[97832]: mds.? [v2:172.21.15.40:6838/959937545,v1:172.21.15.40:6839/959937545] up:boot 2023-07-14T19:15:49.993 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:15:49 smithi130 ceph-mon[97832]: fsmap 10 up:standby 2023-07-14T19:15:49.993 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:15:49 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2023-07-14T19:15:49.993 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:15:49 smithi130 ceph-mon[97832]: from='client.? 172.21.15.40:0/1127856995' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T19:15:50.110 INFO:journalctl@ceph.mds.a.smithi040.stdout:Jul 14 19:15:50 smithi040 systemd[1]: Starting Ceph mds.a for adc52f04-2279-11ee-9b33-001a4aab830c... 2023-07-14T19:15:50.702 INFO:journalctl@ceph.mds.a.smithi040.stdout:Jul 14 19:15:50 smithi040 podman[131207]: 2023-07-14T19:15:50.703 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:15:50 smithi040 ceph-mon[96518]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T19:15:50.971 INFO:journalctl@ceph.mds.a.smithi040.stdout:Jul 14 19:15:50 smithi040 ceph-mds[131247]: -- 172.21.15.40:0/1929343020 <== mon.0 v2:172.21.15.40:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5640821ab1e0 con 0x564082180c00 2023-07-14T19:15:50.971 INFO:journalctl@ceph.mds.a.smithi040.stdout:Jul 14 19:15:50 smithi040 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-mds-a[131223]: starting mds.a at 2023-07-14T19:15:50.972 INFO:journalctl@ceph.mds.a.smithi040.stdout:Jul 14 19:15:50 smithi040 bash[131207]: ff53cd0b9a41361affe1391c268f447a2aca7d861793ec600ed02478c0c61abc 2023-07-14T19:15:50.972 INFO:journalctl@ceph.mds.a.smithi040.stdout:Jul 14 19:15:50 smithi040 systemd[1]: Started Ceph mds.a for adc52f04-2279-11ee-9b33-001a4aab830c. 2023-07-14T19:15:50.972 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:15:50 smithi191 ceph-mon[98311]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T19:15:50.992 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:15:50 smithi130 ceph-mon[97832]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T19:15:51.330 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-07-14T19:15:51.469 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:51.466+0000 7f03aebac700 1 -- 172.21.15.40:0/350680965 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f03a80feb00 msgr2=0x7f03a80fef10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:15:51.469 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:51.466+0000 7f03aebac700 1 --2- 172.21.15.40:0/350680965 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f03a80feb00 0x7f03a80fef10 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f039c009990 tx=0x7f039c009ca0 comp rx=0 tx=0).stop 2023-07-14T19:15:51.469 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:51.467+0000 7f03aebac700 1 -- 172.21.15.40:0/350680965 shutdown_connections 2023-07-14T19:15:51.469 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:51.467+0000 7f03aebac700 1 --2- 172.21.15.40:0/350680965 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f03a8100680 0x7f03a8106f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:15:51.470 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:51.467+0000 7f03aebac700 1 --2- 172.21.15.40:0/350680965 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f03a80feb00 0x7f03a80fef10 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:15:51.470 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:51.467+0000 7f03aebac700 1 --2- 172.21.15.40:0/350680965 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f03a80ffcf0 0x7f03a8100140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:15:51.470 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:51.467+0000 7f03aebac700 1 -- 172.21.15.40:0/350680965 >> 172.21.15.40:0/350680965 conn(0x7f03a80fa100 msgr2=0x7f03a80fc550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:15:51.470 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:51.467+0000 7f03aebac700 1 -- 172.21.15.40:0/350680965 shutdown_connections 2023-07-14T19:15:51.470 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:51.467+0000 7f03aebac700 1 -- 172.21.15.40:0/350680965 wait complete. 2023-07-14T19:15:51.470 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:51.467+0000 7f03aebac700 1 Processor -- start 2023-07-14T19:15:51.470 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:51.467+0000 7f03aebac700 1 -- start start 2023-07-14T19:15:51.472 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:51.467+0000 7f03aebac700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f03a80feb00 0x7f03a8068310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:15:51.473 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:51.467+0000 7f03aebac700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f03a80ffcf0 0x7f03a819ca10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:15:51.473 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:51.467+0000 7f03aebac700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f03a8100680 0x7f03a819ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:15:51.473 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:51.467+0000 7f03aebac700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03a819f610 con 0x7f03a80ffcf0 2023-07-14T19:15:51.473 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:51.467+0000 7f03aebac700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03a819f780 con 0x7f03a8100680 2023-07-14T19:15:51.473 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:51.467+0000 7f03aebac700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03a819f940 con 0x7f03a80feb00 2023-07-14T19:15:51.473 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:51.467+0000 7f03a7fff700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f03a80ffcf0 0x7f03a819ca10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:15:51.474 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:51.468+0000 7f03a7fff700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f03a80ffcf0 0x7f03a819ca10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:60974/0 (socket says 172.21.15.40:60974) 2023-07-14T19:15:51.474 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:51.468+0000 7f03a7fff700 1 -- 172.21.15.40:0/2078797805 learned_addr learned my addr 172.21.15.40:0/2078797805 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-07-14T19:15:51.474 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:51.468+0000 7f03ac948700 1 --2- 172.21.15.40:0/2078797805 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f03a80feb00 0x7f03a8068310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:15:51.474 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:51.468+0000 7f03ad149700 1 --2- 172.21.15.40:0/2078797805 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f03a8100680 0x7f03a819ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:15:51.474 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:51.468+0000 7f03a7fff700 1 -- 172.21.15.40:0/2078797805 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f03a80feb00 msgr2=0x7f03a8068310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:15:51.475 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:51.468+0000 7f03a7fff700 1 --2- 172.21.15.40:0/2078797805 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f03a80feb00 0x7f03a8068310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:15:51.475 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:51.468+0000 7f03a7fff700 1 -- 172.21.15.40:0/2078797805 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f03a8100680 msgr2=0x7f03a819ef60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:15:51.475 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:51.468+0000 7f03a7fff700 1 --2- 172.21.15.40:0/2078797805 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f03a8100680 0x7f03a819ef60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:15:51.475 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:51.468+0000 7f03a7fff700 1 -- 172.21.15.40:0/2078797805 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f039c0095f0 con 0x7f03a80ffcf0 2023-07-14T19:15:51.476 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:51.468+0000 7f03a7fff700 1 --2- 172.21.15.40:0/2078797805 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f03a80ffcf0 0x7f03a819ca10 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f039400c9b0 tx=0x7f039400ccc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:15:51.476 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:51.468+0000 7f03a5ffb700 1 -- 172.21.15.40:0/2078797805 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0394007b00 con 0x7f03a80ffcf0 2023-07-14T19:15:51.476 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:51.468+0000 7f03a5ffb700 1 -- 172.21.15.40:0/2078797805 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f0394007c60 con 0x7f03a80ffcf0 2023-07-14T19:15:51.477 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:51.468+0000 7f03aebac700 1 -- 172.21.15.40:0/2078797805 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f03a819fc20 con 0x7f03a80ffcf0 2023-07-14T19:15:51.477 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:51.468+0000 7f03a5ffb700 1 -- 172.21.15.40:0/2078797805 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0394018660 con 0x7f03a80ffcf0 2023-07-14T19:15:51.477 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:51.468+0000 7f03aebac700 1 -- 172.21.15.40:0/2078797805 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f03a81a0180 con 0x7f03a80ffcf0 2023-07-14T19:15:51.478 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:51.469+0000 7f03aebac700 1 -- 172.21.15.40:0/2078797805 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f03a810a320 con 0x7f03a80ffcf0 2023-07-14T19:15:51.478 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:51.470+0000 7f03a5ffb700 1 -- 172.21.15.40:0/2078797805 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f039400f450 con 0x7f03a80ffcf0 2023-07-14T19:15:51.478 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:51.471+0000 7f03a5ffb700 1 --2- 172.21.15.40:0/2078797805 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f03980a1220 0x7f03980a36d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:15:51.478 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:51.471+0000 7f03a5ffb700 1 -- 172.21.15.40:0/2078797805 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f03940cc3b0 con 0x7f03a80ffcf0 2023-07-14T19:15:51.479 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:51.471+0000 7f03ac948700 1 --2- 172.21.15.40:0/2078797805 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f03980a1220 0x7f03980a36d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:15:51.480 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:51.471+0000 7f03ac948700 1 --2- 172.21.15.40:0/2078797805 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f03980a1220 0x7f03980a36d0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f039c00b5c0 tx=0x7f039c005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:15:51.480 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:51.475+0000 7f03a5ffb700 1 -- 172.21.15.40:0/2078797805 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f0394095fa0 con 0x7f03a80ffcf0 2023-07-14T19:15:51.725 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:51.723+0000 7f03aebac700 1 -- 172.21.15.40:0/2078797805 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f03a810a510 con 0x7f03a80ffcf0 2023-07-14T19:15:51.725 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:51.724+0000 7f03a5ffb700 1 -- 172.21.15.40:0/2078797805 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 12 v12) v1 ==== 94+0+8840 (secure 0 0 0) 0x7f03940995c0 con 0x7f03a80ffcf0 2023-07-14T19:15:51.726 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-07-14T19:15:51.726 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":12,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14523,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6835/501143718","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6834","nonce":501143718},{"type":"v1","addr":"172.21.15.40:6835","nonce":501143718}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14535,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6837/2634476117","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6836","nonce":2634476117},{"type":"v1","addr":"172.21.15.40:6837","nonce":2634476117}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":14586,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6839/959937545","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6838","nonce":959937545},{"type":"v1","addr":"172.21.15.40:6839","nonce":959937545}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":14598,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6841/1590486077","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6840","nonce":1590486077},{"type":"v1","addr":"172.21.15.40:6841","nonce":1590486077}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":24433,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6833/31509862","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":31509862},{"type":"v1","addr":"172.21.15.191:6833","nonce":31509862}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24452,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.130:6833/3677452688","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6832","nonce":3677452688},{"type":"v1","addr":"172.21.15.130:6833","nonce":3677452688}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24478,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6835/3712674634","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":3712674634},{"type":"v1","addr":"172.21.15.191:6835","nonce":3712674634}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":7},{"gid":24482,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.130:6835/3088490808","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6834","nonce":3088490808},{"type":"v1","addr":"172.21.15.130:6835","nonce":3088490808}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6},{"gid":24490,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6837/2406892975","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6836","nonce":2406892975},{"type":"v1","addr":"172.21.15.191:6837","nonce":2406892975}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":8},{"gid":24506,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.130:6837/2604019369","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6836","nonce":2604019369},{"type":"v1","addr":"172.21.15.130:6837","nonce":2604019369}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":9},{"gid":24508,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6839/1977507302","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":1977507302},{"type":"v1","addr":"172.21.15.191:6839","nonce":1977507302}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[]} 2023-07-14T19:15:51.728 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:51.726+0000 7f03aebac700 1 -- 172.21.15.40:0/2078797805 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f03980a1220 msgr2=0x7f03980a36d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:15:51.728 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:51.726+0000 7f03aebac700 1 --2- 172.21.15.40:0/2078797805 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f03980a1220 0x7f03980a36d0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f039c00b5c0 tx=0x7f039c005fb0 comp rx=0 tx=0).stop 2023-07-14T19:15:51.728 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:51.726+0000 7f03aebac700 1 -- 172.21.15.40:0/2078797805 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f03a80ffcf0 msgr2=0x7f03a819ca10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:15:51.728 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:51.726+0000 7f03aebac700 1 --2- 172.21.15.40:0/2078797805 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f03a80ffcf0 0x7f03a819ca10 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f039400c9b0 tx=0x7f039400ccc0 comp rx=0 tx=0).stop 2023-07-14T19:15:51.729 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:51.726+0000 7f03aebac700 1 -- 172.21.15.40:0/2078797805 shutdown_connections 2023-07-14T19:15:51.729 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:51.726+0000 7f03aebac700 1 --2- 172.21.15.40:0/2078797805 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f03980a1220 0x7f03980a36d0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:15:51.729 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:51.726+0000 7f03aebac700 1 --2- 172.21.15.40:0/2078797805 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f03a80feb00 0x7f03a8068310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:15:51.729 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:51.726+0000 7f03aebac700 1 --2- 172.21.15.40:0/2078797805 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f03a8100680 0x7f03a819ef60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:15:51.729 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:51.726+0000 7f03aebac700 1 --2- 172.21.15.40:0/2078797805 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f03a80ffcf0 0x7f03a819ca10 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:15:51.729 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:51.726+0000 7f03aebac700 1 -- 172.21.15.40:0/2078797805 >> 172.21.15.40:0/2078797805 conn(0x7f03a80fa100 msgr2=0x7f03a80fc270 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:15:51.729 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:51.726+0000 7f03aebac700 1 -- 172.21.15.40:0/2078797805 shutdown_connections 2023-07-14T19:15:51.729 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:51.726+0000 7f03aebac700 1 -- 172.21.15.40:0/2078797805 wait complete. 2023-07-14T19:15:51.730 INFO:teuthology.orchestra.run.smithi040.stderr:dumped fsmap epoch 12 2023-07-14T19:15:52.221 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:15:51 smithi040 ceph-mon[96518]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T19:15:52.221 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:15:51 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:15:52.221 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:15:51 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:15:52.222 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:15:51 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:15:52.222 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:15:51 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-14T19:15:52.222 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:15:51 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-14T19:15:52.222 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:15:51 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:15:52.222 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:15:51 smithi040 ceph-mon[96518]: Deploying daemon mds.b on smithi130 2023-07-14T19:15:52.222 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:15:51 smithi040 ceph-mon[96518]: mds.? [v2:172.21.15.40:6840/1590486077,v1:172.21.15.40:6841/1590486077] up:boot 2023-07-14T19:15:52.222 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:15:51 smithi040 ceph-mon[96518]: fsmap 11 up:standby 2023-07-14T19:15:52.222 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:15:51 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2023-07-14T19:15:52.223 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:15:51 smithi040 ceph-mon[96518]: from='client.? 172.21.15.40:0/2078797805' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T19:15:52.223 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:15:51 smithi191 ceph-mon[98311]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T19:15:52.224 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:15:51 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:15:52.224 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:15:51 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:15:52.224 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:15:51 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:15:52.224 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:15:51 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-14T19:15:52.224 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:15:51 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-14T19:15:52.224 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:15:51 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:15:52.225 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:15:51 smithi191 ceph-mon[98311]: Deploying daemon mds.b on smithi130 2023-07-14T19:15:52.225 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:15:51 smithi191 ceph-mon[98311]: mds.? [v2:172.21.15.40:6840/1590486077,v1:172.21.15.40:6841/1590486077] up:boot 2023-07-14T19:15:52.225 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:15:51 smithi191 ceph-mon[98311]: fsmap 11 up:standby 2023-07-14T19:15:52.225 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:15:51 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2023-07-14T19:15:52.225 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:15:51 smithi191 ceph-mon[98311]: from='client.? 172.21.15.40:0/2078797805' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T19:15:52.238 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:15:51 smithi130 ceph-mon[97832]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T19:15:52.238 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:15:51 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:15:52.239 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:15:51 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:15:52.239 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:15:51 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:15:52.239 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:15:51 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-14T19:15:52.239 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:15:51 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-14T19:15:52.239 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:15:51 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:15:52.239 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:15:51 smithi130 ceph-mon[97832]: Deploying daemon mds.b on smithi130 2023-07-14T19:15:52.239 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:15:51 smithi130 ceph-mon[97832]: mds.? [v2:172.21.15.40:6840/1590486077,v1:172.21.15.40:6841/1590486077] up:boot 2023-07-14T19:15:52.240 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:15:51 smithi130 ceph-mon[97832]: fsmap 11 up:standby 2023-07-14T19:15:52.240 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:15:51 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2023-07-14T19:15:52.240 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:15:51 smithi130 ceph-mon[97832]: from='client.? 172.21.15.40:0/2078797805' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T19:15:52.492 INFO:journalctl@ceph.mds.b.smithi130.stdout:Jul 14 19:15:52 smithi130 systemd[1]: Starting Ceph mds.b for adc52f04-2279-11ee-9b33-001a4aab830c... 2023-07-14T19:15:53.062 INFO:journalctl@ceph.mds.b.smithi130.stdout:Jul 14 19:15:52 smithi130 podman[122827]: 2023-07-14T19:15:53.062 INFO:journalctl@ceph.mds.b.smithi130.stdout:Jul 14 19:15:53 smithi130 ceph-mds[122868]: -- 172.21.15.130:0/1401558699 <== mon.2 v2:172.21.15.130:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5645637431e0 con 0x564563718800 2023-07-14T19:15:53.062 INFO:journalctl@ceph.mds.b.smithi130.stdout:Jul 14 19:15:53 smithi130 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-mds-b[122843]: starting mds.b at 2023-07-14T19:15:53.492 INFO:journalctl@ceph.mds.b.smithi130.stdout:Jul 14 19:15:53 smithi130 bash[122827]: 7b43eb14e326ee8ce76faf06d0121c8401e02471c23356d8c298b7cb827b966a 2023-07-14T19:15:53.492 INFO:journalctl@ceph.mds.b.smithi130.stdout:Jul 14 19:15:53 smithi130 systemd[1]: Started Ceph mds.b for adc52f04-2279-11ee-9b33-001a4aab830c. 2023-07-14T19:15:53.741 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-07-14T19:15:53.879 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:53.877+0000 7f9423459700 1 -- 172.21.15.40:0/4277736311 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f941c0feb00 msgr2=0x7f941c0fef10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:15:53.879 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:53.877+0000 7f9423459700 1 --2- 172.21.15.40:0/4277736311 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f941c0feb00 0x7f941c0fef10 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f9418009990 tx=0x7f9418009ca0 comp rx=0 tx=0).stop 2023-07-14T19:15:53.880 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:53.877+0000 7f9423459700 1 -- 172.21.15.40:0/4277736311 shutdown_connections 2023-07-14T19:15:53.880 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:53.877+0000 7f9423459700 1 --2- 172.21.15.40:0/4277736311 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f941c100680 0x7f941c106f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:15:53.880 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:53.877+0000 7f9423459700 1 --2- 172.21.15.40:0/4277736311 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f941c0feb00 0x7f941c0fef10 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:15:53.880 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:53.877+0000 7f9423459700 1 --2- 172.21.15.40:0/4277736311 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f941c0ffcf0 0x7f941c100140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:15:53.880 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:53.877+0000 7f9423459700 1 -- 172.21.15.40:0/4277736311 >> 172.21.15.40:0/4277736311 conn(0x7f941c0fa100 msgr2=0x7f941c0fc550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:15:53.881 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:53.877+0000 7f9423459700 1 -- 172.21.15.40:0/4277736311 shutdown_connections 2023-07-14T19:15:53.881 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:53.877+0000 7f9423459700 1 -- 172.21.15.40:0/4277736311 wait complete. 2023-07-14T19:15:53.881 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:53.878+0000 7f9423459700 1 Processor -- start 2023-07-14T19:15:53.881 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:53.878+0000 7f9423459700 1 -- start start 2023-07-14T19:15:53.883 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:53.878+0000 7f9423459700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f941c0feb00 0x7f941c068310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:15:53.884 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:53.878+0000 7f9423459700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f941c0ffcf0 0x7f941c19a6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:15:53.884 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:53.878+0000 7f9423459700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f941c100680 0x7f941c19ccf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:15:53.884 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:53.878+0000 7f9423459700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f941c19d490 con 0x7f941c0feb00 2023-07-14T19:15:53.884 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:53.878+0000 7f9423459700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f941c19d600 con 0x7f941c100680 2023-07-14T19:15:53.885 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:53.878+0000 7f9423459700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f941c19d7c0 con 0x7f941c0ffcf0 2023-07-14T19:15:53.885 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:53.878+0000 7f94211f5700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f941c0feb00 0x7f941c068310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:15:53.885 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:53.878+0000 7f94211f5700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f941c0feb00 0x7f941c068310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:60986/0 (socket says 172.21.15.40:60986) 2023-07-14T19:15:53.885 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:53.878+0000 7f94211f5700 1 -- 172.21.15.40:0/409937961 learned_addr learned my addr 172.21.15.40:0/409937961 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-07-14T19:15:53.885 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:53.878+0000 7f94209f4700 1 --2- 172.21.15.40:0/409937961 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f941c0ffcf0 0x7f941c19a6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:15:53.885 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:53.878+0000 7f94219f6700 1 --2- 172.21.15.40:0/409937961 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f941c100680 0x7f941c19ccf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:15:53.885 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:53.878+0000 7f94211f5700 1 -- 172.21.15.40:0/409937961 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f941c0ffcf0 msgr2=0x7f941c19a6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:15:53.886 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:53.878+0000 7f94211f5700 1 --2- 172.21.15.40:0/409937961 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f941c0ffcf0 0x7f941c19a6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:15:53.886 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:53.878+0000 7f94211f5700 1 -- 172.21.15.40:0/409937961 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f941c100680 msgr2=0x7f941c19ccf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:15:53.886 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:53.878+0000 7f94211f5700 1 --2- 172.21.15.40:0/409937961 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f941c100680 0x7f941c19ccf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:15:53.886 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:53.878+0000 7f94211f5700 1 -- 172.21.15.40:0/409937961 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94180095f0 con 0x7f941c0feb00 2023-07-14T19:15:53.886 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:53.878+0000 7f94211f5700 1 --2- 172.21.15.40:0/409937961 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f941c0feb00 0x7f941c068310 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f9418004760 tx=0x7f9418009580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:15:53.886 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:53.878+0000 7f94127fc700 1 -- 172.21.15.40:0/409937961 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f941801d070 con 0x7f941c0feb00 2023-07-14T19:15:53.886 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:53.879+0000 7f94127fc700 1 -- 172.21.15.40:0/409937961 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f9418004cd0 con 0x7f941c0feb00 2023-07-14T19:15:53.887 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:53.879+0000 7f9423459700 1 -- 172.21.15.40:0/409937961 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f941c19da40 con 0x7f941c0feb00 2023-07-14T19:15:53.887 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:53.879+0000 7f94127fc700 1 -- 172.21.15.40:0/409937961 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9418005290 con 0x7f941c0feb00 2023-07-14T19:15:53.887 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:53.879+0000 7f9423459700 1 -- 172.21.15.40:0/409937961 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f941c19dec0 con 0x7f941c0feb00 2023-07-14T19:15:53.887 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:53.880+0000 7f9423459700 1 -- 172.21.15.40:0/409937961 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f941c1060a0 con 0x7f941c0feb00 2023-07-14T19:15:53.887 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:53.881+0000 7f94127fc700 1 -- 172.21.15.40:0/409937961 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f9418003b60 con 0x7f941c0feb00 2023-07-14T19:15:53.888 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:53.881+0000 7f94127fc700 1 --2- 172.21.15.40:0/409937961 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f94080a1220 0x7f94080a36d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:15:53.888 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:53.881+0000 7f94127fc700 1 -- 172.21.15.40:0/409937961 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f94180cd3c0 con 0x7f941c0feb00 2023-07-14T19:15:53.889 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:53.881+0000 7f94209f4700 1 --2- 172.21.15.40:0/409937961 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f94080a1220 0x7f94080a36d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:15:53.889 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:53.882+0000 7f94209f4700 1 --2- 172.21.15.40:0/409937961 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f94080a1220 0x7f94080a36d0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f940c005950 tx=0x7f940c00b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:15:53.889 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:53.885+0000 7f94127fc700 1 -- 172.21.15.40:0/409937961 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f9418097040 con 0x7f941c0feb00 2023-07-14T19:15:54.149 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:54.148+0000 7f9423459700 1 -- 172.21.15.40:0/409937961 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f941c02c8e0 con 0x7f941c0feb00 2023-07-14T19:15:54.151 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:54.148+0000 7f94127fc700 1 -- 172.21.15.40:0/409937961 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 13 v13) v1 ==== 94+0+9601 (secure 0 0 0) 0x7f941809a660 con 0x7f941c0feb00 2023-07-14T19:15:54.151 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-07-14T19:15:54.151 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":13,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14523,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6835/501143718","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6834","nonce":501143718},{"type":"v1","addr":"172.21.15.40:6835","nonce":501143718}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14535,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6837/2634476117","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6836","nonce":2634476117},{"type":"v1","addr":"172.21.15.40:6837","nonce":2634476117}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":14586,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6839/959937545","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6838","nonce":959937545},{"type":"v1","addr":"172.21.15.40:6839","nonce":959937545}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":14598,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6841/1590486077","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6840","nonce":1590486077},{"type":"v1","addr":"172.21.15.40:6841","nonce":1590486077}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":24433,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6833/31509862","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":31509862},{"type":"v1","addr":"172.21.15.191:6833","nonce":31509862}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24452,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.130:6833/3677452688","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6832","nonce":3677452688},{"type":"v1","addr":"172.21.15.130:6833","nonce":3677452688}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24478,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6835/3712674634","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":3712674634},{"type":"v1","addr":"172.21.15.191:6835","nonce":3712674634}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":7},{"gid":24482,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.130:6835/3088490808","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6834","nonce":3088490808},{"type":"v1","addr":"172.21.15.130:6835","nonce":3088490808}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6},{"gid":24490,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6837/2406892975","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6836","nonce":2406892975},{"type":"v1","addr":"172.21.15.191:6837","nonce":2406892975}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":8},{"gid":24506,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.130:6837/2604019369","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6836","nonce":2604019369},{"type":"v1","addr":"172.21.15.130:6837","nonce":2604019369}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":9},{"gid":24508,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6839/1977507302","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":1977507302},{"type":"v1","addr":"172.21.15.191:6839","nonce":1977507302}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10},{"gid":24539,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.130:6839/2145479535","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6838","nonce":2145479535},{"type":"v1","addr":"172.21.15.130:6839","nonce":2145479535}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":13}],"filesystems":[]} 2023-07-14T19:15:54.152 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:54.150+0000 7f9423459700 1 -- 172.21.15.40:0/409937961 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f94080a1220 msgr2=0x7f94080a36d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:15:54.153 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:54.150+0000 7f9423459700 1 --2- 172.21.15.40:0/409937961 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f94080a1220 0x7f94080a36d0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f940c005950 tx=0x7f940c00b410 comp rx=0 tx=0).stop 2023-07-14T19:15:54.153 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:54.150+0000 7f9423459700 1 -- 172.21.15.40:0/409937961 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f941c0feb00 msgr2=0x7f941c068310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:15:54.153 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:54.150+0000 7f9423459700 1 --2- 172.21.15.40:0/409937961 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f941c0feb00 0x7f941c068310 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f9418004760 tx=0x7f9418009580 comp rx=0 tx=0).stop 2023-07-14T19:15:54.153 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:54.151+0000 7f9423459700 1 -- 172.21.15.40:0/409937961 shutdown_connections 2023-07-14T19:15:54.153 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:54.151+0000 7f9423459700 1 --2- 172.21.15.40:0/409937961 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f94080a1220 0x7f94080a36d0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:15:54.153 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:54.151+0000 7f9423459700 1 --2- 172.21.15.40:0/409937961 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f941c100680 0x7f941c19ccf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:15:54.154 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:54.151+0000 7f9423459700 1 --2- 172.21.15.40:0/409937961 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f941c0feb00 0x7f941c068310 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:15:54.154 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:54.151+0000 7f9423459700 1 --2- 172.21.15.40:0/409937961 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f941c0ffcf0 0x7f941c19a6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:15:54.154 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:54.151+0000 7f9423459700 1 -- 172.21.15.40:0/409937961 >> 172.21.15.40:0/409937961 conn(0x7f941c0fa100 msgr2=0x7f941c1031e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:15:54.154 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:54.151+0000 7f9423459700 1 -- 172.21.15.40:0/409937961 shutdown_connections 2023-07-14T19:15:54.154 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:54.151+0000 7f9423459700 1 -- 172.21.15.40:0/409937961 wait complete. 2023-07-14T19:15:54.155 INFO:teuthology.orchestra.run.smithi040.stderr:dumped fsmap epoch 13 2023-07-14T19:15:54.165 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-07-14T19:15:54.294 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:54.292+0000 7fa761e56700 1 -- 172.21.15.40:0/1106120292 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa75c0feb10 msgr2=0x7fa75c0fef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:15:54.295 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:54.292+0000 7fa761e56700 1 --2- 172.21.15.40:0/1106120292 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa75c0feb10 0x7fa75c0fef20 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fa74c009990 tx=0x7fa74c009ca0 comp rx=0 tx=0).stop 2023-07-14T19:15:54.295 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:54.292+0000 7fa761e56700 1 -- 172.21.15.40:0/1106120292 shutdown_connections 2023-07-14T19:15:54.295 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:54.292+0000 7fa761e56700 1 --2- 172.21.15.40:0/1106120292 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa75c100690 0x7fa75c106f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:15:54.295 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:54.292+0000 7fa761e56700 1 --2- 172.21.15.40:0/1106120292 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa75c0feb10 0x7fa75c0fef20 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:15:54.295 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:54.292+0000 7fa761e56700 1 --2- 172.21.15.40:0/1106120292 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa75c0ffd00 0x7fa75c100150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:15:54.296 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:54.292+0000 7fa761e56700 1 -- 172.21.15.40:0/1106120292 >> 172.21.15.40:0/1106120292 conn(0x7fa75c0fa130 msgr2=0x7fa75c0fc560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:15:54.296 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:54.293+0000 7fa761e56700 1 -- 172.21.15.40:0/1106120292 shutdown_connections 2023-07-14T19:15:54.296 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:54.293+0000 7fa761e56700 1 -- 172.21.15.40:0/1106120292 wait complete. 2023-07-14T19:15:54.298 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:54.293+0000 7fa761e56700 1 Processor -- start 2023-07-14T19:15:54.298 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:54.294+0000 7fa761e56700 1 -- start start 2023-07-14T19:15:54.298 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:54.294+0000 7fa761e56700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa75c0feb10 0x7fa75c0695f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:15:54.298 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:54.294+0000 7fa761e56700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa75c0ffd00 0x7fa75c19a620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:15:54.298 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:54.294+0000 7fa761e56700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa75c100690 0x7fa75c19ccb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:15:54.299 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:54.294+0000 7fa761e56700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa75c19d450 con 0x7fa75c0feb10 2023-07-14T19:15:54.299 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:54.294+0000 7fa761e56700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa75c19d5c0 con 0x7fa75c0ffd00 2023-07-14T19:15:54.299 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:54.294+0000 7fa761e56700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa75c19d780 con 0x7fa75c100690 2023-07-14T19:15:54.299 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:54.294+0000 7fa75b7fe700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa75c0feb10 0x7fa75c0695f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:15:54.299 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:54.294+0000 7fa75b7fe700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa75c0feb10 0x7fa75c0695f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:32768/0 (socket says 172.21.15.40:32768) 2023-07-14T19:15:54.299 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:54.294+0000 7fa75b7fe700 1 -- 172.21.15.40:0/95884552 learned_addr learned my addr 172.21.15.40:0/95884552 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-07-14T19:15:54.299 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:54.294+0000 7fa75affd700 1 --2- 172.21.15.40:0/95884552 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa75c0ffd00 0x7fa75c19a620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:15:54.300 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:54.294+0000 7fa75bfff700 1 --2- 172.21.15.40:0/95884552 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa75c100690 0x7fa75c19ccb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:15:54.300 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:54.294+0000 7fa75b7fe700 1 -- 172.21.15.40:0/95884552 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa75c100690 msgr2=0x7fa75c19ccb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:15:54.300 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:54.294+0000 7fa75b7fe700 1 --2- 172.21.15.40:0/95884552 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa75c100690 0x7fa75c19ccb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:15:54.300 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:54.294+0000 7fa75b7fe700 1 -- 172.21.15.40:0/95884552 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa75c0ffd00 msgr2=0x7fa75c19a620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:15:54.300 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:54.294+0000 7fa75b7fe700 1 --2- 172.21.15.40:0/95884552 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa75c0ffd00 0x7fa75c19a620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:15:54.300 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:54.294+0000 7fa75b7fe700 1 -- 172.21.15.40:0/95884552 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa74c0095f0 con 0x7fa75c0feb10 2023-07-14T19:15:54.301 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:54.294+0000 7fa75bfff700 1 --2- 172.21.15.40:0/95884552 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa75c100690 0x7fa75c19ccb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-14T19:15:54.301 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:54.294+0000 7fa75b7fe700 1 --2- 172.21.15.40:0/95884552 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa75c0feb10 0x7fa75c0695f0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fa74c0098e0 tx=0x7fa74c005e80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:15:54.301 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:54.295+0000 7fa758ff9700 1 -- 172.21.15.40:0/95884552 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa74c01d070 con 0x7fa75c0feb10 2023-07-14T19:15:54.301 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:54.295+0000 7fa761e56700 1 -- 172.21.15.40:0/95884552 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa75c19da00 con 0x7fa75c0feb10 2023-07-14T19:15:54.301 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:54.295+0000 7fa758ff9700 1 -- 172.21.15.40:0/95884552 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa74c004ca0 con 0x7fa75c0feb10 2023-07-14T19:15:54.301 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:54.295+0000 7fa758ff9700 1 -- 172.21.15.40:0/95884552 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa74c005510 con 0x7fa75c0feb10 2023-07-14T19:15:54.301 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:54.295+0000 7fa761e56700 1 -- 172.21.15.40:0/95884552 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa75c19de20 con 0x7fa75c0feb10 2023-07-14T19:15:54.301 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:54.295+0000 7fa761e56700 1 -- 172.21.15.40:0/95884552 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa75c106050 con 0x7fa75c0feb10 2023-07-14T19:15:54.302 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:54.297+0000 7fa758ff9700 1 -- 172.21.15.40:0/95884552 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7fa74c00fc40 con 0x7fa75c0feb10 2023-07-14T19:15:54.303 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:54.297+0000 7fa758ff9700 1 --2- 172.21.15.40:0/95884552 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7fa7440a11a0 0x7fa7440a3650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:15:54.303 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:54.297+0000 7fa758ff9700 1 -- 172.21.15.40:0/95884552 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fa74c0cdc60 con 0x7fa75c0feb10 2023-07-14T19:15:54.303 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:54.297+0000 7fa75affd700 1 --2- 172.21.15.40:0/95884552 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7fa7440a11a0 0x7fa7440a3650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:15:54.303 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:54.298+0000 7fa75affd700 1 --2- 172.21.15.40:0/95884552 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7fa7440a11a0 0x7fa7440a3650 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fa750005fd0 tx=0x7fa75000a560 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:15:54.303 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:54.301+0000 7fa758ff9700 1 -- 172.21.15.40:0/95884552 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fa74c0978e0 con 0x7fa75c0feb10 2023-07-14T19:15:54.471 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:15:54 smithi040 ceph-mon[96518]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T19:15:54.471 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:15:54 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:15:54.471 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:15:54 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:15:54.472 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:15:54 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:15:54.472 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:15:54 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:15:54.472 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:15:54 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:15:54.472 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:15:54 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T19:15:54.472 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:15:54 smithi040 ceph-mon[96518]: mds.? [v2:172.21.15.130:6838/2145479535,v1:172.21.15.130:6839/2145479535] up:boot 2023-07-14T19:15:54.472 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:15:54 smithi040 ceph-mon[96518]: fsmap 12 up:standby 2023-07-14T19:15:54.473 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:15:54 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2023-07-14T19:15:54.473 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:15:54 smithi040 ceph-mon[96518]: from='client.? 172.21.15.40:0/409937961' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T19:15:54.473 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:15:54 smithi191 ceph-mon[98311]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T19:15:54.473 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:15:54 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:15:54.474 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:15:54 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:15:54.474 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:15:54 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:15:54.474 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:15:54 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:15:54.474 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:15:54 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:15:54.474 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:15:54 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T19:15:54.474 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:15:54 smithi191 ceph-mon[98311]: mds.? [v2:172.21.15.130:6838/2145479535,v1:172.21.15.130:6839/2145479535] up:boot 2023-07-14T19:15:54.475 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:15:54 smithi191 ceph-mon[98311]: fsmap 12 up:standby 2023-07-14T19:15:54.475 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:15:54 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2023-07-14T19:15:54.475 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:15:54 smithi191 ceph-mon[98311]: from='client.? 172.21.15.40:0/409937961' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T19:15:54.492 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:15:54 smithi130 ceph-mon[97832]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T19:15:54.493 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:15:54 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:15:54.493 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:15:54 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:15:54.493 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:15:54 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:15:54.493 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:15:54 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:15:54.494 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:15:54 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:15:54.494 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:15:54 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T19:15:54.494 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:15:54 smithi130 ceph-mon[97832]: mds.? [v2:172.21.15.130:6838/2145479535,v1:172.21.15.130:6839/2145479535] up:boot 2023-07-14T19:15:54.495 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:15:54 smithi130 ceph-mon[97832]: fsmap 12 up:standby 2023-07-14T19:15:54.495 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:15:54 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2023-07-14T19:15:54.495 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:15:54 smithi130 ceph-mon[97832]: from='client.? 172.21.15.40:0/409937961' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T19:15:54.511 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:54.509+0000 7fa761e56700 1 -- 172.21.15.40:0/95884552 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa75c02c8e0 con 0x7fa75c0feb10 2023-07-14T19:15:54.512 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:54.510+0000 7fa758ff9700 1 -- 172.21.15.40:0/95884552 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v77) v1 ==== 74+0+18424 (secure 0 0 0) 0x7fa74c09af00 con 0x7fa75c0feb10 2023-07-14T19:15:54.513 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-07-14T19:15:54.513 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":77,"fsid":"adc52f04-2279-11ee-9b33-001a4aab830c","created":"2023-07-14T19:08:56.893730+0000","modified":"2023-07-14T19:15:17.644846+0000","last_up_change":"2023-07-14T19:15:16.632979+0000","last_in_change":"2023-07-14T19:14:58.961702+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-07-14T19:12:03.271333+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"0d79356a-95d2-49ad-857b-2092ae9f7e13","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":3599703125},{"type":"v1","addr":"172.21.15.40:6803","nonce":3599703125}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":3599703125},{"type":"v1","addr":"172.21.15.40:6805","nonce":3599703125}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":3599703125},{"type":"v1","addr":"172.21.15.40:6809","nonce":3599703125}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":3599703125},{"type":"v1","addr":"172.21.15.40:6807","nonce":3599703125}]},"public_addr":"172.21.15.40:6803/3599703125","cluster_addr":"172.21.15.40:6805/3599703125","heartbeat_back_addr":"172.21.15.40:6809/3599703125","heartbeat_front_addr":"172.21.15.40:6807/3599703125","state":["exists","up"]},{"osd":1,"uuid":"40f8a3d1-a5e4-473c-b9f2-7600ec22be0d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6800","nonce":2512760702},{"type":"v1","addr":"172.21.15.130:6801","nonce":2512760702}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6802","nonce":2512760702},{"type":"v1","addr":"172.21.15.130:6803","nonce":2512760702}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6806","nonce":2512760702},{"type":"v1","addr":"172.21.15.130:6807","nonce":2512760702}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6804","nonce":2512760702},{"type":"v1","addr":"172.21.15.130:6805","nonce":2512760702}]},"public_addr":"172.21.15.130:6801/2512760702","cluster_addr":"172.21.15.130:6803/2512760702","heartbeat_back_addr":"172.21.15.130:6807/2512760702","heartbeat_front_addr":"172.21.15.130:6805/2512760702","state":["exists","up"]},{"osd":2,"uuid":"5a157891-cb7a-4ea6-9239-f48a7c67c003","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6800","nonce":589569135},{"type":"v1","addr":"172.21.15.191:6801","nonce":589569135}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6802","nonce":589569135},{"type":"v1","addr":"172.21.15.191:6803","nonce":589569135}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6806","nonce":589569135},{"type":"v1","addr":"172.21.15.191:6807","nonce":589569135}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6804","nonce":589569135},{"type":"v1","addr":"172.21.15.191:6805","nonce":589569135}]},"public_addr":"172.21.15.191:6801/589569135","cluster_addr":"172.21.15.191:6803/589569135","heartbeat_back_addr":"172.21.15.191:6807/589569135","heartbeat_front_addr":"172.21.15.191:6805/589569135","state":["exists","up"]},{"osd":3,"uuid":"8020e7dc-ac40-487f-857a-8da40bccb459","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":996287860},{"type":"v1","addr":"172.21.15.40:6811","nonce":996287860}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6812","nonce":996287860},{"type":"v1","addr":"172.21.15.40:6813","nonce":996287860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6816","nonce":996287860},{"type":"v1","addr":"172.21.15.40:6817","nonce":996287860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":996287860},{"type":"v1","addr":"172.21.15.40:6815","nonce":996287860}]},"public_addr":"172.21.15.40:6811/996287860","cluster_addr":"172.21.15.40:6813/996287860","heartbeat_back_addr":"172.21.15.40:6817/996287860","heartbeat_front_addr":"172.21.15.40:6815/996287860","state":["exists","up"]},{"osd":4,"uuid":"c5501b35-f62d-482d-8d22-115c88f0298e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6808","nonce":3507448720},{"type":"v1","addr":"172.21.15.130:6809","nonce":3507448720}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6810","nonce":3507448720},{"type":"v1","addr":"172.21.15.130:6811","nonce":3507448720}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6814","nonce":3507448720},{"type":"v1","addr":"172.21.15.130:6815","nonce":3507448720}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6812","nonce":3507448720},{"type":"v1","addr":"172.21.15.130:6813","nonce":3507448720}]},"public_addr":"172.21.15.130:6809/3507448720","cluster_addr":"172.21.15.130:6811/3507448720","heartbeat_back_addr":"172.21.15.130:6815/3507448720","heartbeat_front_addr":"172.21.15.130:6813/3507448720","state":["exists","up"]},{"osd":5,"uuid":"7770b6d6-5874-493a-986a-84db482cee42","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6808","nonce":1781944663},{"type":"v1","addr":"172.21.15.191:6809","nonce":1781944663}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6810","nonce":1781944663},{"type":"v1","addr":"172.21.15.191:6811","nonce":1781944663}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6814","nonce":1781944663},{"type":"v1","addr":"172.21.15.191:6815","nonce":1781944663}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6812","nonce":1781944663},{"type":"v1","addr":"172.21.15.191:6813","nonce":1781944663}]},"public_addr":"172.21.15.191:6809/1781944663","cluster_addr":"172.21.15.191:6811/1781944663","heartbeat_back_addr":"172.21.15.191:6815/1781944663","heartbeat_front_addr":"172.21.15.191:6813/1781944663","state":["exists","up"]},{"osd":6,"uuid":"ef65ad00-dfee-4300-9961-dfc8573aab06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6818","nonce":693186773},{"type":"v1","addr":"172.21.15.40:6819","nonce":693186773}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6820","nonce":693186773},{"type":"v1","addr":"172.21.15.40:6821","nonce":693186773}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6824","nonce":693186773},{"type":"v1","addr":"172.21.15.40:6825","nonce":693186773}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6822","nonce":693186773},{"type":"v1","addr":"172.21.15.40:6823","nonce":693186773}]},"public_addr":"172.21.15.40:6819/693186773","cluster_addr":"172.21.15.40:6821/693186773","heartbeat_back_addr":"172.21.15.40:6825/693186773","heartbeat_front_addr":"172.21.15.40:6823/693186773","state":["exists","up"]},{"osd":7,"uuid":"1700fbfa-cbfc-4f76-97be-727a3b100740","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":50,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6816","nonce":2384019528},{"type":"v1","addr":"172.21.15.130:6817","nonce":2384019528}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6818","nonce":2384019528},{"type":"v1","addr":"172.21.15.130:6819","nonce":2384019528}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6822","nonce":2384019528},{"type":"v1","addr":"172.21.15.130:6823","nonce":2384019528}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6820","nonce":2384019528},{"type":"v1","addr":"172.21.15.130:6821","nonce":2384019528}]},"public_addr":"172.21.15.130:6817/2384019528","cluster_addr":"172.21.15.130:6819/2384019528","heartbeat_back_addr":"172.21.15.130:6823/2384019528","heartbeat_front_addr":"172.21.15.130:6821/2384019528","state":["exists","up"]},{"osd":8,"uuid":"b76d4033-c1e2-457b-8571-c8a49fe443f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":57,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6816","nonce":2386818417},{"type":"v1","addr":"172.21.15.191:6817","nonce":2386818417}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6818","nonce":2386818417},{"type":"v1","addr":"172.21.15.191:6819","nonce":2386818417}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6822","nonce":2386818417},{"type":"v1","addr":"172.21.15.191:6823","nonce":2386818417}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6820","nonce":2386818417},{"type":"v1","addr":"172.21.15.191:6821","nonce":2386818417}]},"public_addr":"172.21.15.191:6817/2386818417","cluster_addr":"172.21.15.191:6819/2386818417","heartbeat_back_addr":"172.21.15.191:6823/2386818417","heartbeat_front_addr":"172.21.15.191:6821/2386818417","state":["exists","up"]},{"osd":9,"uuid":"df31bf2f-e30e-41b6-b2b9-0e26a07a88bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":64,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6826","nonce":2228380263},{"type":"v1","addr":"172.21.15.40:6827","nonce":2228380263}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6828","nonce":2228380263},{"type":"v1","addr":"172.21.15.40:6829","nonce":2228380263}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":2228380263},{"type":"v1","addr":"172.21.15.40:6833","nonce":2228380263}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6830","nonce":2228380263},{"type":"v1","addr":"172.21.15.40:6831","nonce":2228380263}]},"public_addr":"172.21.15.40:6827/2228380263","cluster_addr":"172.21.15.40:6829/2228380263","heartbeat_back_addr":"172.21.15.40:6833/2228380263","heartbeat_front_addr":"172.21.15.40:6831/2228380263","state":["exists","up"]},{"osd":10,"uuid":"4a93d621-49ee-4ad7-9e49-750614faf912","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6824","nonce":2417067798},{"type":"v1","addr":"172.21.15.130:6825","nonce":2417067798}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6826","nonce":2417067798},{"type":"v1","addr":"172.21.15.130:6827","nonce":2417067798}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6830","nonce":2417067798},{"type":"v1","addr":"172.21.15.130:6831","nonce":2417067798}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6828","nonce":2417067798},{"type":"v1","addr":"172.21.15.130:6829","nonce":2417067798}]},"public_addr":"172.21.15.130:6825/2417067798","cluster_addr":"172.21.15.130:6827/2417067798","heartbeat_back_addr":"172.21.15.130:6831/2417067798","heartbeat_front_addr":"172.21.15.130:6829/2417067798","state":["exists","up"]},{"osd":11,"uuid":"f43a2a6f-82f3-42ac-b77c-4f1b34da6918","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":76,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6824","nonce":2886059446},{"type":"v1","addr":"172.21.15.191:6825","nonce":2886059446}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6826","nonce":2886059446},{"type":"v1","addr":"172.21.15.191:6827","nonce":2886059446}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6830","nonce":2886059446},{"type":"v1","addr":"172.21.15.191:6831","nonce":2886059446}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6828","nonce":2886059446},{"type":"v1","addr":"172.21.15.191:6829","nonce":2886059446}]},"public_addr":"172.21.15.191:6825/2886059446","cluster_addr":"172.21.15.191:6827/2886059446","heartbeat_back_addr":"172.21.15.191:6831/2886059446","heartbeat_front_addr":"172.21.15.191:6829/2886059446","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:11:18.320495+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:11:39.901199+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:12:00.869042+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:12:23.292742+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:12:44.587019+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:13:06.494893+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:13:28.378374+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:13:50.033491+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:14:10.937062+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:14:32.358498+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:14:53.666584+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:15:14.837566+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.40:6801/3698757264":"2023-07-15T19:09:50.732458+0000","172.21.15.40:6800/3698757264":"2023-07-15T19:09:50.732458+0000","172.21.15.40:6800/2569015667":"2023-07-15T19:09:24.441543+0000","172.21.15.40:0/3653961003":"2023-07-15T19:09:24.441543+0000","172.21.15.40:0/3568861680":"2023-07-15T19:09:24.441543+0000","172.21.15.40:0/3249773172":"2023-07-15T19:09:50.732458+0000","172.21.15.40:0/4084215480":"2023-07-15T19:09:24.441543+0000","172.21.15.40:0/1163941760":"2023-07-15T19:09:50.732458+0000","172.21.15.40:6801/2569015667":"2023-07-15T19:09:24.441543+0000","172.21.15.40:0/2326639960":"2023-07-15T19:09:50.732458+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-07-14T19:15:54.514 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:54.512+0000 7fa761e56700 1 -- 172.21.15.40:0/95884552 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7fa7440a11a0 msgr2=0x7fa7440a3650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:15:54.514 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:54.512+0000 7fa761e56700 1 --2- 172.21.15.40:0/95884552 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7fa7440a11a0 0x7fa7440a3650 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fa750005fd0 tx=0x7fa75000a560 comp rx=0 tx=0).stop 2023-07-14T19:15:54.514 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:54.512+0000 7fa761e56700 1 -- 172.21.15.40:0/95884552 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa75c0feb10 msgr2=0x7fa75c0695f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:15:54.515 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:54.512+0000 7fa761e56700 1 --2- 172.21.15.40:0/95884552 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa75c0feb10 0x7fa75c0695f0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fa74c0098e0 tx=0x7fa74c005e80 comp rx=0 tx=0).stop 2023-07-14T19:15:54.515 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:54.512+0000 7fa761e56700 1 -- 172.21.15.40:0/95884552 shutdown_connections 2023-07-14T19:15:54.515 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:54.512+0000 7fa761e56700 1 --2- 172.21.15.40:0/95884552 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7fa7440a11a0 0x7fa7440a3650 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:15:54.515 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:54.512+0000 7fa761e56700 1 --2- 172.21.15.40:0/95884552 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa75c100690 0x7fa75c19ccb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:15:54.515 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:54.512+0000 7fa761e56700 1 --2- 172.21.15.40:0/95884552 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa75c0feb10 0x7fa75c0695f0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:15:54.515 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:54.512+0000 7fa761e56700 1 --2- 172.21.15.40:0/95884552 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa75c0ffd00 0x7fa75c19a620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:15:54.516 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:54.512+0000 7fa761e56700 1 -- 172.21.15.40:0/95884552 >> 172.21.15.40:0/95884552 conn(0x7fa75c0fa130 msgr2=0x7fa75c103190 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:15:54.516 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:54.513+0000 7fa761e56700 1 -- 172.21.15.40:0/95884552 shutdown_connections 2023-07-14T19:15:54.516 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:54.513+0000 7fa761e56700 1 -- 172.21.15.40:0/95884552 wait complete. 2023-07-14T19:15:54.528 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-07-14T19:12:03.271333+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-07-14T19:15:54.529 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-07-14T19:15:54.672 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:54.671+0000 7f9620d9d700 1 -- 172.21.15.40:0/326272050 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f961c100770 msgr2=0x7f961c107010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:15:54.673 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:54.671+0000 7f9620d9d700 1 --2- 172.21.15.40:0/326272050 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f961c100770 0x7f961c107010 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f961400b0a0 tx=0x7f961400b3b0 comp rx=0 tx=0).stop 2023-07-14T19:15:54.673 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:54.671+0000 7f9620d9d700 1 -- 172.21.15.40:0/326272050 shutdown_connections 2023-07-14T19:15:54.673 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:54.671+0000 7f9620d9d700 1 --2- 172.21.15.40:0/326272050 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f961c0feb30 0x7f961c0fef40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:15:54.673 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:54.671+0000 7f9620d9d700 1 --2- 172.21.15.40:0/326272050 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f961c100770 0x7f961c107010 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:15:54.673 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:54.671+0000 7f9620d9d700 1 --2- 172.21.15.40:0/326272050 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f961c0ffdc0 0x7f961c100230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:15:54.674 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:54.671+0000 7f9620d9d700 1 -- 172.21.15.40:0/326272050 >> 172.21.15.40:0/326272050 conn(0x7f961c0fa150 msgr2=0x7f961c0fc580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:15:54.674 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:54.671+0000 7f9620d9d700 1 -- 172.21.15.40:0/326272050 shutdown_connections 2023-07-14T19:15:54.674 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:54.671+0000 7f9620d9d700 1 -- 172.21.15.40:0/326272050 wait complete. 2023-07-14T19:15:54.675 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:54.672+0000 7f9620d9d700 1 Processor -- start 2023-07-14T19:15:54.675 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:54.672+0000 7f9620d9d700 1 -- start start 2023-07-14T19:15:54.676 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:54.672+0000 7f9620d9d700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f961c0feb30 0x7f961c194920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:15:54.676 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:54.672+0000 7f9620d9d700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f961c0ffdc0 0x7f961c194e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:15:54.676 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:54.672+0000 7f9620d9d700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f961c100770 0x7f961c068950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:15:54.676 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:54.672+0000 7f9620d9d700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f961c068f20 con 0x7f961c0feb30 2023-07-14T19:15:54.676 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:54.672+0000 7f9620d9d700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f961c069060 con 0x7f961c0ffdc0 2023-07-14T19:15:54.676 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:54.672+0000 7f9620d9d700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f961c0691f0 con 0x7f961c100770 2023-07-14T19:15:54.677 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:54.672+0000 7f9619d9b700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f961c0ffdc0 0x7f961c194e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:15:54.677 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:54.672+0000 7f9619d9b700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f961c0ffdc0 0x7f961c194e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.40:47938/0 (socket says 172.21.15.40:47938) 2023-07-14T19:15:54.677 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:54.672+0000 7f9619d9b700 1 -- 172.21.15.40:0/374192024 learned_addr learned my addr 172.21.15.40:0/374192024 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-07-14T19:15:54.677 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:54.673+0000 7f9619d9b700 1 -- 172.21.15.40:0/374192024 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f961c100770 msgr2=0x7f961c068950 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2023-07-14T19:15:54.677 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:54.673+0000 7f9619d9b700 1 --2- 172.21.15.40:0/374192024 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f961c100770 0x7f961c068950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:15:54.677 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:54.673+0000 7f961a59c700 1 --2- 172.21.15.40:0/374192024 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f961c0feb30 0x7f961c194920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:15:54.677 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:54.673+0000 7f9619d9b700 1 -- 172.21.15.40:0/374192024 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f961c0feb30 msgr2=0x7f961c194920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:15:54.678 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:54.673+0000 7f9619d9b700 1 --2- 172.21.15.40:0/374192024 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f961c0feb30 0x7f961c194920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:15:54.678 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:54.673+0000 7f9619d9b700 1 -- 172.21.15.40:0/374192024 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9614009d00 con 0x7f961c0ffdc0 2023-07-14T19:15:54.678 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:54.673+0000 7f961a59c700 1 --2- 172.21.15.40:0/374192024 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f961c0feb30 0x7f961c194920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-14T19:15:54.678 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:54.674+0000 7f9619d9b700 1 --2- 172.21.15.40:0/374192024 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f961c0ffdc0 0x7f961c194e60 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f961000b570 tx=0x7f961000b930 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:15:54.679 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:54.674+0000 7f960b7fe700 1 -- 172.21.15.40:0/374192024 <== mon.2 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9610019070 con 0x7f961c0ffdc0 2023-07-14T19:15:54.679 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:54.674+0000 7f9620d9d700 1 -- 172.21.15.40:0/374192024 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f961c0694d0 con 0x7f961c0ffdc0 2023-07-14T19:15:54.680 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:54.674+0000 7f960b7fe700 1 -- 172.21.15.40:0/374192024 <== mon.2 v2:172.21.15.130:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f961001b9e0 con 0x7f961c0ffdc0 2023-07-14T19:15:54.680 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:54.674+0000 7f960b7fe700 1 -- 172.21.15.40:0/374192024 <== mon.2 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f961001da70 con 0x7f961c0ffdc0 2023-07-14T19:15:54.680 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:54.674+0000 7f9620d9d700 1 -- 172.21.15.40:0/374192024 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f961c199ec0 con 0x7f961c0ffdc0 2023-07-14T19:15:54.680 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:54.675+0000 7f9620d9d700 1 -- 172.21.15.40:0/374192024 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f961c105a60 con 0x7f961c0ffdc0 2023-07-14T19:15:54.680 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:54.676+0000 7f960b7fe700 1 -- 172.21.15.40:0/374192024 <== mon.2 v2:172.21.15.130:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f961001dbd0 con 0x7f961c0ffdc0 2023-07-14T19:15:54.680 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:54.677+0000 7f960b7fe700 1 --2- 172.21.15.40:0/374192024 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f96040a1150 0x7f96040a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:15:54.681 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:54.677+0000 7f960b7fe700 1 -- 172.21.15.40:0/374192024 <== mon.2 v2:172.21.15.130:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f96100d4210 con 0x7f961c0ffdc0 2023-07-14T19:15:54.681 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:54.677+0000 7f961a59c700 1 --2- 172.21.15.40:0/374192024 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f96040a1150 0x7f96040a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:15:54.681 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:54.677+0000 7f961a59c700 1 --2- 172.21.15.40:0/374192024 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f96040a1150 0x7f96040a3600 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f960c009be0 tx=0x7f960c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:15:54.683 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:54.682+0000 7f960b7fe700 1 -- 172.21.15.40:0/374192024 <== mon.2 v2:172.21.15.130:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f9610057780 con 0x7f961c0ffdc0 2023-07-14T19:15:54.894 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:54.892+0000 7f9620d9d700 1 -- 172.21.15.40:0/374192024 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f961c069660 con 0x7f961c0ffdc0 2023-07-14T19:15:54.894 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:54.893+0000 7f960b7fe700 1 -- 172.21.15.40:0/374192024 <== mon.2 v2:172.21.15.130:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v77) v1 ==== 93+0+10 (secure 0 0 0) 0x7f961c069660 con 0x7f961c0ffdc0 2023-07-14T19:15:54.895 INFO:teuthology.orchestra.run.smithi040.stdout:pg_num: 1 2023-07-14T19:15:54.897 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:54.895+0000 7f9620d9d700 1 -- 172.21.15.40:0/374192024 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f96040a1150 msgr2=0x7f96040a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:15:54.897 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:54.895+0000 7f9620d9d700 1 --2- 172.21.15.40:0/374192024 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f96040a1150 0x7f96040a3600 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f960c009be0 tx=0x7f960c009260 comp rx=0 tx=0).stop 2023-07-14T19:15:54.897 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:54.895+0000 7f9620d9d700 1 -- 172.21.15.40:0/374192024 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f961c0ffdc0 msgr2=0x7f961c194e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:15:54.897 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:54.895+0000 7f9620d9d700 1 --2- 172.21.15.40:0/374192024 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f961c0ffdc0 0x7f961c194e60 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f961000b570 tx=0x7f961000b930 comp rx=0 tx=0).stop 2023-07-14T19:15:54.897 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:54.895+0000 7f9620d9d700 1 -- 172.21.15.40:0/374192024 shutdown_connections 2023-07-14T19:15:54.897 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:54.895+0000 7f9620d9d700 1 --2- 172.21.15.40:0/374192024 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f96040a1150 0x7f96040a3600 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:15:54.898 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:54.895+0000 7f9620d9d700 1 --2- 172.21.15.40:0/374192024 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f961c100770 0x7f961c068950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:15:54.898 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:54.895+0000 7f9620d9d700 1 --2- 172.21.15.40:0/374192024 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f961c0feb30 0x7f961c194920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:15:54.898 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:54.895+0000 7f9620d9d700 1 --2- 172.21.15.40:0/374192024 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f961c0ffdc0 0x7f961c194e60 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:15:54.898 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:54.895+0000 7f9620d9d700 1 -- 172.21.15.40:0/374192024 >> 172.21.15.40:0/374192024 conn(0x7f961c0fa150 msgr2=0x7f961c0fc310 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:15:54.898 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:54.895+0000 7f9620d9d700 1 -- 172.21.15.40:0/374192024 shutdown_connections 2023-07-14T19:15:54.898 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:54.895+0000 7f9620d9d700 1 -- 172.21.15.40:0/374192024 wait complete. 2023-07-14T19:15:54.909 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2023-07-14T19:15:55.041 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:55.039+0000 7fcfd6bc4700 1 -- 172.21.15.40:0/401524335 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcfd00fdec0 msgr2=0x7fcfd00fe2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:15:55.041 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:55.039+0000 7fcfd6bc4700 1 --2- 172.21.15.40:0/401524335 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcfd00fdec0 0x7fcfd00fe2d0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fcfc4009990 tx=0x7fcfc4009ca0 comp rx=0 tx=0).stop 2023-07-14T19:15:55.041 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:55.039+0000 7fcfd6bc4700 1 -- 172.21.15.40:0/401524335 shutdown_connections 2023-07-14T19:15:55.042 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:55.039+0000 7fcfd6bc4700 1 --2- 172.21.15.40:0/401524335 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fcfd01018c0 0x7fcfd0108180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:15:55.042 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:55.039+0000 7fcfd6bc4700 1 --2- 172.21.15.40:0/401524335 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcfd00fdec0 0x7fcfd00fe2d0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:15:55.042 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:55.039+0000 7fcfd6bc4700 1 --2- 172.21.15.40:0/401524335 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fcfd0100f30 0x7fcfd0101380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:15:55.042 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:55.039+0000 7fcfd6bc4700 1 -- 172.21.15.40:0/401524335 >> 172.21.15.40:0/401524335 conn(0x7fcfd00f98a0 msgr2=0x7fcfd00fbcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:15:55.042 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:55.039+0000 7fcfd6bc4700 1 -- 172.21.15.40:0/401524335 shutdown_connections 2023-07-14T19:15:55.042 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:55.039+0000 7fcfd6bc4700 1 -- 172.21.15.40:0/401524335 wait complete. 2023-07-14T19:15:55.042 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:55.039+0000 7fcfd6bc4700 1 Processor -- start 2023-07-14T19:15:55.043 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:55.039+0000 7fcfd6bc4700 1 -- start start 2023-07-14T19:15:55.044 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:55.040+0000 7fcfd6bc4700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fcfd00fdec0 0x7fcfd01958f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:15:55.044 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:55.040+0000 7fcfd6bc4700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fcfd0100f30 0x7fcfd0195e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:15:55.045 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:55.040+0000 7fcfd6bc4700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcfd01018c0 0x7fcfd019cea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:15:55.045 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:55.040+0000 7fcfd6bc4700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfd019d470 con 0x7fcfd01018c0 2023-07-14T19:15:55.045 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:55.040+0000 7fcfd6bc4700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfd019d5b0 con 0x7fcfd00fdec0 2023-07-14T19:15:55.045 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:55.040+0000 7fcfd6bc4700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfd019d740 con 0x7fcfd0100f30 2023-07-14T19:15:55.045 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:55.040+0000 7fcfd5161700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcfd01018c0 0x7fcfd019cea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:15:55.045 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:55.040+0000 7fcfd4960700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fcfd00fdec0 0x7fcfd01958f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:15:55.045 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:55.040+0000 7fcfd5161700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcfd01018c0 0x7fcfd019cea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:32798/0 (socket says 172.21.15.40:32798) 2023-07-14T19:15:55.045 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:55.040+0000 7fcfd5161700 1 -- 172.21.15.40:0/3637734798 learned_addr learned my addr 172.21.15.40:0/3637734798 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-07-14T19:15:55.046 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:55.040+0000 7fcfcffff700 1 --2- 172.21.15.40:0/3637734798 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fcfd0100f30 0x7fcfd0195e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:15:55.046 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:55.040+0000 7fcfd5161700 1 -- 172.21.15.40:0/3637734798 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fcfd0100f30 msgr2=0x7fcfd0195e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:15:55.046 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:55.040+0000 7fcfd5161700 1 --2- 172.21.15.40:0/3637734798 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fcfd0100f30 0x7fcfd0195e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:15:55.046 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:55.040+0000 7fcfd5161700 1 -- 172.21.15.40:0/3637734798 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fcfd00fdec0 msgr2=0x7fcfd01958f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:15:55.046 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:55.040+0000 7fcfd5161700 1 --2- 172.21.15.40:0/3637734798 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fcfd00fdec0 0x7fcfd01958f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:15:55.046 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:55.040+0000 7fcfd5161700 1 -- 172.21.15.40:0/3637734798 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcfc40095f0 con 0x7fcfd01018c0 2023-07-14T19:15:55.046 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:55.040+0000 7fcfd5161700 1 --2- 172.21.15.40:0/3637734798 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcfd01018c0 0x7fcfd019cea0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fcfc800ef30 tx=0x7fcfc800c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:15:55.047 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:55.040+0000 7fcfcdffb700 1 -- 172.21.15.40:0/3637734798 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfc8018070 con 0x7fcfd01018c0 2023-07-14T19:15:55.047 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:55.040+0000 7fcfcdffb700 1 -- 172.21.15.40:0/3637734798 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fcfc800f040 con 0x7fcfd01018c0 2023-07-14T19:15:55.047 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:55.040+0000 7fcfd6bc4700 1 -- 172.21.15.40:0/3637734798 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcfd019d9c0 con 0x7fcfd01018c0 2023-07-14T19:15:55.047 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:55.040+0000 7fcfcdffb700 1 -- 172.21.15.40:0/3637734798 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfc800cd00 con 0x7fcfd01018c0 2023-07-14T19:15:55.047 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:55.041+0000 7fcfd6bc4700 1 -- 172.21.15.40:0/3637734798 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcfd019ded0 con 0x7fcfd01018c0 2023-07-14T19:15:55.047 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:55.042+0000 7fcfd6bc4700 1 -- 172.21.15.40:0/3637734798 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcfd010b3d0 con 0x7fcfd01018c0 2023-07-14T19:15:55.047 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:55.042+0000 7fcfcdffb700 1 -- 172.21.15.40:0/3637734798 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7fcfc8003a00 con 0x7fcfd01018c0 2023-07-14T19:15:55.048 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:55.043+0000 7fcfcdffb700 1 --2- 172.21.15.40:0/3637734798 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7fcfc00a1220 0x7fcfc00a36d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:15:55.048 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:55.043+0000 7fcfcdffb700 1 -- 172.21.15.40:0/3637734798 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fcfc80cbf30 con 0x7fcfd01018c0 2023-07-14T19:15:55.048 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:55.043+0000 7fcfd4960700 1 --2- 172.21.15.40:0/3637734798 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7fcfc00a1220 0x7fcfc00a36d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:15:55.048 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:55.044+0000 7fcfd4960700 1 --2- 172.21.15.40:0/3637734798 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7fcfc00a1220 0x7fcfc00a36d0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fcfc4006010 tx=0x7fcfc4005960 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:15:55.049 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:55.047+0000 7fcfcdffb700 1 -- 172.21.15.40:0/3637734798 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fcfc8095bb0 con 0x7fcfd01018c0 2023-07-14T19:15:55.253 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:55.252+0000 7fcfd6bc4700 1 -- 172.21.15.40:0/3637734798 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7fcfd0106c20 con 0x7fcfd01018c0 2023-07-14T19:15:55.254 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:55.252+0000 7fcfcdffb700 1 -- 172.21.15.40:0/3637734798 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v77) v1 ==== 84+0+65 (secure 0 0 0) 0x7fcfc80991d0 con 0x7fcfd01018c0 2023-07-14T19:15:55.254 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-07-14T19:15:55.254 INFO:teuthology.orchestra.run.smithi040.stdout:[ 2023-07-14T19:15:55.255 INFO:teuthology.orchestra.run.smithi040.stdout: { 2023-07-14T19:15:55.255 INFO:teuthology.orchestra.run.smithi040.stdout: "poolnum": 1, 2023-07-14T19:15:55.255 INFO:teuthology.orchestra.run.smithi040.stdout: "poolname": ".mgr" 2023-07-14T19:15:55.255 INFO:teuthology.orchestra.run.smithi040.stdout: } 2023-07-14T19:15:55.255 INFO:teuthology.orchestra.run.smithi040.stdout:] 2023-07-14T19:15:55.256 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:55.254+0000 7fcfd6bc4700 1 -- 172.21.15.40:0/3637734798 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7fcfc00a1220 msgr2=0x7fcfc00a36d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:15:55.256 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:55.254+0000 7fcfd6bc4700 1 --2- 172.21.15.40:0/3637734798 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7fcfc00a1220 0x7fcfc00a36d0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fcfc4006010 tx=0x7fcfc4005960 comp rx=0 tx=0).stop 2023-07-14T19:15:55.256 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:55.254+0000 7fcfd6bc4700 1 -- 172.21.15.40:0/3637734798 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcfd01018c0 msgr2=0x7fcfd019cea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:15:55.257 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:55.254+0000 7fcfd6bc4700 1 --2- 172.21.15.40:0/3637734798 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcfd01018c0 0x7fcfd019cea0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fcfc800ef30 tx=0x7fcfc800c3a0 comp rx=0 tx=0).stop 2023-07-14T19:15:55.257 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:55.254+0000 7fcfd6bc4700 1 -- 172.21.15.40:0/3637734798 shutdown_connections 2023-07-14T19:15:55.257 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:55.254+0000 7fcfd6bc4700 1 --2- 172.21.15.40:0/3637734798 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7fcfc00a1220 0x7fcfc00a36d0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:15:55.257 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:55.254+0000 7fcfd6bc4700 1 --2- 172.21.15.40:0/3637734798 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcfd01018c0 0x7fcfd019cea0 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:15:55.257 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:55.254+0000 7fcfd6bc4700 1 --2- 172.21.15.40:0/3637734798 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fcfd00fdec0 0x7fcfd01958f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:15:55.257 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:55.254+0000 7fcfd6bc4700 1 --2- 172.21.15.40:0/3637734798 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fcfd0100f30 0x7fcfd0195e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:15:55.258 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:55.254+0000 7fcfd6bc4700 1 -- 172.21.15.40:0/3637734798 >> 172.21.15.40:0/3637734798 conn(0x7fcfd00f98a0 msgr2=0x7fcfd00fbad0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:15:55.258 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:55.254+0000 7fcfd6bc4700 1 -- 172.21.15.40:0/3637734798 shutdown_connections 2023-07-14T19:15:55.258 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:55.254+0000 7fcfd6bc4700 1 -- 172.21.15.40:0/3637734798 wait complete. 2023-07-14T19:15:55.268 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2023-07-14T19:15:55.269 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2023-07-14T19:15:55.408 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:55.406+0000 7f78f2172700 1 -- 172.21.15.40:0/1098283521 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f78ec0ffd00 msgr2=0x7f78ec100150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:15:55.409 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:55.406+0000 7f78f2172700 1 --2- 172.21.15.40:0/1098283521 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f78ec0ffd00 0x7f78ec100150 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f78d4009990 tx=0x7f78d4009ca0 comp rx=0 tx=0).stop 2023-07-14T19:15:55.409 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:55.406+0000 7f78f2172700 1 -- 172.21.15.40:0/1098283521 shutdown_connections 2023-07-14T19:15:55.409 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:55.406+0000 7f78f2172700 1 --2- 172.21.15.40:0/1098283521 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f78ec100690 0x7f78ec106f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:15:55.409 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:55.406+0000 7f78f2172700 1 --2- 172.21.15.40:0/1098283521 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f78ec0feb10 0x7f78ec0fef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:15:55.409 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:55.406+0000 7f78f2172700 1 --2- 172.21.15.40:0/1098283521 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f78ec0ffd00 0x7f78ec100150 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:15:55.410 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:55.406+0000 7f78f2172700 1 -- 172.21.15.40:0/1098283521 >> 172.21.15.40:0/1098283521 conn(0x7f78ec0fa130 msgr2=0x7f78ec0fc560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:15:55.410 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:55.406+0000 7f78f2172700 1 -- 172.21.15.40:0/1098283521 shutdown_connections 2023-07-14T19:15:55.410 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:55.406+0000 7f78f2172700 1 -- 172.21.15.40:0/1098283521 wait complete. 2023-07-14T19:15:55.410 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:55.407+0000 7f78f2172700 1 Processor -- start 2023-07-14T19:15:55.412 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:55.407+0000 7f78f2172700 1 -- start start 2023-07-14T19:15:55.413 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:55.407+0000 7f78f2172700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f78ec0feb10 0x7f78ec068330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:15:55.413 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:55.407+0000 7f78f2172700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f78ec0ffd00 0x7f78ec19a6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:15:55.413 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:55.407+0000 7f78f2172700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f78ec100690 0x7f78ec19cd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:15:55.413 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:55.407+0000 7f78f2172700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78ec19d4e0 con 0x7f78ec100690 2023-07-14T19:15:55.413 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:55.407+0000 7f78f2172700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78ec19d650 con 0x7f78ec0feb10 2023-07-14T19:15:55.413 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:55.407+0000 7f78f2172700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78ec19d810 con 0x7f78ec0ffd00 2023-07-14T19:15:55.414 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:55.407+0000 7f78ebfff700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f78ec100690 0x7f78ec19cd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:15:55.414 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:55.407+0000 7f78eb7fe700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f78ec0feb10 0x7f78ec068330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:15:55.414 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:55.407+0000 7f78ebfff700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f78ec100690 0x7f78ec19cd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:32812/0 (socket says 172.21.15.40:32812) 2023-07-14T19:15:55.414 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:55.407+0000 7f78ebfff700 1 -- 172.21.15.40:0/2128360787 learned_addr learned my addr 172.21.15.40:0/2128360787 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-07-14T19:15:55.414 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:55.407+0000 7f78eaffd700 1 --2- 172.21.15.40:0/2128360787 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f78ec0ffd00 0x7f78ec19a6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:15:55.414 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:55.407+0000 7f78ebfff700 1 -- 172.21.15.40:0/2128360787 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f78ec0ffd00 msgr2=0x7f78ec19a6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:15:55.415 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:55.407+0000 7f78ebfff700 1 --2- 172.21.15.40:0/2128360787 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f78ec0ffd00 0x7f78ec19a6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:15:55.415 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:55.407+0000 7f78ebfff700 1 -- 172.21.15.40:0/2128360787 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f78ec0feb10 msgr2=0x7f78ec068330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:15:55.415 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:55.407+0000 7f78ebfff700 1 --2- 172.21.15.40:0/2128360787 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f78ec0feb10 0x7f78ec068330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:15:55.415 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:55.407+0000 7f78ebfff700 1 -- 172.21.15.40:0/2128360787 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78d40095f0 con 0x7f78ec100690 2023-07-14T19:15:55.415 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:55.408+0000 7f78ebfff700 1 --2- 172.21.15.40:0/2128360787 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f78ec100690 0x7f78ec19cd10 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f78e400ada0 tx=0x7f78e400c330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:15:55.415 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:55.408+0000 7f78e8ff9700 1 -- 172.21.15.40:0/2128360787 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78e400f070 con 0x7f78ec100690 2023-07-14T19:15:55.415 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:55.408+0000 7f78f2172700 1 -- 172.21.15.40:0/2128360787 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f78ec19daf0 con 0x7f78ec100690 2023-07-14T19:15:55.416 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:55.408+0000 7f78e8ff9700 1 -- 172.21.15.40:0/2128360787 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f78e4007b30 con 0x7f78ec100690 2023-07-14T19:15:55.416 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:55.408+0000 7f78e8ff9700 1 -- 172.21.15.40:0/2128360787 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78e40053e0 con 0x7f78ec100690 2023-07-14T19:15:55.416 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:55.408+0000 7f78f2172700 1 -- 172.21.15.40:0/2128360787 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f78ec19e050 con 0x7f78ec100690 2023-07-14T19:15:55.416 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:55.409+0000 7f78f2172700 1 -- 172.21.15.40:0/2128360787 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f78ec10a310 con 0x7f78ec100690 2023-07-14T19:15:55.416 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:55.410+0000 7f78e8ff9700 1 -- 172.21.15.40:0/2128360787 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f78e400c980 con 0x7f78ec100690 2023-07-14T19:15:55.417 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:55.411+0000 7f78e8ff9700 1 --2- 172.21.15.40:0/2128360787 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f78d80a1150 0x7f78d80a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:15:55.417 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:55.411+0000 7f78eb7fe700 1 --2- 172.21.15.40:0/2128360787 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f78d80a1150 0x7f78d80a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:15:55.417 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:55.411+0000 7f78e8ff9700 1 -- 172.21.15.40:0/2128360787 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f78e40cbfd0 con 0x7f78ec100690 2023-07-14T19:15:55.417 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:55.412+0000 7f78eb7fe700 1 --2- 172.21.15.40:0/2128360787 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f78d80a1150 0x7f78d80a3600 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f78ec0ffb60 tx=0x7f78dc007400 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:15:55.417 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:55.414+0000 7f78e8ff9700 1 -- 172.21.15.40:0/2128360787 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f78e4095c50 con 0x7f78ec100690 2023-07-14T19:15:55.471 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:15:55 smithi040 ceph-mon[96518]: from='client.? 172.21.15.40:0/95884552' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T19:15:55.471 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:15:55 smithi040 ceph-mon[96518]: from='client.? 172.21.15.40:0/374192024' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-07-14T19:15:55.472 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:15:55 smithi191 ceph-mon[98311]: from='client.? 172.21.15.40:0/95884552' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T19:15:55.472 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:15:55 smithi191 ceph-mon[98311]: from='client.? 172.21.15.40:0/374192024' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-07-14T19:15:55.492 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:15:55 smithi130 ceph-mon[97832]: from='client.? 172.21.15.40:0/95884552' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T19:15:55.493 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:15:55 smithi130 ceph-mon[97832]: from='client.? 172.21.15.40:0/374192024' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-07-14T19:15:55.633 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:55.632+0000 7f78f2172700 1 -- 172.21.15.40:0/2128360787 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f78ec105a10 con 0x7f78ec100690 2023-07-14T19:15:56.223 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:56.221+0000 7f78e8ff9700 1 -- 172.21.15.40:0/2128360787 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v78) v1 ==== 138+0+0 (secure 0 0 0) 0x7f78e4099270 con 0x7f78ec100690 2023-07-14T19:15:56.227 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:56.226+0000 7f78f2172700 1 -- 172.21.15.40:0/2128360787 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f78d80a1150 msgr2=0x7f78d80a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:15:56.227 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:56.226+0000 7f78f2172700 1 --2- 172.21.15.40:0/2128360787 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f78d80a1150 0x7f78d80a3600 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f78ec0ffb60 tx=0x7f78dc007400 comp rx=0 tx=0).stop 2023-07-14T19:15:56.228 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:56.226+0000 7f78f2172700 1 -- 172.21.15.40:0/2128360787 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f78ec100690 msgr2=0x7f78ec19cd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:15:56.228 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:56.226+0000 7f78f2172700 1 --2- 172.21.15.40:0/2128360787 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f78ec100690 0x7f78ec19cd10 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f78e400ada0 tx=0x7f78e400c330 comp rx=0 tx=0).stop 2023-07-14T19:15:56.228 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:56.226+0000 7f78f2172700 1 -- 172.21.15.40:0/2128360787 shutdown_connections 2023-07-14T19:15:56.229 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:56.226+0000 7f78f2172700 1 --2- 172.21.15.40:0/2128360787 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f78d80a1150 0x7f78d80a3600 secure :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f78ec0ffb60 tx=0x7f78dc007400 comp rx=0 tx=0).stop 2023-07-14T19:15:56.229 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:56.226+0000 7f78f2172700 1 --2- 172.21.15.40:0/2128360787 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f78ec100690 0x7f78ec19cd10 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:15:56.229 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:56.226+0000 7f78f2172700 1 --2- 172.21.15.40:0/2128360787 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f78ec0feb10 0x7f78ec068330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:15:56.229 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:56.226+0000 7f78f2172700 1 --2- 172.21.15.40:0/2128360787 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f78ec0ffd00 0x7f78ec19a6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:15:56.229 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:56.226+0000 7f78f2172700 1 -- 172.21.15.40:0/2128360787 >> 172.21.15.40:0/2128360787 conn(0x7f78ec0fa130 msgr2=0x7f78ec0fc380 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:15:56.229 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:56.226+0000 7f78f2172700 1 -- 172.21.15.40:0/2128360787 shutdown_connections 2023-07-14T19:15:56.229 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:56.226+0000 7f78f2172700 1 -- 172.21.15.40:0/2128360787 wait complete. 2023-07-14T19:15:56.230 INFO:teuthology.orchestra.run.smithi040.stderr:pool 'cephfs_metadata' created 2023-07-14T19:15:56.243 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2023-07-14T19:15:56.376 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:56.374+0000 7fc984c69700 1 -- 172.21.15.40:0/176179245 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc9800feb10 msgr2=0x7fc9800fef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:15:56.377 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:56.374+0000 7fc984c69700 1 --2- 172.21.15.40:0/176179245 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc9800feb10 0x7fc9800fef20 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fc970009990 tx=0x7fc970009ca0 comp rx=0 tx=0).stop 2023-07-14T19:15:56.377 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:56.374+0000 7fc984c69700 1 -- 172.21.15.40:0/176179245 shutdown_connections 2023-07-14T19:15:56.377 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:56.374+0000 7fc984c69700 1 --2- 172.21.15.40:0/176179245 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc980100690 0x7fc980106f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:15:56.377 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:56.374+0000 7fc984c69700 1 --2- 172.21.15.40:0/176179245 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc9800feb10 0x7fc9800fef20 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:15:56.377 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:56.374+0000 7fc984c69700 1 --2- 172.21.15.40:0/176179245 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc9800ffd00 0x7fc980100150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:15:56.377 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:56.374+0000 7fc984c69700 1 -- 172.21.15.40:0/176179245 >> 172.21.15.40:0/176179245 conn(0x7fc9800fa130 msgr2=0x7fc9800fc560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:15:56.378 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:56.374+0000 7fc984c69700 1 -- 172.21.15.40:0/176179245 shutdown_connections 2023-07-14T19:15:56.378 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:56.374+0000 7fc984c69700 1 -- 172.21.15.40:0/176179245 wait complete. 2023-07-14T19:15:56.380 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:56.375+0000 7fc984c69700 1 Processor -- start 2023-07-14T19:15:56.380 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:56.375+0000 7fc984c69700 1 -- start start 2023-07-14T19:15:56.380 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:56.375+0000 7fc984c69700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc9800feb10 0x7fc9801959d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:15:56.380 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:56.375+0000 7fc984c69700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc9800ffd00 0x7fc980195f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:15:56.381 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:56.375+0000 7fc984c69700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc980100690 0x7fc98019cf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:15:56.381 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:56.375+0000 7fc984c69700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc98019d550 con 0x7fc9800ffd00 2023-07-14T19:15:56.381 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:56.375+0000 7fc984c69700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc98019d690 con 0x7fc9800feb10 2023-07-14T19:15:56.381 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:56.375+0000 7fc984c69700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc98019d820 con 0x7fc980100690 2023-07-14T19:15:56.381 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:56.375+0000 7fc97dd9b700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc9800ffd00 0x7fc980195f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:15:56.382 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:56.376+0000 7fc97dd9b700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc9800ffd00 0x7fc980195f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:32824/0 (socket says 172.21.15.40:32824) 2023-07-14T19:15:56.382 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:56.376+0000 7fc97dd9b700 1 -- 172.21.15.40:0/1441159222 learned_addr learned my addr 172.21.15.40:0/1441159222 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-07-14T19:15:56.382 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:56.376+0000 7fc97e59c700 1 --2- 172.21.15.40:0/1441159222 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc9800feb10 0x7fc9801959d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:15:56.382 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:56.376+0000 7fc97ed9d700 1 --2- 172.21.15.40:0/1441159222 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc980100690 0x7fc98019cf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:15:56.382 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:56.376+0000 7fc97dd9b700 1 -- 172.21.15.40:0/1441159222 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc980100690 msgr2=0x7fc98019cf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:15:56.382 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:56.376+0000 7fc97dd9b700 1 --2- 172.21.15.40:0/1441159222 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc980100690 0x7fc98019cf80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:15:56.382 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:56.376+0000 7fc97dd9b700 1 -- 172.21.15.40:0/1441159222 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc9800feb10 msgr2=0x7fc9801959d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:15:56.382 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:56.376+0000 7fc97dd9b700 1 --2- 172.21.15.40:0/1441159222 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc9800feb10 0x7fc9801959d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:15:56.383 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:56.376+0000 7fc97dd9b700 1 -- 172.21.15.40:0/1441159222 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9700095f0 con 0x7fc9800ffd00 2023-07-14T19:15:56.383 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:56.376+0000 7fc97ed9d700 1 --2- 172.21.15.40:0/1441159222 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc980100690 0x7fc98019cf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-14T19:15:56.383 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:56.376+0000 7fc97e59c700 1 --2- 172.21.15.40:0/1441159222 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc9800feb10 0x7fc9801959d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-14T19:15:56.383 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:56.376+0000 7fc97dd9b700 1 --2- 172.21.15.40:0/1441159222 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc9800ffd00 0x7fc980195f10 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fc97400d980 tx=0x7fc97400dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:15:56.383 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:56.376+0000 7fc96f7fe700 1 -- 172.21.15.40:0/1441159222 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc974004220 con 0x7fc9800ffd00 2023-07-14T19:15:56.383 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:56.376+0000 7fc96f7fe700 1 -- 172.21.15.40:0/1441159222 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc974007640 con 0x7fc9800ffd00 2023-07-14T19:15:56.383 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:56.376+0000 7fc984c69700 1 -- 172.21.15.40:0/1441159222 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc98019db00 con 0x7fc9800ffd00 2023-07-14T19:15:56.384 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:56.376+0000 7fc96f7fe700 1 -- 172.21.15.40:0/1441159222 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc974005610 con 0x7fc9800ffd00 2023-07-14T19:15:56.384 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:56.376+0000 7fc984c69700 1 -- 172.21.15.40:0/1441159222 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc98019e060 con 0x7fc9800ffd00 2023-07-14T19:15:56.384 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:56.377+0000 7fc984c69700 1 -- 172.21.15.40:0/1441159222 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc98010a330 con 0x7fc9800ffd00 2023-07-14T19:15:56.384 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:56.378+0000 7fc96f7fe700 1 -- 172.21.15.40:0/1441159222 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7fc974004750 con 0x7fc9800ffd00 2023-07-14T19:15:56.385 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:56.379+0000 7fc96f7fe700 1 --2- 172.21.15.40:0/1441159222 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7fc9680a11d0 0x7fc9680a3680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:15:56.385 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:56.379+0000 7fc97e59c700 1 --2- 172.21.15.40:0/1441159222 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7fc9680a11d0 0x7fc9680a3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:15:56.385 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:56.379+0000 7fc96f7fe700 1 -- 172.21.15.40:0/1441159222 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 7497+0+0 (secure 0 0 0) 0x7fc9740cc8d0 con 0x7fc9800ffd00 2023-07-14T19:15:56.385 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:56.380+0000 7fc97e59c700 1 --2- 172.21.15.40:0/1441159222 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7fc9680a11d0 0x7fc9680a3680 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fc97000b5c0 tx=0x7fc970005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:15:56.386 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:56.383+0000 7fc96f7fe700 1 -- 172.21.15.40:0/1441159222 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fc974096350 con 0x7fc9800ffd00 2023-07-14T19:15:56.471 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:15:56 smithi040 ceph-mon[96518]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T19:15:56.471 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:15:56 smithi040 ceph-mon[96518]: from='client.? 172.21.15.40:0/3637734798' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2023-07-14T19:15:56.471 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:15:56 smithi040 ceph-mon[96518]: from='client.? 172.21.15.40:0/2128360787' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2023-07-14T19:15:56.472 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:15:56 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:15:56.472 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:15:56 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:15:56.472 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:15:56 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:15:56.472 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:15:56 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:15:56.472 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:15:56 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:15:56.473 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:15:56 smithi191 ceph-mon[98311]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T19:15:56.473 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:15:56 smithi191 ceph-mon[98311]: from='client.? 172.21.15.40:0/3637734798' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2023-07-14T19:15:56.473 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:15:56 smithi191 ceph-mon[98311]: from='client.? 172.21.15.40:0/2128360787' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2023-07-14T19:15:56.473 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:15:56 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:15:56.473 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:15:56 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:15:56.473 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:15:56 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:15:56.473 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:15:56 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:15:56.474 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:15:56 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:15:56.492 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:15:56 smithi130 ceph-mon[97832]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T19:15:56.492 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:15:56 smithi130 ceph-mon[97832]: from='client.? 172.21.15.40:0/3637734798' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2023-07-14T19:15:56.492 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:15:56 smithi130 ceph-mon[97832]: from='client.? 172.21.15.40:0/2128360787' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2023-07-14T19:15:56.492 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:15:56 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:15:56.492 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:15:56 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:15:56.493 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:15:56 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:15:56.493 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:15:56 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:15:56.493 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:15:56 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:15:56.585 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:56.583+0000 7fc984c69700 1 -- 172.21.15.40:0/1441159222 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7fc98010a4f0 con 0x7fc9800ffd00 2023-07-14T19:15:57.227 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:57.225+0000 7fc96f7fe700 1 -- 172.21.15.40:0/1441159222 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v79) v1 ==== 170+0+0 (secure 0 0 0) 0x7fc974099970 con 0x7fc9800ffd00 2023-07-14T19:15:57.237 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:57.235+0000 7fc984c69700 1 -- 172.21.15.40:0/1441159222 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7fc9680a11d0 msgr2=0x7fc9680a3680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:15:57.237 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:57.235+0000 7fc984c69700 1 --2- 172.21.15.40:0/1441159222 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7fc9680a11d0 0x7fc9680a3680 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fc97000b5c0 tx=0x7fc970005fb0 comp rx=0 tx=0).stop 2023-07-14T19:15:57.237 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:57.235+0000 7fc984c69700 1 -- 172.21.15.40:0/1441159222 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc9800ffd00 msgr2=0x7fc980195f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:15:57.237 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:57.235+0000 7fc984c69700 1 --2- 172.21.15.40:0/1441159222 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc9800ffd00 0x7fc980195f10 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fc97400d980 tx=0x7fc97400dc90 comp rx=0 tx=0).stop 2023-07-14T19:15:57.242 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:57.240+0000 7fc984c69700 1 -- 172.21.15.40:0/1441159222 shutdown_connections 2023-07-14T19:15:57.242 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:57.240+0000 7fc984c69700 1 --2- 172.21.15.40:0/1441159222 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7fc9680a11d0 0x7fc9680a3680 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:15:57.243 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:57.240+0000 7fc984c69700 1 --2- 172.21.15.40:0/1441159222 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc980100690 0x7fc98019cf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:15:57.243 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:57.240+0000 7fc984c69700 1 --2- 172.21.15.40:0/1441159222 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc9800feb10 0x7fc9801959d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:15:57.243 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:57.240+0000 7fc984c69700 1 --2- 172.21.15.40:0/1441159222 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc9800ffd00 0x7fc980195f10 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:15:57.243 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:57.240+0000 7fc984c69700 1 -- 172.21.15.40:0/1441159222 >> 172.21.15.40:0/1441159222 conn(0x7fc9800fa130 msgr2=0x7fc9800677e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:15:57.244 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:57.241+0000 7fc984c69700 1 -- 172.21.15.40:0/1441159222 shutdown_connections 2023-07-14T19:15:57.244 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:57.242+0000 7fc984c69700 1 -- 172.21.15.40:0/1441159222 wait complete. 2023-07-14T19:15:57.246 INFO:teuthology.orchestra.run.smithi040.stderr:pool 'cephfs_data' created 2023-07-14T19:15:57.260 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2023-07-14T19:15:57.395 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:57.390+0000 7f3c5e587700 1 -- 172.21.15.40:0/3927328934 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3c58100730 msgr2=0x7f3c58102b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:15:57.395 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:57.390+0000 7f3c5e587700 1 --2- 172.21.15.40:0/3927328934 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3c58100730 0x7f3c58102b10 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f3c48009990 tx=0x7f3c48009ca0 comp rx=0 tx=0).stop 2023-07-14T19:15:57.396 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:57.390+0000 7f3c5e587700 1 -- 172.21.15.40:0/3927328934 shutdown_connections 2023-07-14T19:15:57.396 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:57.390+0000 7f3c5e587700 1 --2- 172.21.15.40:0/3927328934 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3c58105970 0x7f3c58107d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:15:57.396 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:57.390+0000 7f3c5e587700 1 --2- 172.21.15.40:0/3927328934 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3c58100730 0x7f3c58102b10 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:15:57.396 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:57.390+0000 7f3c5e587700 1 --2- 172.21.15.40:0/3927328934 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3c58103050 0x7f3c58105430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:15:57.396 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:57.390+0000 7f3c5e587700 1 -- 172.21.15.40:0/3927328934 >> 172.21.15.40:0/3927328934 conn(0x7f3c580fa130 msgr2=0x7f3c580fc560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:15:57.396 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:57.390+0000 7f3c5e587700 1 -- 172.21.15.40:0/3927328934 shutdown_connections 2023-07-14T19:15:57.397 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:57.391+0000 7f3c5e587700 1 -- 172.21.15.40:0/3927328934 wait complete. 2023-07-14T19:15:57.397 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:57.391+0000 7f3c5e587700 1 Processor -- start 2023-07-14T19:15:57.397 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:57.391+0000 7f3c5e587700 1 -- start start 2023-07-14T19:15:57.397 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:57.391+0000 7f3c5e587700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3c58100730 0x7f3c581000a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:15:57.397 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:57.391+0000 7f3c5e587700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3c58103050 0x7f3c580fe6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:15:57.397 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:57.391+0000 7f3c5e587700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3c58105970 0x7f3c580fec20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:15:57.397 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:57.391+0000 7f3c5e587700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c580ff3c0 con 0x7f3c58105970 2023-07-14T19:15:57.397 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:57.391+0000 7f3c5e587700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c580ff530 con 0x7f3c58103050 2023-07-14T19:15:57.398 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:57.391+0000 7f3c5e587700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c5819f9e0 con 0x7f3c58100730 2023-07-14T19:15:57.398 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:57.392+0000 7f3c5cb24700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3c58105970 0x7f3c580fec20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:15:57.398 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:57.392+0000 7f3c577fe700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3c58103050 0x7f3c580fe6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:15:57.398 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:57.392+0000 7f3c5cb24700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3c58105970 0x7f3c580fec20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:32844/0 (socket says 172.21.15.40:32844) 2023-07-14T19:15:57.398 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:57.392+0000 7f3c5cb24700 1 -- 172.21.15.40:0/3212858467 learned_addr learned my addr 172.21.15.40:0/3212858467 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-07-14T19:15:57.398 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:57.392+0000 7f3c57fff700 1 --2- 172.21.15.40:0/3212858467 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3c58100730 0x7f3c581000a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:15:57.398 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:57.392+0000 7f3c5cb24700 1 -- 172.21.15.40:0/3212858467 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3c58100730 msgr2=0x7f3c581000a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:15:57.399 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:57.392+0000 7f3c5cb24700 1 --2- 172.21.15.40:0/3212858467 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3c58100730 0x7f3c581000a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:15:57.399 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:57.392+0000 7f3c5cb24700 1 -- 172.21.15.40:0/3212858467 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3c58103050 msgr2=0x7f3c580fe6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:15:57.399 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:57.392+0000 7f3c5cb24700 1 --2- 172.21.15.40:0/3212858467 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3c58103050 0x7f3c580fe6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:15:57.399 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:57.392+0000 7f3c5cb24700 1 -- 172.21.15.40:0/3212858467 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c480095f0 con 0x7f3c58105970 2023-07-14T19:15:57.399 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:57.392+0000 7f3c5cb24700 1 --2- 172.21.15.40:0/3212858467 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3c58105970 0x7f3c580fec20 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f3c5000ef30 tx=0x7f3c5000c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:15:57.399 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:57.392+0000 7f3c557fa700 1 -- 172.21.15.40:0/3212858467 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c50018070 con 0x7f3c58105970 2023-07-14T19:15:57.399 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:57.392+0000 7f3c5e587700 1 -- 172.21.15.40:0/3212858467 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3c5819fc60 con 0x7f3c58105970 2023-07-14T19:15:57.399 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:57.392+0000 7f3c557fa700 1 -- 172.21.15.40:0/3212858467 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3c5000f040 con 0x7f3c58105970 2023-07-14T19:15:57.400 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:57.392+0000 7f3c557fa700 1 -- 172.21.15.40:0/3212858467 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c5000cd00 con 0x7f3c58105970 2023-07-14T19:15:57.400 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:57.392+0000 7f3c5e587700 1 -- 172.21.15.40:0/3212858467 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c581a0170 con 0x7f3c58105970 2023-07-14T19:15:57.401 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:57.394+0000 7f3c5e587700 1 -- 172.21.15.40:0/3212858467 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3c580fbca0 con 0x7f3c58105970 2023-07-14T19:15:57.401 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:57.395+0000 7f3c557fa700 1 -- 172.21.15.40:0/3212858467 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f3c50003a00 con 0x7f3c58105970 2023-07-14T19:15:57.401 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:57.395+0000 7f3c557fa700 1 --2- 172.21.15.40:0/3212858467 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f3c400a1150 0x7f3c400a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:15:57.402 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:57.396+0000 7f3c557fa700 1 -- 172.21.15.40:0/3212858467 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 7879+0+0 (secure 0 0 0) 0x7f3c500ccc30 con 0x7f3c58105970 2023-07-14T19:15:57.402 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:57.396+0000 7f3c57fff700 1 --2- 172.21.15.40:0/3212858467 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f3c400a1150 0x7f3c400a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:15:57.402 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:57.396+0000 7f3c57fff700 1 --2- 172.21.15.40:0/3212858467 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f3c400a1150 0x7f3c400a3600 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f3c4800b2e0 tx=0x7f3c48005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:15:57.402 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:57.399+0000 7f3c557fa700 1 -- 172.21.15.40:0/3212858467 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f3c500965c0 con 0x7f3c58105970 2023-07-14T19:15:57.471 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:15:57 smithi040 ceph-mon[96518]: from='client.? 172.21.15.40:0/2128360787' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2023-07-14T19:15:57.471 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:15:57 smithi040 ceph-mon[96518]: osdmap e78: 12 total, 12 up, 12 in 2023-07-14T19:15:57.472 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:15:57 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:15:57.472 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:15:57 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:15:57.472 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:15:57 smithi040 ceph-mon[96518]: from='client.? 172.21.15.40:0/1441159222' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2023-07-14T19:15:57.473 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:15:57 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:15:57.473 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:15:57 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T19:15:57.473 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:15:57 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:15:57.474 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:15:57 smithi191 ceph-mon[98311]: from='client.? 172.21.15.40:0/2128360787' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2023-07-14T19:15:57.474 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:15:57 smithi191 ceph-mon[98311]: osdmap e78: 12 total, 12 up, 12 in 2023-07-14T19:15:57.474 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:15:57 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:15:57.475 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:15:57 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:15:57.475 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:15:57 smithi191 ceph-mon[98311]: from='client.? 172.21.15.40:0/1441159222' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2023-07-14T19:15:57.475 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:15:57 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:15:57.475 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:15:57 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T19:15:57.476 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:15:57 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:15:57.492 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:15:57 smithi130 ceph-mon[97832]: from='client.? 172.21.15.40:0/2128360787' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2023-07-14T19:15:57.493 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:15:57 smithi130 ceph-mon[97832]: osdmap e78: 12 total, 12 up, 12 in 2023-07-14T19:15:57.493 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:15:57 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:15:57.493 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:15:57 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:15:57.494 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:15:57 smithi130 ceph-mon[97832]: from='client.? 172.21.15.40:0/1441159222' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2023-07-14T19:15:57.494 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:15:57 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:15:57.494 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:15:57 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T19:15:57.494 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:15:57 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:15:57.650 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:57.648+0000 7f3c5e587700 1 -- 172.21.15.40:0/3212858467 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f3c5802c8e0 con 0x7f3c58105970 2023-07-14T19:15:58.492 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:15:58 smithi130 ceph-mon[97832]: pgmap v251: 33 pgs: 4 creating+peering, 28 unknown, 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T19:15:58.492 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:15:58 smithi130 ceph-mon[97832]: from='client.? 172.21.15.40:0/1441159222' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2023-07-14T19:15:58.493 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:15:58 smithi130 ceph-mon[97832]: osdmap e79: 12 total, 12 up, 12 in 2023-07-14T19:15:58.493 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:15:58 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T19:15:58.493 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:15:58 smithi130 ceph-mon[97832]: from='client.? 172.21.15.40:0/3212858467' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2023-07-14T19:15:58.493 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:15:58 smithi130 ceph-mon[97832]: daemon mds.b assigned to filesystem cephfs as rank 0 2023-07-14T19:15:58.563 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:58.562+0000 7f3c557fa700 1 -- 172.21.15.40:0/3212858467 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 2 and data pool 3 v14) v1 ==== 172+0+0 (secure 0 0 0) 0x7f3c50099be0 con 0x7f3c58105970 2023-07-14T19:15:58.566 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:58.564+0000 7f3c5e587700 1 -- 172.21.15.40:0/3212858467 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f3c400a1150 msgr2=0x7f3c400a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:15:58.566 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:58.564+0000 7f3c5e587700 1 --2- 172.21.15.40:0/3212858467 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f3c400a1150 0x7f3c400a3600 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f3c4800b2e0 tx=0x7f3c48005fb0 comp rx=0 tx=0).stop 2023-07-14T19:15:58.566 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:58.564+0000 7f3c5e587700 1 -- 172.21.15.40:0/3212858467 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3c58105970 msgr2=0x7f3c580fec20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:15:58.566 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:58.564+0000 7f3c5e587700 1 --2- 172.21.15.40:0/3212858467 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3c58105970 0x7f3c580fec20 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f3c5000ef30 tx=0x7f3c5000c3a0 comp rx=0 tx=0).stop 2023-07-14T19:15:58.567 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:58.565+0000 7f3c5e587700 1 -- 172.21.15.40:0/3212858467 shutdown_connections 2023-07-14T19:15:58.567 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:58.565+0000 7f3c5e587700 1 --2- 172.21.15.40:0/3212858467 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f3c400a1150 0x7f3c400a3600 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:15:58.567 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:58.565+0000 7f3c5e587700 1 --2- 172.21.15.40:0/3212858467 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3c58100730 0x7f3c581000a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:15:58.567 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:58.565+0000 7f3c5e587700 1 --2- 172.21.15.40:0/3212858467 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3c58105970 0x7f3c580fec20 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:15:58.568 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:58.565+0000 7f3c5e587700 1 --2- 172.21.15.40:0/3212858467 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3c58103050 0x7f3c580fe6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:15:58.568 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:58.565+0000 7f3c5e587700 1 -- 172.21.15.40:0/3212858467 >> 172.21.15.40:0/3212858467 conn(0x7f3c580fa130 msgr2=0x7f3c58101390 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:15:58.568 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:58.565+0000 7f3c5e587700 1 -- 172.21.15.40:0/3212858467 shutdown_connections 2023-07-14T19:15:58.568 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:58.565+0000 7f3c5e587700 1 -- 172.21.15.40:0/3212858467 wait complete. 2023-07-14T19:15:58.569 INFO:teuthology.orchestra.run.smithi040.stderr:new fs with metadata pool 2 and data pool 3 2023-07-14T19:15:58.580 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:15:58 smithi040 ceph-mon[96518]: pgmap v251: 33 pgs: 4 creating+peering, 28 unknown, 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T19:15:58.580 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:15:58 smithi040 ceph-mon[96518]: from='client.? 172.21.15.40:0/1441159222' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2023-07-14T19:15:58.581 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:15:58 smithi040 ceph-mon[96518]: osdmap e79: 12 total, 12 up, 12 in 2023-07-14T19:15:58.581 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:15:58 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T19:15:58.581 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:15:58 smithi040 ceph-mon[96518]: from='client.? 172.21.15.40:0/3212858467' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2023-07-14T19:15:58.582 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:15:58 smithi040 ceph-mon[96518]: daemon mds.b assigned to filesystem cephfs as rank 0 2023-07-14T19:15:58.583 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-07-14T19:15:58.717 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:58.715+0000 7f9cd5ecf700 1 -- 172.21.15.40:0/990074018 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9cd00feb00 msgr2=0x7f9cd00fef10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:15:58.718 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:58.715+0000 7f9cd5ecf700 1 --2- 172.21.15.40:0/990074018 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9cd00feb00 0x7f9cd00fef10 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f9cc0009990 tx=0x7f9cc0009ca0 comp rx=0 tx=0).stop 2023-07-14T19:15:58.718 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:58.715+0000 7f9cd5ecf700 1 -- 172.21.15.40:0/990074018 shutdown_connections 2023-07-14T19:15:58.718 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:58.715+0000 7f9cd5ecf700 1 --2- 172.21.15.40:0/990074018 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9cd0100680 0x7f9cd0106f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:15:58.718 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:58.715+0000 7f9cd5ecf700 1 --2- 172.21.15.40:0/990074018 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9cd00feb00 0x7f9cd00fef10 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:15:58.718 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:58.715+0000 7f9cd5ecf700 1 --2- 172.21.15.40:0/990074018 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9cd00ffcf0 0x7f9cd0100140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:15:58.718 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:58.715+0000 7f9cd5ecf700 1 -- 172.21.15.40:0/990074018 >> 172.21.15.40:0/990074018 conn(0x7f9cd00fa100 msgr2=0x7f9cd00fc550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:15:58.719 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:58.716+0000 7f9cd5ecf700 1 -- 172.21.15.40:0/990074018 shutdown_connections 2023-07-14T19:15:58.719 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:58.716+0000 7f9cd5ecf700 1 -- 172.21.15.40:0/990074018 wait complete. 2023-07-14T19:15:58.721 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:58.716+0000 7f9cd5ecf700 1 Processor -- start 2023-07-14T19:15:58.721 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:58.717+0000 7f9cd5ecf700 1 -- start start 2023-07-14T19:15:58.721 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:58.717+0000 7f9cd5ecf700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9cd00feb00 0x7f9cd0194860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:15:58.721 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:58.717+0000 7f9cd5ecf700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9cd00ffcf0 0x7f9cd0194da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:15:58.721 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:58.717+0000 7f9cd5ecf700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9cd0100680 0x7f9cd019be10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:15:58.721 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:58.717+0000 7f9cd5ecf700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9cd019c3e0 con 0x7f9cd00ffcf0 2023-07-14T19:15:58.722 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:58.717+0000 7f9cd5ecf700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9cd019c520 con 0x7f9cd0100680 2023-07-14T19:15:58.722 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:58.717+0000 7f9cd5ecf700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9cd019c6b0 con 0x7f9cd00feb00 2023-07-14T19:15:58.722 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:58.717+0000 7f9cceffd700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9cd00ffcf0 0x7f9cd0194da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:15:58.722 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:58.717+0000 7f9cceffd700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9cd00ffcf0 0x7f9cd0194da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:32858/0 (socket says 172.21.15.40:32858) 2023-07-14T19:15:58.722 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:58.717+0000 7f9cceffd700 1 -- 172.21.15.40:0/1610971755 learned_addr learned my addr 172.21.15.40:0/1610971755 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-07-14T19:15:58.722 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:58.717+0000 7f9ccf7fe700 1 --2- 172.21.15.40:0/1610971755 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9cd00feb00 0x7f9cd0194860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:15:58.722 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:58.717+0000 7f9ccffff700 1 --2- 172.21.15.40:0/1610971755 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9cd0100680 0x7f9cd019be10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:15:58.722 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:58.717+0000 7f9cceffd700 1 -- 172.21.15.40:0/1610971755 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9cd00feb00 msgr2=0x7f9cd0194860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:15:58.723 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:58.717+0000 7f9cceffd700 1 --2- 172.21.15.40:0/1610971755 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9cd00feb00 0x7f9cd0194860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:15:58.723 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:58.717+0000 7f9cceffd700 1 -- 172.21.15.40:0/1610971755 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9cd0100680 msgr2=0x7f9cd019be10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:15:58.723 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:58.717+0000 7f9cceffd700 1 --2- 172.21.15.40:0/1610971755 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9cd0100680 0x7f9cd019be10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:15:58.723 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:58.717+0000 7f9cceffd700 1 -- 172.21.15.40:0/1610971755 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9cc00095f0 con 0x7f9cd00ffcf0 2023-07-14T19:15:58.723 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:58.717+0000 7f9cceffd700 1 --2- 172.21.15.40:0/1610971755 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9cd00ffcf0 0x7f9cd0194da0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f9cc400d950 tx=0x7f9cc400dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:15:58.723 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:58.718+0000 7f9cccff9700 1 -- 172.21.15.40:0/1610971755 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9cc400f890 con 0x7f9cd00ffcf0 2023-07-14T19:15:58.723 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:58.718+0000 7f9cccff9700 1 -- 172.21.15.40:0/1610971755 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f9cc4010460 con 0x7f9cd00ffcf0 2023-07-14T19:15:58.724 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:58.718+0000 7f9cd5ecf700 1 -- 172.21.15.40:0/1610971755 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9cd019c990 con 0x7f9cd00ffcf0 2023-07-14T19:15:58.724 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:58.718+0000 7f9cccff9700 1 -- 172.21.15.40:0/1610971755 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9cc400e610 con 0x7f9cd00ffcf0 2023-07-14T19:15:58.724 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:58.718+0000 7f9cd5ecf700 1 -- 172.21.15.40:0/1610971755 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9cd019cef0 con 0x7f9cd00ffcf0 2023-07-14T19:15:58.724 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:58.719+0000 7f9cd5ecf700 1 -- 172.21.15.40:0/1610971755 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9cd01060a0 con 0x7f9cd00ffcf0 2023-07-14T19:15:58.725 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:15:58 smithi191 ceph-mon[98311]: pgmap v251: 33 pgs: 4 creating+peering, 28 unknown, 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T19:15:58.725 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:15:58 smithi191 ceph-mon[98311]: from='client.? 172.21.15.40:0/1441159222' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2023-07-14T19:15:58.725 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:15:58 smithi191 ceph-mon[98311]: osdmap e79: 12 total, 12 up, 12 in 2023-07-14T19:15:58.725 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:15:58 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T19:15:58.725 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:15:58 smithi191 ceph-mon[98311]: from='client.? 172.21.15.40:0/3212858467' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2023-07-14T19:15:58.725 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:15:58 smithi191 ceph-mon[98311]: daemon mds.b assigned to filesystem cephfs as rank 0 2023-07-14T19:15:58.726 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:58.720+0000 7f9cccff9700 1 -- 172.21.15.40:0/1610971755 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f9cc40108d0 con 0x7f9cd00ffcf0 2023-07-14T19:15:58.726 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:58.720+0000 7f9cccff9700 1 --2- 172.21.15.40:0/1610971755 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f9cb80a1220 0x7f9cb80a36d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:15:58.726 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:58.720+0000 7f9ccf7fe700 1 --2- 172.21.15.40:0/1610971755 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f9cb80a1220 0x7f9cb80a36d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:15:58.726 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:58.720+0000 7f9cccff9700 1 -- 172.21.15.40:0/1610971755 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f9cc40cc510 con 0x7f9cd00ffcf0 2023-07-14T19:15:58.727 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:58.721+0000 7f9ccf7fe700 1 --2- 172.21.15.40:0/1610971755 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f9cb80a1220 0x7f9cb80a36d0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f9cc000b5c0 tx=0x7f9cc0005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:15:58.727 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:58.724+0000 7f9cccff9700 1 -- 172.21.15.40:0/1610971755 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f9cc4095db0 con 0x7f9cd00ffcf0 2023-07-14T19:15:58.939 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:58.938+0000 7f9cd5ecf700 1 -- 172.21.15.40:0/1610971755 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9cd002cb30 con 0x7f9cd00ffcf0 2023-07-14T19:15:58.941 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:58.939+0000 7f9cccff9700 1 -- 172.21.15.40:0/1610971755 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v80) v1 ==== 74+0+22134 (secure 0 0 0) 0x7f9cc40993d0 con 0x7f9cd00ffcf0 2023-07-14T19:15:58.941 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-07-14T19:15:58.942 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":80,"fsid":"adc52f04-2279-11ee-9b33-001a4aab830c","created":"2023-07-14T19:08:56.893730+0000","modified":"2023-07-14T19:15:57.650312+0000","last_up_change":"2023-07-14T19:15:16.632979+0000","last_in_change":"2023-07-14T19:14:58.961702+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-07-14T19:12:03.271333+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-07-14T19:15:56.073464+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-07-14T19:15:57.018726+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"0d79356a-95d2-49ad-857b-2092ae9f7e13","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":3599703125},{"type":"v1","addr":"172.21.15.40:6803","nonce":3599703125}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":3599703125},{"type":"v1","addr":"172.21.15.40:6805","nonce":3599703125}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":3599703125},{"type":"v1","addr":"172.21.15.40:6809","nonce":3599703125}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":3599703125},{"type":"v1","addr":"172.21.15.40:6807","nonce":3599703125}]},"public_addr":"172.21.15.40:6803/3599703125","cluster_addr":"172.21.15.40:6805/3599703125","heartbeat_back_addr":"172.21.15.40:6809/3599703125","heartbeat_front_addr":"172.21.15.40:6807/3599703125","state":["exists","up"]},{"osd":1,"uuid":"40f8a3d1-a5e4-473c-b9f2-7600ec22be0d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6800","nonce":2512760702},{"type":"v1","addr":"172.21.15.130:6801","nonce":2512760702}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6802","nonce":2512760702},{"type":"v1","addr":"172.21.15.130:6803","nonce":2512760702}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6806","nonce":2512760702},{"type":"v1","addr":"172.21.15.130:6807","nonce":2512760702}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6804","nonce":2512760702},{"type":"v1","addr":"172.21.15.130:6805","nonce":2512760702}]},"public_addr":"172.21.15.130:6801/2512760702","cluster_addr":"172.21.15.130:6803/2512760702","heartbeat_back_addr":"172.21.15.130:6807/2512760702","heartbeat_front_addr":"172.21.15.130:6805/2512760702","state":["exists","up"]},{"osd":2,"uuid":"5a157891-cb7a-4ea6-9239-f48a7c67c003","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6800","nonce":589569135},{"type":"v1","addr":"172.21.15.191:6801","nonce":589569135}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6802","nonce":589569135},{"type":"v1","addr":"172.21.15.191:6803","nonce":589569135}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6806","nonce":589569135},{"type":"v1","addr":"172.21.15.191:6807","nonce":589569135}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6804","nonce":589569135},{"type":"v1","addr":"172.21.15.191:6805","nonce":589569135}]},"public_addr":"172.21.15.191:6801/589569135","cluster_addr":"172.21.15.191:6803/589569135","heartbeat_back_addr":"172.21.15.191:6807/589569135","heartbeat_front_addr":"172.21.15.191:6805/589569135","state":["exists","up"]},{"osd":3,"uuid":"8020e7dc-ac40-487f-857a-8da40bccb459","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":996287860},{"type":"v1","addr":"172.21.15.40:6811","nonce":996287860}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6812","nonce":996287860},{"type":"v1","addr":"172.21.15.40:6813","nonce":996287860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6816","nonce":996287860},{"type":"v1","addr":"172.21.15.40:6817","nonce":996287860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":996287860},{"type":"v1","addr":"172.21.15.40:6815","nonce":996287860}]},"public_addr":"172.21.15.40:6811/996287860","cluster_addr":"172.21.15.40:6813/996287860","heartbeat_back_addr":"172.21.15.40:6817/996287860","heartbeat_front_addr":"172.21.15.40:6815/996287860","state":["exists","up"]},{"osd":4,"uuid":"c5501b35-f62d-482d-8d22-115c88f0298e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6808","nonce":3507448720},{"type":"v1","addr":"172.21.15.130:6809","nonce":3507448720}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6810","nonce":3507448720},{"type":"v1","addr":"172.21.15.130:6811","nonce":3507448720}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6814","nonce":3507448720},{"type":"v1","addr":"172.21.15.130:6815","nonce":3507448720}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6812","nonce":3507448720},{"type":"v1","addr":"172.21.15.130:6813","nonce":3507448720}]},"public_addr":"172.21.15.130:6809/3507448720","cluster_addr":"172.21.15.130:6811/3507448720","heartbeat_back_addr":"172.21.15.130:6815/3507448720","heartbeat_front_addr":"172.21.15.130:6813/3507448720","state":["exists","up"]},{"osd":5,"uuid":"7770b6d6-5874-493a-986a-84db482cee42","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6808","nonce":1781944663},{"type":"v1","addr":"172.21.15.191:6809","nonce":1781944663}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6810","nonce":1781944663},{"type":"v1","addr":"172.21.15.191:6811","nonce":1781944663}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6814","nonce":1781944663},{"type":"v1","addr":"172.21.15.191:6815","nonce":1781944663}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6812","nonce":1781944663},{"type":"v1","addr":"172.21.15.191:6813","nonce":1781944663}]},"public_addr":"172.21.15.191:6809/1781944663","cluster_addr":"172.21.15.191:6811/1781944663","heartbeat_back_addr":"172.21.15.191:6815/1781944663","heartbeat_front_addr":"172.21.15.191:6813/1781944663","state":["exists","up"]},{"osd":6,"uuid":"ef65ad00-dfee-4300-9961-dfc8573aab06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6818","nonce":693186773},{"type":"v1","addr":"172.21.15.40:6819","nonce":693186773}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6820","nonce":693186773},{"type":"v1","addr":"172.21.15.40:6821","nonce":693186773}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6824","nonce":693186773},{"type":"v1","addr":"172.21.15.40:6825","nonce":693186773}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6822","nonce":693186773},{"type":"v1","addr":"172.21.15.40:6823","nonce":693186773}]},"public_addr":"172.21.15.40:6819/693186773","cluster_addr":"172.21.15.40:6821/693186773","heartbeat_back_addr":"172.21.15.40:6825/693186773","heartbeat_front_addr":"172.21.15.40:6823/693186773","state":["exists","up"]},{"osd":7,"uuid":"1700fbfa-cbfc-4f76-97be-727a3b100740","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":50,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6816","nonce":2384019528},{"type":"v1","addr":"172.21.15.130:6817","nonce":2384019528}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6818","nonce":2384019528},{"type":"v1","addr":"172.21.15.130:6819","nonce":2384019528}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6822","nonce":2384019528},{"type":"v1","addr":"172.21.15.130:6823","nonce":2384019528}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6820","nonce":2384019528},{"type":"v1","addr":"172.21.15.130:6821","nonce":2384019528}]},"public_addr":"172.21.15.130:6817/2384019528","cluster_addr":"172.21.15.130:6819/2384019528","heartbeat_back_addr":"172.21.15.130:6823/2384019528","heartbeat_front_addr":"172.21.15.130:6821/2384019528","state":["exists","up"]},{"osd":8,"uuid":"b76d4033-c1e2-457b-8571-c8a49fe443f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":57,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6816","nonce":2386818417},{"type":"v1","addr":"172.21.15.191:6817","nonce":2386818417}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6818","nonce":2386818417},{"type":"v1","addr":"172.21.15.191:6819","nonce":2386818417}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6822","nonce":2386818417},{"type":"v1","addr":"172.21.15.191:6823","nonce":2386818417}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6820","nonce":2386818417},{"type":"v1","addr":"172.21.15.191:6821","nonce":2386818417}]},"public_addr":"172.21.15.191:6817/2386818417","cluster_addr":"172.21.15.191:6819/2386818417","heartbeat_back_addr":"172.21.15.191:6823/2386818417","heartbeat_front_addr":"172.21.15.191:6821/2386818417","state":["exists","up"]},{"osd":9,"uuid":"df31bf2f-e30e-41b6-b2b9-0e26a07a88bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":64,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6826","nonce":2228380263},{"type":"v1","addr":"172.21.15.40:6827","nonce":2228380263}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6828","nonce":2228380263},{"type":"v1","addr":"172.21.15.40:6829","nonce":2228380263}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":2228380263},{"type":"v1","addr":"172.21.15.40:6833","nonce":2228380263}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6830","nonce":2228380263},{"type":"v1","addr":"172.21.15.40:6831","nonce":2228380263}]},"public_addr":"172.21.15.40:6827/2228380263","cluster_addr":"172.21.15.40:6829/2228380263","heartbeat_back_addr":"172.21.15.40:6833/2228380263","heartbeat_front_addr":"172.21.15.40:6831/2228380263","state":["exists","up"]},{"osd":10,"uuid":"4a93d621-49ee-4ad7-9e49-750614faf912","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6824","nonce":2417067798},{"type":"v1","addr":"172.21.15.130:6825","nonce":2417067798}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6826","nonce":2417067798},{"type":"v1","addr":"172.21.15.130:6827","nonce":2417067798}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6830","nonce":2417067798},{"type":"v1","addr":"172.21.15.130:6831","nonce":2417067798}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6828","nonce":2417067798},{"type":"v1","addr":"172.21.15.130:6829","nonce":2417067798}]},"public_addr":"172.21.15.130:6825/2417067798","cluster_addr":"172.21.15.130:6827/2417067798","heartbeat_back_addr":"172.21.15.130:6831/2417067798","heartbeat_front_addr":"172.21.15.130:6829/2417067798","state":["exists","up"]},{"osd":11,"uuid":"f43a2a6f-82f3-42ac-b77c-4f1b34da6918","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":76,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6824","nonce":2886059446},{"type":"v1","addr":"172.21.15.191:6825","nonce":2886059446}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6826","nonce":2886059446},{"type":"v1","addr":"172.21.15.191:6827","nonce":2886059446}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6830","nonce":2886059446},{"type":"v1","addr":"172.21.15.191:6831","nonce":2886059446}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6828","nonce":2886059446},{"type":"v1","addr":"172.21.15.191:6829","nonce":2886059446}]},"public_addr":"172.21.15.191:6825/2886059446","cluster_addr":"172.21.15.191:6827/2886059446","heartbeat_back_addr":"172.21.15.191:6831/2886059446","heartbeat_front_addr":"172.21.15.191:6829/2886059446","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:11:18.320495+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:11:39.901199+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:12:00.869042+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:12:23.292742+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:12:44.587019+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:13:06.494893+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:13:28.378374+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:13:50.033491+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:14:10.937062+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:14:32.358498+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:14:53.666584+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:15:14.837566+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.40:6801/3698757264":"2023-07-15T19:09:50.732458+0000","172.21.15.40:6800/3698757264":"2023-07-15T19:09:50.732458+0000","172.21.15.40:6800/2569015667":"2023-07-15T19:09:24.441543+0000","172.21.15.40:0/3653961003":"2023-07-15T19:09:24.441543+0000","172.21.15.40:0/3568861680":"2023-07-15T19:09:24.441543+0000","172.21.15.40:0/3249773172":"2023-07-15T19:09:50.732458+0000","172.21.15.40:0/4084215480":"2023-07-15T19:09:24.441543+0000","172.21.15.40:0/1163941760":"2023-07-15T19:09:50.732458+0000","172.21.15.40:6801/2569015667":"2023-07-15T19:09:24.441543+0000","172.21.15.40:0/2326639960":"2023-07-15T19:09:50.732458+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-07-14T19:15:58.943 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:58.941+0000 7f9cd5ecf700 1 -- 172.21.15.40:0/1610971755 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f9cb80a1220 msgr2=0x7f9cb80a36d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:15:58.943 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:58.941+0000 7f9cd5ecf700 1 --2- 172.21.15.40:0/1610971755 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f9cb80a1220 0x7f9cb80a36d0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f9cc000b5c0 tx=0x7f9cc0005fb0 comp rx=0 tx=0).stop 2023-07-14T19:15:58.944 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:58.941+0000 7f9cd5ecf700 1 -- 172.21.15.40:0/1610971755 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9cd00ffcf0 msgr2=0x7f9cd0194da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:15:58.944 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:58.941+0000 7f9cd5ecf700 1 --2- 172.21.15.40:0/1610971755 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9cd00ffcf0 0x7f9cd0194da0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f9cc400d950 tx=0x7f9cc400dc60 comp rx=0 tx=0).stop 2023-07-14T19:15:58.944 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:58.942+0000 7f9cd5ecf700 1 -- 172.21.15.40:0/1610971755 shutdown_connections 2023-07-14T19:15:58.944 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:58.942+0000 7f9cd5ecf700 1 --2- 172.21.15.40:0/1610971755 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f9cb80a1220 0x7f9cb80a36d0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:15:58.944 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:58.942+0000 7f9cd5ecf700 1 --2- 172.21.15.40:0/1610971755 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9cd00feb00 0x7f9cd0194860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:15:58.944 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:58.942+0000 7f9cd5ecf700 1 --2- 172.21.15.40:0/1610971755 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9cd0100680 0x7f9cd019be10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:15:58.945 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:58.942+0000 7f9cd5ecf700 1 --2- 172.21.15.40:0/1610971755 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9cd00ffcf0 0x7f9cd0194da0 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:15:58.945 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:58.942+0000 7f9cd5ecf700 1 -- 172.21.15.40:0/1610971755 >> 172.21.15.40:0/1610971755 conn(0x7f9cd00fa100 msgr2=0x7f9cd01031e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:15:58.945 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:58.942+0000 7f9cd5ecf700 1 -- 172.21.15.40:0/1610971755 shutdown_connections 2023-07-14T19:15:58.945 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:58.942+0000 7f9cd5ecf700 1 -- 172.21.15.40:0/1610971755 wait complete. 2023-07-14T19:15:58.956 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-07-14T19:15:59.094 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:59.092+0000 7fc620c3e700 1 -- 172.21.15.40:0/2450370573 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc61c0717c0 msgr2=0x7fc61c071bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:15:59.095 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:59.092+0000 7fc620c3e700 1 --2- 172.21.15.40:0/2450370573 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc61c0717c0 0x7fc61c071bd0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fc60c009990 tx=0x7fc60c009ca0 comp rx=0 tx=0).stop 2023-07-14T19:15:59.095 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:59.092+0000 7fc620c3e700 1 -- 172.21.15.40:0/2450370573 shutdown_connections 2023-07-14T19:15:59.095 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:59.092+0000 7fc620c3e700 1 --2- 172.21.15.40:0/2450370573 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc61c107ff0 0x7fc61c110990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:15:59.095 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:59.092+0000 7fc620c3e700 1 --2- 172.21.15.40:0/2450370573 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc61c0717c0 0x7fc61c071bd0 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:15:59.095 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:59.092+0000 7fc620c3e700 1 --2- 172.21.15.40:0/2450370573 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc61c107640 0x7fc61c107ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:15:59.095 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:59.092+0000 7fc620c3e700 1 -- 172.21.15.40:0/2450370573 >> 172.21.15.40:0/2450370573 conn(0x7fc61c06ce80 msgr2=0x7fc61c06f2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:15:59.096 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:59.093+0000 7fc620c3e700 1 -- 172.21.15.40:0/2450370573 shutdown_connections 2023-07-14T19:15:59.096 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:59.093+0000 7fc620c3e700 1 -- 172.21.15.40:0/2450370573 wait complete. 2023-07-14T19:15:59.097 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:59.093+0000 7fc620c3e700 1 Processor -- start 2023-07-14T19:15:59.098 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:59.093+0000 7fc620c3e700 1 -- start start 2023-07-14T19:15:59.098 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:59.093+0000 7fc620c3e700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc61c0717c0 0x7fc61c19e150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:15:59.098 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:59.093+0000 7fc620c3e700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc61c107640 0x7fc61c19e690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:15:59.098 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:59.093+0000 7fc620c3e700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc61c107ff0 0x7fc61c1a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:15:59.098 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:59.094+0000 7fc620c3e700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc61c1a5cd0 con 0x7fc61c0717c0 2023-07-14T19:15:59.098 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:59.094+0000 7fc620c3e700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc61c1a5e10 con 0x7fc61c107640 2023-07-14T19:15:59.099 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:59.094+0000 7fc620c3e700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc61c1a5fa0 con 0x7fc61c107ff0 2023-07-14T19:15:59.099 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:59.094+0000 7fc61a59c700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc61c0717c0 0x7fc61c19e150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:15:59.099 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:59.094+0000 7fc619d9b700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc61c107640 0x7fc61c19e690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:15:59.099 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:59.094+0000 7fc61a59c700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc61c0717c0 0x7fc61c19e150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:32870/0 (socket says 172.21.15.40:32870) 2023-07-14T19:15:59.099 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:59.094+0000 7fc61a59c700 1 -- 172.21.15.40:0/2609154881 learned_addr learned my addr 172.21.15.40:0/2609154881 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-07-14T19:15:59.099 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:59.094+0000 7fc61ad9d700 1 --2- 172.21.15.40:0/2609154881 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc61c107ff0 0x7fc61c1a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:15:59.099 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:59.094+0000 7fc61a59c700 1 -- 172.21.15.40:0/2609154881 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc61c107ff0 msgr2=0x7fc61c1a5700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:15:59.099 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:59.094+0000 7fc61a59c700 1 --2- 172.21.15.40:0/2609154881 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc61c107ff0 0x7fc61c1a5700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:15:59.100 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:59.094+0000 7fc61a59c700 1 -- 172.21.15.40:0/2609154881 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc61c107640 msgr2=0x7fc61c19e690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:15:59.100 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:59.094+0000 7fc61a59c700 1 --2- 172.21.15.40:0/2609154881 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc61c107640 0x7fc61c19e690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:15:59.100 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:59.094+0000 7fc61a59c700 1 -- 172.21.15.40:0/2609154881 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc60c0095f0 con 0x7fc61c0717c0 2023-07-14T19:15:59.100 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:59.094+0000 7fc619d9b700 1 --2- 172.21.15.40:0/2609154881 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc61c107640 0x7fc61c19e690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-14T19:15:59.100 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:59.094+0000 7fc61a59c700 1 --2- 172.21.15.40:0/2609154881 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc61c0717c0 0x7fc61c19e150 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fc60c004730 tx=0x7fc60c004760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:15:59.100 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:59.094+0000 7fc60b7fe700 1 -- 172.21.15.40:0/2609154881 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc60c01d070 con 0x7fc61c0717c0 2023-07-14T19:15:59.100 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:59.094+0000 7fc620c3e700 1 -- 172.21.15.40:0/2609154881 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc61c1a6220 con 0x7fc61c0717c0 2023-07-14T19:15:59.101 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:59.094+0000 7fc60b7fe700 1 -- 172.21.15.40:0/2609154881 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc60c004c90 con 0x7fc61c0717c0 2023-07-14T19:15:59.101 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:59.094+0000 7fc60b7fe700 1 -- 172.21.15.40:0/2609154881 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc60c0054b0 con 0x7fc61c0717c0 2023-07-14T19:15:59.101 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:59.095+0000 7fc620c3e700 1 -- 172.21.15.40:0/2609154881 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc61c1a65e0 con 0x7fc61c0717c0 2023-07-14T19:15:59.101 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:59.096+0000 7fc620c3e700 1 -- 172.21.15.40:0/2609154881 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc61c10f430 con 0x7fc61c0717c0 2023-07-14T19:15:59.102 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:59.097+0000 7fc60b7fe700 1 -- 172.21.15.40:0/2609154881 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7fc60c00fbc0 con 0x7fc61c0717c0 2023-07-14T19:15:59.102 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:59.097+0000 7fc60b7fe700 1 --2- 172.21.15.40:0/2609154881 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7fc6040a11a0 0x7fc6040a3650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:15:59.102 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:59.097+0000 7fc619d9b700 1 --2- 172.21.15.40:0/2609154881 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7fc6040a11a0 0x7fc6040a3650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:15:59.102 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:59.097+0000 7fc60b7fe700 1 -- 172.21.15.40:0/2609154881 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 7979+0+0 (secure 0 0 0) 0x7fc60c0cdea0 con 0x7fc61c0717c0 2023-07-14T19:15:59.102 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:59.098+0000 7fc619d9b700 1 --2- 172.21.15.40:0/2609154881 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7fc6040a11a0 0x7fc6040a3650 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fc610011490 tx=0x7fc610009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:15:59.104 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:59.102+0000 7fc60b7fe700 1 -- 172.21.15.40:0/2609154881 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fc60c0977d0 con 0x7fc61c0717c0 2023-07-14T19:15:59.314 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:59.313+0000 7fc620c3e700 1 -- 172.21.15.40:0/2609154881 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc61c02cb30 con 0x7fc61c0717c0 2023-07-14T19:15:59.316 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:59.314+0000 7fc60b7fe700 1 -- 172.21.15.40:0/2609154881 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v81) v1 ==== 74+0+22121 (secure 0 0 0) 0x7fc60c09adf0 con 0x7fc61c0717c0 2023-07-14T19:15:59.316 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-07-14T19:15:59.317 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":81,"fsid":"adc52f04-2279-11ee-9b33-001a4aab830c","created":"2023-07-14T19:08:56.893730+0000","modified":"2023-07-14T19:15:59.225912+0000","last_up_change":"2023-07-14T19:15:16.632979+0000","last_in_change":"2023-07-14T19:14:58.961702+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-07-14T19:12:03.271333+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-07-14T19:15:56.073464+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-07-14T19:15:57.018726+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"0d79356a-95d2-49ad-857b-2092ae9f7e13","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":3599703125},{"type":"v1","addr":"172.21.15.40:6803","nonce":3599703125}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":3599703125},{"type":"v1","addr":"172.21.15.40:6805","nonce":3599703125}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":3599703125},{"type":"v1","addr":"172.21.15.40:6809","nonce":3599703125}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":3599703125},{"type":"v1","addr":"172.21.15.40:6807","nonce":3599703125}]},"public_addr":"172.21.15.40:6803/3599703125","cluster_addr":"172.21.15.40:6805/3599703125","heartbeat_back_addr":"172.21.15.40:6809/3599703125","heartbeat_front_addr":"172.21.15.40:6807/3599703125","state":["exists","up"]},{"osd":1,"uuid":"40f8a3d1-a5e4-473c-b9f2-7600ec22be0d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6800","nonce":2512760702},{"type":"v1","addr":"172.21.15.130:6801","nonce":2512760702}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6802","nonce":2512760702},{"type":"v1","addr":"172.21.15.130:6803","nonce":2512760702}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6806","nonce":2512760702},{"type":"v1","addr":"172.21.15.130:6807","nonce":2512760702}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6804","nonce":2512760702},{"type":"v1","addr":"172.21.15.130:6805","nonce":2512760702}]},"public_addr":"172.21.15.130:6801/2512760702","cluster_addr":"172.21.15.130:6803/2512760702","heartbeat_back_addr":"172.21.15.130:6807/2512760702","heartbeat_front_addr":"172.21.15.130:6805/2512760702","state":["exists","up"]},{"osd":2,"uuid":"5a157891-cb7a-4ea6-9239-f48a7c67c003","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6800","nonce":589569135},{"type":"v1","addr":"172.21.15.191:6801","nonce":589569135}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6802","nonce":589569135},{"type":"v1","addr":"172.21.15.191:6803","nonce":589569135}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6806","nonce":589569135},{"type":"v1","addr":"172.21.15.191:6807","nonce":589569135}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6804","nonce":589569135},{"type":"v1","addr":"172.21.15.191:6805","nonce":589569135}]},"public_addr":"172.21.15.191:6801/589569135","cluster_addr":"172.21.15.191:6803/589569135","heartbeat_back_addr":"172.21.15.191:6807/589569135","heartbeat_front_addr":"172.21.15.191:6805/589569135","state":["exists","up"]},{"osd":3,"uuid":"8020e7dc-ac40-487f-857a-8da40bccb459","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":996287860},{"type":"v1","addr":"172.21.15.40:6811","nonce":996287860}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6812","nonce":996287860},{"type":"v1","addr":"172.21.15.40:6813","nonce":996287860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6816","nonce":996287860},{"type":"v1","addr":"172.21.15.40:6817","nonce":996287860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":996287860},{"type":"v1","addr":"172.21.15.40:6815","nonce":996287860}]},"public_addr":"172.21.15.40:6811/996287860","cluster_addr":"172.21.15.40:6813/996287860","heartbeat_back_addr":"172.21.15.40:6817/996287860","heartbeat_front_addr":"172.21.15.40:6815/996287860","state":["exists","up"]},{"osd":4,"uuid":"c5501b35-f62d-482d-8d22-115c88f0298e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6808","nonce":3507448720},{"type":"v1","addr":"172.21.15.130:6809","nonce":3507448720}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6810","nonce":3507448720},{"type":"v1","addr":"172.21.15.130:6811","nonce":3507448720}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6814","nonce":3507448720},{"type":"v1","addr":"172.21.15.130:6815","nonce":3507448720}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6812","nonce":3507448720},{"type":"v1","addr":"172.21.15.130:6813","nonce":3507448720}]},"public_addr":"172.21.15.130:6809/3507448720","cluster_addr":"172.21.15.130:6811/3507448720","heartbeat_back_addr":"172.21.15.130:6815/3507448720","heartbeat_front_addr":"172.21.15.130:6813/3507448720","state":["exists","up"]},{"osd":5,"uuid":"7770b6d6-5874-493a-986a-84db482cee42","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6808","nonce":1781944663},{"type":"v1","addr":"172.21.15.191:6809","nonce":1781944663}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6810","nonce":1781944663},{"type":"v1","addr":"172.21.15.191:6811","nonce":1781944663}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6814","nonce":1781944663},{"type":"v1","addr":"172.21.15.191:6815","nonce":1781944663}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6812","nonce":1781944663},{"type":"v1","addr":"172.21.15.191:6813","nonce":1781944663}]},"public_addr":"172.21.15.191:6809/1781944663","cluster_addr":"172.21.15.191:6811/1781944663","heartbeat_back_addr":"172.21.15.191:6815/1781944663","heartbeat_front_addr":"172.21.15.191:6813/1781944663","state":["exists","up"]},{"osd":6,"uuid":"ef65ad00-dfee-4300-9961-dfc8573aab06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6818","nonce":693186773},{"type":"v1","addr":"172.21.15.40:6819","nonce":693186773}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6820","nonce":693186773},{"type":"v1","addr":"172.21.15.40:6821","nonce":693186773}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6824","nonce":693186773},{"type":"v1","addr":"172.21.15.40:6825","nonce":693186773}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6822","nonce":693186773},{"type":"v1","addr":"172.21.15.40:6823","nonce":693186773}]},"public_addr":"172.21.15.40:6819/693186773","cluster_addr":"172.21.15.40:6821/693186773","heartbeat_back_addr":"172.21.15.40:6825/693186773","heartbeat_front_addr":"172.21.15.40:6823/693186773","state":["exists","up"]},{"osd":7,"uuid":"1700fbfa-cbfc-4f76-97be-727a3b100740","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":50,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6816","nonce":2384019528},{"type":"v1","addr":"172.21.15.130:6817","nonce":2384019528}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6818","nonce":2384019528},{"type":"v1","addr":"172.21.15.130:6819","nonce":2384019528}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6822","nonce":2384019528},{"type":"v1","addr":"172.21.15.130:6823","nonce":2384019528}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6820","nonce":2384019528},{"type":"v1","addr":"172.21.15.130:6821","nonce":2384019528}]},"public_addr":"172.21.15.130:6817/2384019528","cluster_addr":"172.21.15.130:6819/2384019528","heartbeat_back_addr":"172.21.15.130:6823/2384019528","heartbeat_front_addr":"172.21.15.130:6821/2384019528","state":["exists","up"]},{"osd":8,"uuid":"b76d4033-c1e2-457b-8571-c8a49fe443f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":57,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6816","nonce":2386818417},{"type":"v1","addr":"172.21.15.191:6817","nonce":2386818417}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6818","nonce":2386818417},{"type":"v1","addr":"172.21.15.191:6819","nonce":2386818417}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6822","nonce":2386818417},{"type":"v1","addr":"172.21.15.191:6823","nonce":2386818417}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6820","nonce":2386818417},{"type":"v1","addr":"172.21.15.191:6821","nonce":2386818417}]},"public_addr":"172.21.15.191:6817/2386818417","cluster_addr":"172.21.15.191:6819/2386818417","heartbeat_back_addr":"172.21.15.191:6823/2386818417","heartbeat_front_addr":"172.21.15.191:6821/2386818417","state":["exists","up"]},{"osd":9,"uuid":"df31bf2f-e30e-41b6-b2b9-0e26a07a88bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":64,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6826","nonce":2228380263},{"type":"v1","addr":"172.21.15.40:6827","nonce":2228380263}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6828","nonce":2228380263},{"type":"v1","addr":"172.21.15.40:6829","nonce":2228380263}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":2228380263},{"type":"v1","addr":"172.21.15.40:6833","nonce":2228380263}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6830","nonce":2228380263},{"type":"v1","addr":"172.21.15.40:6831","nonce":2228380263}]},"public_addr":"172.21.15.40:6827/2228380263","cluster_addr":"172.21.15.40:6829/2228380263","heartbeat_back_addr":"172.21.15.40:6833/2228380263","heartbeat_front_addr":"172.21.15.40:6831/2228380263","state":["exists","up"]},{"osd":10,"uuid":"4a93d621-49ee-4ad7-9e49-750614faf912","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6824","nonce":2417067798},{"type":"v1","addr":"172.21.15.130:6825","nonce":2417067798}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6826","nonce":2417067798},{"type":"v1","addr":"172.21.15.130:6827","nonce":2417067798}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6830","nonce":2417067798},{"type":"v1","addr":"172.21.15.130:6831","nonce":2417067798}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6828","nonce":2417067798},{"type":"v1","addr":"172.21.15.130:6829","nonce":2417067798}]},"public_addr":"172.21.15.130:6825/2417067798","cluster_addr":"172.21.15.130:6827/2417067798","heartbeat_back_addr":"172.21.15.130:6831/2417067798","heartbeat_front_addr":"172.21.15.130:6829/2417067798","state":["exists","up"]},{"osd":11,"uuid":"f43a2a6f-82f3-42ac-b77c-4f1b34da6918","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":76,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6824","nonce":2886059446},{"type":"v1","addr":"172.21.15.191:6825","nonce":2886059446}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6826","nonce":2886059446},{"type":"v1","addr":"172.21.15.191:6827","nonce":2886059446}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6830","nonce":2886059446},{"type":"v1","addr":"172.21.15.191:6831","nonce":2886059446}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6828","nonce":2886059446},{"type":"v1","addr":"172.21.15.191:6829","nonce":2886059446}]},"public_addr":"172.21.15.191:6825/2886059446","cluster_addr":"172.21.15.191:6827/2886059446","heartbeat_back_addr":"172.21.15.191:6831/2886059446","heartbeat_front_addr":"172.21.15.191:6829/2886059446","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:11:18.320495+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:11:39.901199+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:12:00.869042+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:12:23.292742+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:12:44.587019+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:13:06.494893+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:13:28.378374+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:13:50.033491+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:14:10.937062+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:14:32.358498+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:14:53.666584+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:15:14.837566+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.40:6801/3698757264":"2023-07-15T19:09:50.732458+0000","172.21.15.40:6800/3698757264":"2023-07-15T19:09:50.732458+0000","172.21.15.40:6800/2569015667":"2023-07-15T19:09:24.441543+0000","172.21.15.40:0/3653961003":"2023-07-15T19:09:24.441543+0000","172.21.15.40:0/3568861680":"2023-07-15T19:09:24.441543+0000","172.21.15.40:0/3249773172":"2023-07-15T19:09:50.732458+0000","172.21.15.40:0/4084215480":"2023-07-15T19:09:24.441543+0000","172.21.15.40:0/1163941760":"2023-07-15T19:09:50.732458+0000","172.21.15.40:6801/2569015667":"2023-07-15T19:09:24.441543+0000","172.21.15.40:0/2326639960":"2023-07-15T19:09:50.732458+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-07-14T19:15:59.318 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:59.316+0000 7fc620c3e700 1 -- 172.21.15.40:0/2609154881 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7fc6040a11a0 msgr2=0x7fc6040a3650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:15:59.318 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:59.316+0000 7fc620c3e700 1 --2- 172.21.15.40:0/2609154881 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7fc6040a11a0 0x7fc6040a3650 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fc610011490 tx=0x7fc610009260 comp rx=0 tx=0).stop 2023-07-14T19:15:59.318 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:59.316+0000 7fc620c3e700 1 -- 172.21.15.40:0/2609154881 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc61c0717c0 msgr2=0x7fc61c19e150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:15:59.319 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:59.316+0000 7fc620c3e700 1 --2- 172.21.15.40:0/2609154881 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc61c0717c0 0x7fc61c19e150 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fc60c004730 tx=0x7fc60c004760 comp rx=0 tx=0).stop 2023-07-14T19:15:59.319 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:59.316+0000 7fc620c3e700 1 -- 172.21.15.40:0/2609154881 shutdown_connections 2023-07-14T19:15:59.319 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:59.316+0000 7fc620c3e700 1 --2- 172.21.15.40:0/2609154881 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7fc6040a11a0 0x7fc6040a3650 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:15:59.319 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:59.316+0000 7fc620c3e700 1 --2- 172.21.15.40:0/2609154881 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc61c107ff0 0x7fc61c1a5700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:15:59.319 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:59.316+0000 7fc620c3e700 1 --2- 172.21.15.40:0/2609154881 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc61c0717c0 0x7fc61c19e150 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:15:59.319 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:59.316+0000 7fc620c3e700 1 --2- 172.21.15.40:0/2609154881 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc61c107640 0x7fc61c19e690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:15:59.319 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:59.316+0000 7fc620c3e700 1 -- 172.21.15.40:0/2609154881 >> 172.21.15.40:0/2609154881 conn(0x7fc61c06ce80 msgr2=0x7fc61c06f0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:15:59.319 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:59.316+0000 7fc620c3e700 1 -- 172.21.15.40:0/2609154881 shutdown_connections 2023-07-14T19:15:59.320 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:59.316+0000 7fc620c3e700 1 -- 172.21.15.40:0/2609154881 wait complete. 2023-07-14T19:15:59.331 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2023-07-14T19:15:59.465 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:59.463+0000 7f3d0dac0700 1 -- 172.21.15.40:0/866175154 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3d080ffd80 msgr2=0x7f3d081001d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:15:59.466 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:59.463+0000 7f3d0dac0700 1 --2- 172.21.15.40:0/866175154 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3d080ffd80 0x7f3d081001d0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f3cfc009990 tx=0x7f3cfc009ca0 comp rx=0 tx=0).stop 2023-07-14T19:15:59.466 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:59.463+0000 7f3d0dac0700 1 -- 172.21.15.40:0/866175154 shutdown_connections 2023-07-14T19:15:59.466 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:59.463+0000 7f3d0dac0700 1 --2- 172.21.15.40:0/866175154 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3d080feb90 0x7f3d080fefa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:15:59.466 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:59.463+0000 7f3d0dac0700 1 --2- 172.21.15.40:0/866175154 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3d08100710 0x7f3d08106fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:15:59.466 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:59.463+0000 7f3d0dac0700 1 --2- 172.21.15.40:0/866175154 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3d080ffd80 0x7f3d081001d0 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:15:59.467 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:59.463+0000 7f3d0dac0700 1 -- 172.21.15.40:0/866175154 >> 172.21.15.40:0/866175154 conn(0x7f3d080fa1d0 msgr2=0x7f3d080fc5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:15:59.467 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:59.464+0000 7f3d0dac0700 1 -- 172.21.15.40:0/866175154 shutdown_connections 2023-07-14T19:15:59.467 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:59.464+0000 7f3d0dac0700 1 -- 172.21.15.40:0/866175154 wait complete. 2023-07-14T19:15:59.467 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:59.464+0000 7f3d0dac0700 1 Processor -- start 2023-07-14T19:15:59.468 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:59.464+0000 7f3d0dac0700 1 -- start start 2023-07-14T19:15:59.469 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:59.464+0000 7f3d0dac0700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3d080feb90 0x7f3d081958d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:15:59.469 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:59.464+0000 7f3d0dac0700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3d080ffd80 0x7f3d08195e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:15:59.469 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:59.464+0000 7f3d0dac0700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3d08100710 0x7f3d0819ce80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:15:59.469 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:59.464+0000 7f3d0dac0700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d0819d450 con 0x7f3d080ffd80 2023-07-14T19:15:59.469 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:59.464+0000 7f3d0dac0700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d0819d590 con 0x7f3d08100710 2023-07-14T19:15:59.469 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:59.464+0000 7f3d0dac0700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d0819d720 con 0x7f3d080feb90 2023-07-14T19:15:59.469 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:59.465+0000 7f3d06ffd700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3d080ffd80 0x7f3d08195e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:15:59.470 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:59.465+0000 7f3d06ffd700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3d080ffd80 0x7f3d08195e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:32886/0 (socket says 172.21.15.40:32886) 2023-07-14T19:15:59.470 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:59.465+0000 7f3d06ffd700 1 -- 172.21.15.40:0/1679054317 learned_addr learned my addr 172.21.15.40:0/1679054317 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-07-14T19:15:59.470 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:59.465+0000 7f3d077fe700 1 --2- 172.21.15.40:0/1679054317 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3d080feb90 0x7f3d081958d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:15:59.470 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:59.465+0000 7f3d07fff700 1 --2- 172.21.15.40:0/1679054317 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3d08100710 0x7f3d0819ce80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:15:59.470 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:59.465+0000 7f3d06ffd700 1 -- 172.21.15.40:0/1679054317 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3d080feb90 msgr2=0x7f3d081958d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:15:59.470 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:59.465+0000 7f3d06ffd700 1 --2- 172.21.15.40:0/1679054317 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3d080feb90 0x7f3d081958d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:15:59.470 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:59.465+0000 7f3d06ffd700 1 -- 172.21.15.40:0/1679054317 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3d08100710 msgr2=0x7f3d0819ce80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:15:59.470 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:59.465+0000 7f3d06ffd700 1 --2- 172.21.15.40:0/1679054317 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3d08100710 0x7f3d0819ce80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:15:59.471 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:59.465+0000 7f3d06ffd700 1 -- 172.21.15.40:0/1679054317 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3cfc0095f0 con 0x7f3d080ffd80 2023-07-14T19:15:59.471 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:59.465+0000 7f3d07fff700 1 --2- 172.21.15.40:0/1679054317 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3d08100710 0x7f3d0819ce80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T19:15:59.471 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:59.465+0000 7f3d06ffd700 1 --2- 172.21.15.40:0/1679054317 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3d080ffd80 0x7f3d08195e10 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f3cfc005770 tx=0x7f3cfc0057a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:15:59.471 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:59.465+0000 7f3d04ff9700 1 -- 172.21.15.40:0/1679054317 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3cfc01d070 con 0x7f3d080ffd80 2023-07-14T19:15:59.471 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:59.465+0000 7f3d0dac0700 1 -- 172.21.15.40:0/1679054317 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3d0819d9a0 con 0x7f3d080ffd80 2023-07-14T19:15:59.471 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:59.465+0000 7f3d04ff9700 1 -- 172.21.15.40:0/1679054317 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3cfc0041a0 con 0x7f3d080ffd80 2023-07-14T19:15:59.471 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:59.465+0000 7f3d04ff9700 1 -- 172.21.15.40:0/1679054317 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3cfc022e70 con 0x7f3d080ffd80 2023-07-14T19:15:59.472 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:59.465+0000 7f3d0dac0700 1 -- 172.21.15.40:0/1679054317 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d0819de20 con 0x7f3d080ffd80 2023-07-14T19:15:59.472 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:59.466+0000 7f3d0dac0700 1 -- 172.21.15.40:0/1679054317 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3d0810a2f0 con 0x7f3d080ffd80 2023-07-14T19:15:59.472 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:59.467+0000 7f3d04ff9700 1 -- 172.21.15.40:0/1679054317 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f3cfc022570 con 0x7f3d080ffd80 2023-07-14T19:15:59.473 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:59.468+0000 7f3d04ff9700 1 --2- 172.21.15.40:0/1679054317 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f3cf00a1150 0x7f3cf00a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:15:59.473 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:59.468+0000 7f3d04ff9700 1 -- 172.21.15.40:0/1679054317 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f3cfc0cdba0 con 0x7f3d080ffd80 2023-07-14T19:15:59.473 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:59.468+0000 7f3d077fe700 1 --2- 172.21.15.40:0/1679054317 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f3cf00a1150 0x7f3cf00a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:15:59.473 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:59.468+0000 7f3d077fe700 1 --2- 172.21.15.40:0/1679054317 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f3cf00a1150 0x7f3cf00a3600 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f3d080ffbe0 tx=0x7f3cf800a3b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:15:59.474 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:59.472+0000 7f3d04ff9700 1 -- 172.21.15.40:0/1679054317 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f3cfc05c780 con 0x7f3d080ffd80 2023-07-14T19:15:59.492 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:15:59 smithi130 ceph-mon[97832]: osdmap e80: 12 total, 12 up, 12 in 2023-07-14T19:15:59.492 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:15:59 smithi130 ceph-mon[97832]: from='client.? 172.21.15.40:0/3212858467' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2023-07-14T19:15:59.493 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:15:59 smithi130 ceph-mon[97832]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2023-07-14T19:15:59.493 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:15:59 smithi130 ceph-mon[97832]: daemon mds.b is now active in filesystem cephfs as rank 0 2023-07-14T19:15:59.493 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:15:59 smithi130 ceph-mon[97832]: from='client.? 172.21.15.40:0/1610971755' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T19:15:59.721 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:15:59 smithi040 ceph-mon[96518]: osdmap e80: 12 total, 12 up, 12 in 2023-07-14T19:15:59.721 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:15:59 smithi040 ceph-mon[96518]: from='client.? 172.21.15.40:0/3212858467' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2023-07-14T19:15:59.721 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:15:59 smithi040 ceph-mon[96518]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2023-07-14T19:15:59.721 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:15:59 smithi040 ceph-mon[96518]: daemon mds.b is now active in filesystem cephfs as rank 0 2023-07-14T19:15:59.721 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:15:59 smithi040 ceph-mon[96518]: from='client.? 172.21.15.40:0/1610971755' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T19:15:59.723 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:15:59 smithi191 ceph-mon[98311]: osdmap e80: 12 total, 12 up, 12 in 2023-07-14T19:15:59.723 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:15:59 smithi191 ceph-mon[98311]: from='client.? 172.21.15.40:0/3212858467' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2023-07-14T19:15:59.723 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:15:59 smithi191 ceph-mon[98311]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2023-07-14T19:15:59.723 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:15:59 smithi191 ceph-mon[98311]: daemon mds.b is now active in filesystem cephfs as rank 0 2023-07-14T19:15:59.723 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:15:59 smithi191 ceph-mon[98311]: from='client.? 172.21.15.40:0/1610971755' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T19:15:59.736 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:15:59.734+0000 7f3d0dac0700 1 -- 172.21.15.40:0/1679054317 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f3d0819e460 con 0x7f3d080ffd80 2023-07-14T19:16:00.492 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:16:00 smithi130 ceph-mon[97832]: pgmap v254: 97 pgs: 14 creating+peering, 55 unknown, 28 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s wr, 4 op/s 2023-07-14T19:16:00.493 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:16:00 smithi130 ceph-mon[97832]: osdmap e81: 12 total, 12 up, 12 in 2023-07-14T19:16:00.493 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:16:00 smithi130 ceph-mon[97832]: from='client.? 172.21.15.40:0/2609154881' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T19:16:00.493 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:16:00 smithi130 ceph-mon[97832]: mds.? [v2:172.21.15.130:6838/2145479535,v1:172.21.15.130:6839/2145479535] up:active 2023-07-14T19:16:00.493 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:16:00 smithi130 ceph-mon[97832]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2023-07-14T19:16:00.493 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:16:00 smithi130 ceph-mon[97832]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2023-07-14T19:16:00.494 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:16:00 smithi130 ceph-mon[97832]: from='client.? 172.21.15.40:0/1679054317' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2023-07-14T19:16:00.494 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:16:00 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:16:00.494 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:16:00 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:16:00.494 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:16:00 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:16:00.494 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:16:00 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:16:00.494 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:16:00 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:16:00.494 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:16:00 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:16:00.495 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:16:00 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:16:00.495 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:16:00 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T19:16:00.495 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:16:00 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:16:00.577 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:00.576+0000 7f3d04ff9700 1 -- 172.21.15.40:0/1679054317 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v17) v1 ==== 118+0+0 (secure 0 0 0) 0x7f3cfc0974d0 con 0x7f3d080ffd80 2023-07-14T19:16:00.580 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:00.578+0000 7f3d0dac0700 1 -- 172.21.15.40:0/1679054317 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f3cf00a1150 msgr2=0x7f3cf00a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:16:00.580 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:00.578+0000 7f3d0dac0700 1 --2- 172.21.15.40:0/1679054317 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f3cf00a1150 0x7f3cf00a3600 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f3d080ffbe0 tx=0x7f3cf800a3b0 comp rx=0 tx=0).stop 2023-07-14T19:16:00.581 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:00.578+0000 7f3d0dac0700 1 -- 172.21.15.40:0/1679054317 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3d080ffd80 msgr2=0x7f3d08195e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:16:00.581 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:00.578+0000 7f3d0dac0700 1 --2- 172.21.15.40:0/1679054317 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3d080ffd80 0x7f3d08195e10 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f3cfc005770 tx=0x7f3cfc0057a0 comp rx=0 tx=0).stop 2023-07-14T19:16:00.581 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:00.578+0000 7f3d0dac0700 1 -- 172.21.15.40:0/1679054317 shutdown_connections 2023-07-14T19:16:00.581 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:00.578+0000 7f3d0dac0700 1 --2- 172.21.15.40:0/1679054317 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f3cf00a1150 0x7f3cf00a3600 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:00.581 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:00.578+0000 7f3d0dac0700 1 --2- 172.21.15.40:0/1679054317 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3d080feb90 0x7f3d081958d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:00.581 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:00.578+0000 7f3d0dac0700 1 --2- 172.21.15.40:0/1679054317 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3d08100710 0x7f3d0819ce80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:00.582 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:00.578+0000 7f3d0dac0700 1 --2- 172.21.15.40:0/1679054317 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3d080ffd80 0x7f3d08195e10 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:00.582 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:00.578+0000 7f3d0dac0700 1 -- 172.21.15.40:0/1679054317 >> 172.21.15.40:0/1679054317 conn(0x7f3d080fa1d0 msgr2=0x7f3d08068990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:16:00.582 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:00.579+0000 7f3d0dac0700 1 -- 172.21.15.40:0/1679054317 shutdown_connections 2023-07-14T19:16:00.582 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:00.579+0000 7f3d0dac0700 1 -- 172.21.15.40:0/1679054317 wait complete. 2023-07-14T19:16:00.593 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:16:00 smithi040 ceph-mon[96518]: pgmap v254: 97 pgs: 14 creating+peering, 55 unknown, 28 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s wr, 4 op/s 2023-07-14T19:16:00.593 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:16:00 smithi040 ceph-mon[96518]: osdmap e81: 12 total, 12 up, 12 in 2023-07-14T19:16:00.593 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:16:00 smithi040 ceph-mon[96518]: from='client.? 172.21.15.40:0/2609154881' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T19:16:00.593 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:16:00 smithi040 ceph-mon[96518]: mds.? [v2:172.21.15.130:6838/2145479535,v1:172.21.15.130:6839/2145479535] up:active 2023-07-14T19:16:00.594 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:16:00 smithi040 ceph-mon[96518]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2023-07-14T19:16:00.594 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:16:00 smithi040 ceph-mon[96518]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2023-07-14T19:16:00.594 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:16:00 smithi040 ceph-mon[96518]: from='client.? 172.21.15.40:0/1679054317' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2023-07-14T19:16:00.594 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:16:00 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:16:00.594 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:16:00 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:16:00.594 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:16:00 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:16:00.594 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:16:00 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:16:00.595 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:16:00 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:16:00.595 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:16:00 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:16:00.595 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:16:00 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:16:00.596 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:16:00 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T19:16:00.596 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:16:00 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:16:00.597 DEBUG:tasks.cephfs.filesystem:fs_config: {'max_mds': 5, 'session_timeout': 300, 'standby_replay': True, 'subvols': {'create': 2, 'subvol_options': '--namespace-isolated --size 25000000000'}} 2023-07-14T19:16:00.597 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs max_mds 5 2023-07-14T19:16:00.722 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:16:00 smithi191 ceph-mon[98311]: pgmap v254: 97 pgs: 14 creating+peering, 55 unknown, 28 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s wr, 4 op/s 2023-07-14T19:16:00.722 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:16:00 smithi191 ceph-mon[98311]: osdmap e81: 12 total, 12 up, 12 in 2023-07-14T19:16:00.722 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:16:00 smithi191 ceph-mon[98311]: from='client.? 172.21.15.40:0/2609154881' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T19:16:00.723 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:16:00 smithi191 ceph-mon[98311]: mds.? [v2:172.21.15.130:6838/2145479535,v1:172.21.15.130:6839/2145479535] up:active 2023-07-14T19:16:00.723 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:16:00 smithi191 ceph-mon[98311]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2023-07-14T19:16:00.723 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:16:00 smithi191 ceph-mon[98311]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2023-07-14T19:16:00.723 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:16:00 smithi191 ceph-mon[98311]: from='client.? 172.21.15.40:0/1679054317' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2023-07-14T19:16:00.723 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:16:00 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:16:00.723 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:16:00 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:16:00.723 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:16:00 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:16:00.724 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:16:00 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:16:00.724 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:16:00 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:16:00.724 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:16:00 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:16:00.724 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:16:00 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:16:00.724 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:16:00 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T19:16:00.724 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:16:00 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:16:00.729 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:00.727+0000 7f48d5eb1700 1 -- 172.21.15.40:0/1637923004 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f48d0102fb0 msgr2=0x7f48d0105390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:16:00.730 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:00.727+0000 7f48d5eb1700 1 --2- 172.21.15.40:0/1637923004 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f48d0102fb0 0x7f48d0105390 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f48c400b0a0 tx=0x7f48c400b3b0 comp rx=0 tx=0).stop 2023-07-14T19:16:00.730 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:00.727+0000 7f48d5eb1700 1 -- 172.21.15.40:0/1637923004 shutdown_connections 2023-07-14T19:16:00.730 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:00.727+0000 7f48d5eb1700 1 --2- 172.21.15.40:0/1637923004 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f48d01058d0 0x7f48d0107d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:00.730 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:00.727+0000 7f48d5eb1700 1 --2- 172.21.15.40:0/1637923004 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f48d0100690 0x7f48d0102a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:00.730 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:00.727+0000 7f48d5eb1700 1 --2- 172.21.15.40:0/1637923004 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f48d0102fb0 0x7f48d0105390 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:00.730 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:00.727+0000 7f48d5eb1700 1 -- 172.21.15.40:0/1637923004 >> 172.21.15.40:0/1637923004 conn(0x7f48d00fa130 msgr2=0x7f48d00fc560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:16:00.730 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:00.727+0000 7f48d5eb1700 1 -- 172.21.15.40:0/1637923004 shutdown_connections 2023-07-14T19:16:00.731 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:00.727+0000 7f48d5eb1700 1 -- 172.21.15.40:0/1637923004 wait complete. 2023-07-14T19:16:00.731 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:00.728+0000 7f48d5eb1700 1 Processor -- start 2023-07-14T19:16:00.732 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:00.728+0000 7f48d5eb1700 1 -- start start 2023-07-14T19:16:00.733 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:00.728+0000 7f48d5eb1700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f48d0100690 0x7f48d0197c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:16:00.733 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:00.728+0000 7f48d5eb1700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f48d0102fb0 0x7f48d01981d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:16:00.733 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:00.728+0000 7f48d5eb1700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f48d01058d0 0x7f48d019f230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:16:00.733 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:00.728+0000 7f48d5eb1700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48d019f800 con 0x7f48d01058d0 2023-07-14T19:16:00.733 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:00.728+0000 7f48d5eb1700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48d019f970 con 0x7f48d0100690 2023-07-14T19:16:00.733 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:00.728+0000 7f48d5eb1700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48d019fb30 con 0x7f48d0102fb0 2023-07-14T19:16:00.734 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:00.728+0000 7f48cffff700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f48d01058d0 0x7f48d019f230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:16:00.734 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:00.728+0000 7f48cffff700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f48d01058d0 0x7f48d019f230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:32902/0 (socket says 172.21.15.40:32902) 2023-07-14T19:16:00.734 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:00.728+0000 7f48cffff700 1 -- 172.21.15.40:0/1314310411 learned_addr learned my addr 172.21.15.40:0/1314310411 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-07-14T19:16:00.734 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:00.729+0000 7f48ceffd700 1 --2- 172.21.15.40:0/1314310411 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f48d0102fb0 0x7f48d01981d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:16:00.734 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:00.729+0000 7f48cf7fe700 1 --2- 172.21.15.40:0/1314310411 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f48d0100690 0x7f48d0197c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:16:00.734 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:00.729+0000 7f48cffff700 1 -- 172.21.15.40:0/1314310411 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f48d0102fb0 msgr2=0x7f48d01981d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:16:00.734 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:00.729+0000 7f48cffff700 1 --2- 172.21.15.40:0/1314310411 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f48d0102fb0 0x7f48d01981d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:00.734 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:00.729+0000 7f48cffff700 1 -- 172.21.15.40:0/1314310411 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f48d0100690 msgr2=0x7f48d0197c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:16:00.735 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:00.729+0000 7f48cffff700 1 --2- 172.21.15.40:0/1314310411 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f48d0100690 0x7f48d0197c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:00.735 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:00.729+0000 7f48cffff700 1 -- 172.21.15.40:0/1314310411 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48c4009d00 con 0x7f48d01058d0 2023-07-14T19:16:00.735 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:00.729+0000 7f48cffff700 1 --2- 172.21.15.40:0/1314310411 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f48d01058d0 0x7f48d019f230 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f48c800e940 tx=0x7f48c800ec50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:16:00.735 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:00.729+0000 7f48ccff9700 1 -- 172.21.15.40:0/1314310411 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48c80097b0 con 0x7f48d01058d0 2023-07-14T19:16:00.735 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:00.729+0000 7f48ccff9700 1 -- 172.21.15.40:0/1314310411 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f48c800ee90 con 0x7f48d01058d0 2023-07-14T19:16:00.735 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:00.729+0000 7f48d5eb1700 1 -- 172.21.15.40:0/1314310411 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f48d019fe10 con 0x7f48d01058d0 2023-07-14T19:16:00.735 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:00.729+0000 7f48ccff9700 1 -- 172.21.15.40:0/1314310411 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48c8005520 con 0x7f48d01058d0 2023-07-14T19:16:00.735 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:00.729+0000 7f48d5eb1700 1 -- 172.21.15.40:0/1314310411 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f48d01a0290 con 0x7f48d01058d0 2023-07-14T19:16:00.736 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:00.730+0000 7f48d5eb1700 1 -- 172.21.15.40:0/1314310411 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f48d0107490 con 0x7f48d01058d0 2023-07-14T19:16:00.736 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:00.731+0000 7f48ccff9700 1 -- 172.21.15.40:0/1314310411 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f48c8004750 con 0x7f48d01058d0 2023-07-14T19:16:00.736 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:00.732+0000 7f48ccff9700 1 --2- 172.21.15.40:0/1314310411 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f48b80a1150 0x7f48b80a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:16:00.737 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:00.732+0000 7f48cf7fe700 1 --2- 172.21.15.40:0/1314310411 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f48b80a1150 0x7f48b80a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:16:00.737 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:00.732+0000 7f48ccff9700 1 -- 172.21.15.40:0/1314310411 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f48c8014070 con 0x7f48d01058d0 2023-07-14T19:16:00.737 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:00.732+0000 7f48cf7fe700 1 --2- 172.21.15.40:0/1314310411 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f48b80a1150 0x7f48b80a3600 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f48d0198b60 tx=0x7f48c0009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:16:00.738 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:00.736+0000 7f48ccff9700 1 -- 172.21.15.40:0/1314310411 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f48c8096240 con 0x7f48d01058d0 2023-07-14T19:16:00.984 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:00.982+0000 7f48d5eb1700 1 -- 172.21.15.40:0/1314310411 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"} v 0) v1 -- 0x7f48d002cb30 con 0x7f48d01058d0 2023-07-14T19:16:01.563 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:01.561+0000 7f48ccff9700 1 -- 172.21.15.40:0/1314310411 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]=0 v18) v1 ==== 105+0+0 (secure 0 0 0) 0x7f48c8099860 con 0x7f48d01058d0 2023-07-14T19:16:01.566 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:01.564+0000 7f48d5eb1700 1 -- 172.21.15.40:0/1314310411 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f48b80a1150 msgr2=0x7f48b80a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:16:01.566 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:01.564+0000 7f48d5eb1700 1 --2- 172.21.15.40:0/1314310411 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f48b80a1150 0x7f48b80a3600 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f48d0198b60 tx=0x7f48c0009260 comp rx=0 tx=0).stop 2023-07-14T19:16:01.566 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:01.564+0000 7f48d5eb1700 1 -- 172.21.15.40:0/1314310411 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f48d01058d0 msgr2=0x7f48d019f230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:16:01.566 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:01.564+0000 7f48d5eb1700 1 --2- 172.21.15.40:0/1314310411 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f48d01058d0 0x7f48d019f230 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f48c800e940 tx=0x7f48c800ec50 comp rx=0 tx=0).stop 2023-07-14T19:16:01.567 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:01.564+0000 7f48d5eb1700 1 -- 172.21.15.40:0/1314310411 shutdown_connections 2023-07-14T19:16:01.567 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:01.564+0000 7f48d5eb1700 1 --2- 172.21.15.40:0/1314310411 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f48b80a1150 0x7f48b80a3600 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:01.567 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:01.564+0000 7f48d5eb1700 1 --2- 172.21.15.40:0/1314310411 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f48d01058d0 0x7f48d019f230 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:01.567 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:01.564+0000 7f48d5eb1700 1 --2- 172.21.15.40:0/1314310411 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f48d0100690 0x7f48d0197c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:01.567 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:01.564+0000 7f48d5eb1700 1 --2- 172.21.15.40:0/1314310411 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f48d0102fb0 0x7f48d01981d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:01.567 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:01.564+0000 7f48d5eb1700 1 -- 172.21.15.40:0/1314310411 >> 172.21.15.40:0/1314310411 conn(0x7f48d00fa130 msgr2=0x7f48d00fed10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:16:01.567 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:01.564+0000 7f48d5eb1700 1 -- 172.21.15.40:0/1314310411 shutdown_connections 2023-07-14T19:16:01.568 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:01.564+0000 7f48d5eb1700 1 -- 172.21.15.40:0/1314310411 wait complete. 2023-07-14T19:16:01.585 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay true 2023-07-14T19:16:01.721 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:01.719+0000 7f4234bdb700 1 -- 172.21.15.40:0/3272651866 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f42300ffd00 msgr2=0x7f4230100150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:16:01.721 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:01.719+0000 7f4234bdb700 1 --2- 172.21.15.40:0/3272651866 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f42300ffd00 0x7f4230100150 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f4218009990 tx=0x7f4218009ca0 comp rx=0 tx=0).stop 2023-07-14T19:16:01.721 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:01.719+0000 7f4234bdb700 1 -- 172.21.15.40:0/3272651866 shutdown_connections 2023-07-14T19:16:01.721 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:01.719+0000 7f4234bdb700 1 --2- 172.21.15.40:0/3272651866 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f42300feb10 0x7f42300fef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:01.721 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:01.719+0000 7f4234bdb700 1 --2- 172.21.15.40:0/3272651866 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4230100690 0x7f4230106f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:01.722 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:01.719+0000 7f4234bdb700 1 --2- 172.21.15.40:0/3272651866 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f42300ffd00 0x7f4230100150 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:01.722 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:01.719+0000 7f4234bdb700 1 -- 172.21.15.40:0/3272651866 >> 172.21.15.40:0/3272651866 conn(0x7f42300fa130 msgr2=0x7f42300fc560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:16:01.722 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:01.719+0000 7f4234bdb700 1 -- 172.21.15.40:0/3272651866 shutdown_connections 2023-07-14T19:16:01.722 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:01.719+0000 7f4234bdb700 1 -- 172.21.15.40:0/3272651866 wait complete. 2023-07-14T19:16:01.724 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:01.720+0000 7f4234bdb700 1 Processor -- start 2023-07-14T19:16:01.724 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:01.720+0000 7f4234bdb700 1 -- start start 2023-07-14T19:16:01.724 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:01.720+0000 7f4234bdb700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f42300feb10 0x7f4230195970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:16:01.724 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:01.720+0000 7f4234bdb700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f42300ffd00 0x7f4230195eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:16:01.724 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:01.720+0000 7f4234bdb700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4230100690 0x7f423019cf20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:16:01.724 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:01.720+0000 7f4234bdb700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f423019d4f0 con 0x7f4230100690 2023-07-14T19:16:01.725 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:01.720+0000 7f4234bdb700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f423019d630 con 0x7f42300feb10 2023-07-14T19:16:01.725 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:01.720+0000 7f4234bdb700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f423019d7c0 con 0x7f42300ffd00 2023-07-14T19:16:01.725 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:01.720+0000 7f422ed9d700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4230100690 0x7f423019cf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:16:01.725 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:01.720+0000 7f422ed9d700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4230100690 0x7f423019cf20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:32916/0 (socket says 172.21.15.40:32916) 2023-07-14T19:16:01.725 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:01.720+0000 7f422ed9d700 1 -- 172.21.15.40:0/3571769864 learned_addr learned my addr 172.21.15.40:0/3571769864 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-07-14T19:16:01.725 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:01.720+0000 7f422dd9b700 1 --2- 172.21.15.40:0/3571769864 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f42300ffd00 0x7f4230195eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:16:01.725 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:01.720+0000 7f422e59c700 1 --2- 172.21.15.40:0/3571769864 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f42300feb10 0x7f4230195970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:16:01.726 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:01.720+0000 7f422ed9d700 1 -- 172.21.15.40:0/3571769864 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f42300ffd00 msgr2=0x7f4230195eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:16:01.726 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:01.720+0000 7f422ed9d700 1 --2- 172.21.15.40:0/3571769864 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f42300ffd00 0x7f4230195eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:01.726 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:01.720+0000 7f422ed9d700 1 -- 172.21.15.40:0/3571769864 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f42300feb10 msgr2=0x7f4230195970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:16:01.726 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:01.720+0000 7f422ed9d700 1 --2- 172.21.15.40:0/3571769864 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f42300feb10 0x7f4230195970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:01.726 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:01.720+0000 7f422ed9d700 1 -- 172.21.15.40:0/3571769864 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42180095f0 con 0x7f4230100690 2023-07-14T19:16:01.726 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:01.720+0000 7f422ed9d700 1 --2- 172.21.15.40:0/3571769864 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4230100690 0x7f423019cf20 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f4228007ee0 tx=0x7f422800d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:16:01.726 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:01.721+0000 7f42277fe700 1 -- 172.21.15.40:0/3571769864 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4228017070 con 0x7f4230100690 2023-07-14T19:16:01.726 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:01.721+0000 7f42277fe700 1 -- 172.21.15.40:0/3571769864 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f422800f040 con 0x7f4230100690 2023-07-14T19:16:01.727 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:01.721+0000 7f4234bdb700 1 -- 172.21.15.40:0/3571769864 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f423019daa0 con 0x7f4230100690 2023-07-14T19:16:01.727 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:01.721+0000 7f42277fe700 1 -- 172.21.15.40:0/3571769864 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4228005740 con 0x7f4230100690 2023-07-14T19:16:01.727 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:01.721+0000 7f4234bdb700 1 -- 172.21.15.40:0/3571769864 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f423019e000 con 0x7f4230100690 2023-07-14T19:16:01.727 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:01.722+0000 7f4234bdb700 1 -- 172.21.15.40:0/3571769864 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4230106c40 con 0x7f4230100690 2023-07-14T19:16:01.728 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:01.723+0000 7f42277fe700 1 -- 172.21.15.40:0/3571769864 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f422800dc60 con 0x7f4230100690 2023-07-14T19:16:01.728 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:01.723+0000 7f42277fe700 1 --2- 172.21.15.40:0/3571769864 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f421c0a11d0 0x7f421c0a3680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:16:01.728 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:01.723+0000 7f42277fe700 1 -- 172.21.15.40:0/3571769864 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f42280cbf10 con 0x7f4230100690 2023-07-14T19:16:01.728 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:01.724+0000 7f422e59c700 1 --2- 172.21.15.40:0/3571769864 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f421c0a11d0 0x7f421c0a3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:16:01.728 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:01.724+0000 7f422e59c700 1 --2- 172.21.15.40:0/3571769864 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f421c0a11d0 0x7f421c0a3680 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f4220005fd0 tx=0x7f4220005dd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:16:01.729 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:01.728+0000 7f42277fe700 1 -- 172.21.15.40:0/3571769864 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f42280957b0 con 0x7f4230100690 2023-07-14T19:16:01.971 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:16:01 smithi040 ceph-mon[96518]: from='client.? 172.21.15.40:0/1679054317' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2023-07-14T19:16:01.971 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:16:01 smithi040 ceph-mon[96518]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2023-07-14T19:16:01.972 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:16:01 smithi040 ceph-mon[96518]: from='client.? 172.21.15.40:0/1314310411' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]: dispatch 2023-07-14T19:16:01.972 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:16:01 smithi040 ceph-mon[96518]: daemon mds.i assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2023-07-14T19:16:01.972 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:16:01 smithi040 ceph-mon[96518]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2023-07-14T19:16:01.973 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:16:01 smithi040 ceph-mon[96518]: from='client.? 172.21.15.40:0/1314310411' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]': finished 2023-07-14T19:16:01.973 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:16:01 smithi040 ceph-mon[96518]: fsmap cephfs:2 {0=b=up:active,1=i=up:creating} 10 up:standby 2023-07-14T19:16:01.974 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:16:01 smithi191 ceph-mon[98311]: from='client.? 172.21.15.40:0/1679054317' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2023-07-14T19:16:01.974 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:16:01 smithi191 ceph-mon[98311]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2023-07-14T19:16:01.974 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:16:01 smithi191 ceph-mon[98311]: from='client.? 172.21.15.40:0/1314310411' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]: dispatch 2023-07-14T19:16:01.974 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:16:01 smithi191 ceph-mon[98311]: daemon mds.i assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2023-07-14T19:16:01.975 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:16:01 smithi191 ceph-mon[98311]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2023-07-14T19:16:01.975 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:16:01 smithi191 ceph-mon[98311]: from='client.? 172.21.15.40:0/1314310411' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]': finished 2023-07-14T19:16:01.975 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:16:01 smithi191 ceph-mon[98311]: fsmap cephfs:2 {0=b=up:active,1=i=up:creating} 10 up:standby 2023-07-14T19:16:01.977 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:01.975+0000 7f4234bdb700 1 -- 172.21.15.40:0/3571769864 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"} v 0) v1 -- 0x7f423002c8e0 con 0x7f4230100690 2023-07-14T19:16:01.992 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:16:01 smithi130 ceph-mon[97832]: from='client.? 172.21.15.40:0/1679054317' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2023-07-14T19:16:01.992 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:16:01 smithi130 ceph-mon[97832]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2023-07-14T19:16:01.992 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:16:01 smithi130 ceph-mon[97832]: from='client.? 172.21.15.40:0/1314310411' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]: dispatch 2023-07-14T19:16:01.993 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:16:01 smithi130 ceph-mon[97832]: daemon mds.i assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2023-07-14T19:16:01.993 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:16:01 smithi130 ceph-mon[97832]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2023-07-14T19:16:01.993 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:16:01 smithi130 ceph-mon[97832]: from='client.? 172.21.15.40:0/1314310411' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]': finished 2023-07-14T19:16:01.993 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:16:01 smithi130 ceph-mon[97832]: fsmap cephfs:2 {0=b=up:active,1=i=up:creating} 10 up:standby 2023-07-14T19:16:02.581 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:02.580+0000 7f42277fe700 1 -- 172.21.15.40:0/3571769864 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]=0 v19) v1 ==== 121+0+0 (secure 0 0 0) 0x7f4228098dd0 con 0x7f4230100690 2023-07-14T19:16:02.584 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:02.583+0000 7f4234bdb700 1 -- 172.21.15.40:0/3571769864 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f421c0a11d0 msgr2=0x7f421c0a3680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:16:02.585 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:02.583+0000 7f4234bdb700 1 --2- 172.21.15.40:0/3571769864 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f421c0a11d0 0x7f421c0a3680 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f4220005fd0 tx=0x7f4220005dd0 comp rx=0 tx=0).stop 2023-07-14T19:16:02.585 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:02.583+0000 7f4234bdb700 1 -- 172.21.15.40:0/3571769864 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4230100690 msgr2=0x7f423019cf20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:16:02.585 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:02.583+0000 7f4234bdb700 1 --2- 172.21.15.40:0/3571769864 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4230100690 0x7f423019cf20 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f4228007ee0 tx=0x7f422800d3b0 comp rx=0 tx=0).stop 2023-07-14T19:16:02.585 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:02.583+0000 7f4234bdb700 1 -- 172.21.15.40:0/3571769864 shutdown_connections 2023-07-14T19:16:02.585 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:02.583+0000 7f4234bdb700 1 --2- 172.21.15.40:0/3571769864 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f421c0a11d0 0x7f421c0a3680 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:02.585 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:02.583+0000 7f4234bdb700 1 --2- 172.21.15.40:0/3571769864 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4230100690 0x7f423019cf20 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:02.586 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:02.583+0000 7f4234bdb700 1 --2- 172.21.15.40:0/3571769864 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f42300feb10 0x7f4230195970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:02.586 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:02.583+0000 7f4234bdb700 1 --2- 172.21.15.40:0/3571769864 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f42300ffd00 0x7f4230195eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:02.586 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:02.583+0000 7f4234bdb700 1 -- 172.21.15.40:0/3571769864 >> 172.21.15.40:0/3571769864 conn(0x7f42300fa130 msgr2=0x7f42300fc3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:16:02.586 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:02.583+0000 7f4234bdb700 1 -- 172.21.15.40:0/3571769864 shutdown_connections 2023-07-14T19:16:02.586 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:02.583+0000 7f4234bdb700 1 -- 172.21.15.40:0/3571769864 wait complete. 2023-07-14T19:16:02.605 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs session_timeout 300 2023-07-14T19:16:02.742 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:02.740+0000 7f3aacc06700 1 -- 172.21.15.40:0/1717103929 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3aa80ffd00 msgr2=0x7f3aa8100150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:16:02.742 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:02.740+0000 7f3aacc06700 1 --2- 172.21.15.40:0/1717103929 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3aa80ffd00 0x7f3aa8100150 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f3a9c009990 tx=0x7f3a9c009ca0 comp rx=0 tx=0).stop 2023-07-14T19:16:02.742 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:02.740+0000 7f3aacc06700 1 -- 172.21.15.40:0/1717103929 shutdown_connections 2023-07-14T19:16:02.742 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:02.740+0000 7f3aacc06700 1 --2- 172.21.15.40:0/1717103929 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3aa80feb10 0x7f3aa80fef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:02.742 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:02.740+0000 7f3aacc06700 1 --2- 172.21.15.40:0/1717103929 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3aa8100690 0x7f3aa8106f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:02.743 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:02.740+0000 7f3aacc06700 1 --2- 172.21.15.40:0/1717103929 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3aa80ffd00 0x7f3aa8100150 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:02.743 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:02.740+0000 7f3aacc06700 1 -- 172.21.15.40:0/1717103929 >> 172.21.15.40:0/1717103929 conn(0x7f3aa80fa130 msgr2=0x7f3aa80fc560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:16:02.743 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:02.740+0000 7f3aacc06700 1 -- 172.21.15.40:0/1717103929 shutdown_connections 2023-07-14T19:16:02.743 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:02.740+0000 7f3aacc06700 1 -- 172.21.15.40:0/1717103929 wait complete. 2023-07-14T19:16:02.745 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:02.740+0000 7f3aacc06700 1 Processor -- start 2023-07-14T19:16:02.745 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:02.741+0000 7f3aacc06700 1 -- start start 2023-07-14T19:16:02.745 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:02.741+0000 7f3aacc06700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3aa80feb10 0x7f3aa81959b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:16:02.745 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:02.741+0000 7f3aacc06700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3aa80ffd00 0x7f3aa8195ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:16:02.745 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:02.741+0000 7f3aacc06700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3aa8100690 0x7f3aa819cf60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:16:02.745 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:02.741+0000 7f3aacc06700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3aa819d530 con 0x7f3aa80feb10 2023-07-14T19:16:02.746 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:02.741+0000 7f3aacc06700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3aa819d670 con 0x7f3aa80ffd00 2023-07-14T19:16:02.746 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:02.741+0000 7f3aacc06700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3aa819d800 con 0x7f3aa8100690 2023-07-14T19:16:02.746 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:02.741+0000 7f3aa659c700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3aa80feb10 0x7f3aa81959b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:16:02.746 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:02.741+0000 7f3aa659c700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3aa80feb10 0x7f3aa81959b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:32926/0 (socket says 172.21.15.40:32926) 2023-07-14T19:16:02.746 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:02.741+0000 7f3aa659c700 1 -- 172.21.15.40:0/1485056403 learned_addr learned my addr 172.21.15.40:0/1485056403 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-07-14T19:16:02.746 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:02.741+0000 7f3aa6d9d700 1 --2- 172.21.15.40:0/1485056403 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3aa8100690 0x7f3aa819cf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:16:02.746 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:02.741+0000 7f3aa5d9b700 1 --2- 172.21.15.40:0/1485056403 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3aa80ffd00 0x7f3aa8195ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:16:02.747 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:02.741+0000 7f3aa659c700 1 -- 172.21.15.40:0/1485056403 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3aa8100690 msgr2=0x7f3aa819cf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:16:02.747 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:02.741+0000 7f3aa659c700 1 --2- 172.21.15.40:0/1485056403 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3aa8100690 0x7f3aa819cf60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:02.747 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:02.741+0000 7f3aa659c700 1 -- 172.21.15.40:0/1485056403 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3aa80ffd00 msgr2=0x7f3aa8195ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:16:02.747 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:02.741+0000 7f3aa659c700 1 --2- 172.21.15.40:0/1485056403 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3aa80ffd00 0x7f3aa8195ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:02.747 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:02.741+0000 7f3aa659c700 1 -- 172.21.15.40:0/1485056403 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a9c0095f0 con 0x7f3aa80feb10 2023-07-14T19:16:02.747 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:02.741+0000 7f3aa659c700 1 --2- 172.21.15.40:0/1485056403 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3aa80feb10 0x7f3aa81959b0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f3a9800d980 tx=0x7f3a9800dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:16:02.747 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:02.741+0000 7f3a977fe700 1 -- 172.21.15.40:0/1485056403 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a98004020 con 0x7f3aa80feb10 2023-07-14T19:16:02.747 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:02.742+0000 7f3aacc06700 1 -- 172.21.15.40:0/1485056403 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3aa819dae0 con 0x7f3aa80feb10 2023-07-14T19:16:02.748 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:02.742+0000 7f3a977fe700 1 -- 172.21.15.40:0/1485056403 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3a9800f690 con 0x7f3aa80feb10 2023-07-14T19:16:02.748 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:02.742+0000 7f3a977fe700 1 -- 172.21.15.40:0/1485056403 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a98003a60 con 0x7f3aa80feb10 2023-07-14T19:16:02.748 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:02.742+0000 7f3aacc06700 1 -- 172.21.15.40:0/1485056403 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3aa819e040 con 0x7f3aa80feb10 2023-07-14T19:16:02.748 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:02.742+0000 7f3aacc06700 1 -- 172.21.15.40:0/1485056403 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3aa81060b0 con 0x7f3aa80feb10 2023-07-14T19:16:02.749 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:02.744+0000 7f3a977fe700 1 -- 172.21.15.40:0/1485056403 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f3a9800fb00 con 0x7f3aa80feb10 2023-07-14T19:16:02.749 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:02.744+0000 7f3a977fe700 1 --2- 172.21.15.40:0/1485056403 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f3a900a1150 0x7f3a900a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:16:02.749 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:02.744+0000 7f3a977fe700 1 -- 172.21.15.40:0/1485056403 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f3a980cc6f0 con 0x7f3aa80feb10 2023-07-14T19:16:02.749 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:02.744+0000 7f3aa5d9b700 1 --2- 172.21.15.40:0/1485056403 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f3a900a1150 0x7f3a900a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:16:02.749 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:02.745+0000 7f3aa5d9b700 1 --2- 172.21.15.40:0/1485056403 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f3a900a1150 0x7f3a900a3600 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f3a9c00b5c0 tx=0x7f3a9c005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:16:02.750 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:02.748+0000 7f3a977fe700 1 -- 172.21.15.40:0/1485056403 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f3a98095f90 con 0x7f3aa80feb10 2023-07-14T19:16:02.971 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:16:02 smithi040 ceph-mon[96518]: pgmap v256: 97 pgs: 14 creating+peering, 16 unknown, 67 active+clean; 579 KiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s wr, 8 op/s 2023-07-14T19:16:02.971 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:16:02 smithi040 ceph-mon[96518]: daemon mds.i is now active in filesystem cephfs as rank 1 2023-07-14T19:16:02.971 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:16:02 smithi040 ceph-mon[96518]: from='client.? 172.21.15.40:0/3571769864' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2023-07-14T19:16:02.972 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:16:02 smithi191 ceph-mon[98311]: pgmap v256: 97 pgs: 14 creating+peering, 16 unknown, 67 active+clean; 579 KiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s wr, 8 op/s 2023-07-14T19:16:02.972 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:16:02 smithi191 ceph-mon[98311]: daemon mds.i is now active in filesystem cephfs as rank 1 2023-07-14T19:16:02.972 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:16:02 smithi191 ceph-mon[98311]: from='client.? 172.21.15.40:0/3571769864' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2023-07-14T19:16:02.992 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:16:02 smithi130 ceph-mon[97832]: pgmap v256: 97 pgs: 14 creating+peering, 16 unknown, 67 active+clean; 579 KiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s wr, 8 op/s 2023-07-14T19:16:02.993 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:16:02 smithi130 ceph-mon[97832]: daemon mds.i is now active in filesystem cephfs as rank 1 2023-07-14T19:16:02.993 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:16:02 smithi130 ceph-mon[97832]: from='client.? 172.21.15.40:0/3571769864' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2023-07-14T19:16:02.996 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:02.994+0000 7f3aacc06700 1 -- 172.21.15.40:0/1485056403 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"} v 0) v1 -- 0x7f3aa802c8e0 con 0x7f3aa80feb10 2023-07-14T19:16:03.601 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:03.599+0000 7f3a977fe700 1 -- 172.21.15.40:0/1485056403 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]=0 v21) v1 ==== 115+0+0 (secure 0 0 0) 0x7f3a980995b0 con 0x7f3aa80feb10 2023-07-14T19:16:03.605 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:03.603+0000 7f3aacc06700 1 -- 172.21.15.40:0/1485056403 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f3a900a1150 msgr2=0x7f3a900a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:16:03.605 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:03.603+0000 7f3aacc06700 1 --2- 172.21.15.40:0/1485056403 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f3a900a1150 0x7f3a900a3600 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f3a9c00b5c0 tx=0x7f3a9c005fb0 comp rx=0 tx=0).stop 2023-07-14T19:16:03.605 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:03.603+0000 7f3aacc06700 1 -- 172.21.15.40:0/1485056403 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3aa80feb10 msgr2=0x7f3aa81959b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:16:03.605 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:03.603+0000 7f3aacc06700 1 --2- 172.21.15.40:0/1485056403 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3aa80feb10 0x7f3aa81959b0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f3a9800d980 tx=0x7f3a9800dc90 comp rx=0 tx=0).stop 2023-07-14T19:16:03.605 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:03.603+0000 7f3aacc06700 1 -- 172.21.15.40:0/1485056403 shutdown_connections 2023-07-14T19:16:03.606 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:03.603+0000 7f3aacc06700 1 --2- 172.21.15.40:0/1485056403 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f3a900a1150 0x7f3a900a3600 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:03.606 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:03.603+0000 7f3aacc06700 1 --2- 172.21.15.40:0/1485056403 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3aa8100690 0x7f3aa819cf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:03.606 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:03.603+0000 7f3aacc06700 1 --2- 172.21.15.40:0/1485056403 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3aa80feb10 0x7f3aa81959b0 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:03.606 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:03.603+0000 7f3aacc06700 1 --2- 172.21.15.40:0/1485056403 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3aa80ffd00 0x7f3aa8195ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:03.606 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:03.603+0000 7f3aacc06700 1 -- 172.21.15.40:0/1485056403 >> 172.21.15.40:0/1485056403 conn(0x7f3aa80fa130 msgr2=0x7f3aa81031f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:16:03.606 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:03.603+0000 7f3aacc06700 1 -- 172.21.15.40:0/1485056403 shutdown_connections 2023-07-14T19:16:03.606 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:03.603+0000 7f3aacc06700 1 -- 172.21.15.40:0/1485056403 wait complete. 2023-07-14T19:16:03.624 DEBUG:tasks.cephfs.filesystem:Creating {'create': 2, 'subvol_options': '--namespace-isolated --size 25000000000'} subvols for filesystem 'cephfs' 2023-07-14T19:16:03.625 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume create cephfs sv_0 '' 2023-07-14T19:16:03.762 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:03.759+0000 7f24c9a9f700 1 -- 172.21.15.40:0/3408440527 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f24c40feb10 msgr2=0x7f24c40fef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:16:03.763 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:03.759+0000 7f24c9a9f700 1 --2- 172.21.15.40:0/3408440527 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f24c40feb10 0x7f24c40fef20 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f24b4009990 tx=0x7f24b4009ca0 comp rx=0 tx=0).stop 2023-07-14T19:16:03.763 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:03.759+0000 7f24c9a9f700 1 -- 172.21.15.40:0/3408440527 shutdown_connections 2023-07-14T19:16:03.763 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:03.759+0000 7f24c9a9f700 1 --2- 172.21.15.40:0/3408440527 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f24c4100690 0x7f24c4106f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:03.763 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:03.759+0000 7f24c9a9f700 1 --2- 172.21.15.40:0/3408440527 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f24c40feb10 0x7f24c40fef20 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:03.763 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:03.759+0000 7f24c9a9f700 1 --2- 172.21.15.40:0/3408440527 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f24c40ffd00 0x7f24c4100150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:03.764 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:03.759+0000 7f24c9a9f700 1 -- 172.21.15.40:0/3408440527 >> 172.21.15.40:0/3408440527 conn(0x7f24c40fa130 msgr2=0x7f24c40fc560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:16:03.764 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:03.759+0000 7f24c9a9f700 1 -- 172.21.15.40:0/3408440527 shutdown_connections 2023-07-14T19:16:03.764 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:03.759+0000 7f24c9a9f700 1 -- 172.21.15.40:0/3408440527 wait complete. 2023-07-14T19:16:03.764 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:03.760+0000 7f24c9a9f700 1 Processor -- start 2023-07-14T19:16:03.764 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:03.760+0000 7f24c9a9f700 1 -- start start 2023-07-14T19:16:03.764 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:03.760+0000 7f24c9a9f700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f24c40feb10 0x7f24c4194840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:16:03.764 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:03.760+0000 7f24c9a9f700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f24c40ffd00 0x7f24c4194d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:16:03.765 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:03.760+0000 7f24c9a9f700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f24c4100690 0x7f24c419bdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:16:03.765 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:03.760+0000 7f24c9a9f700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24c419c3c0 con 0x7f24c40feb10 2023-07-14T19:16:03.765 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:03.760+0000 7f24c9a9f700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24c419c500 con 0x7f24c40ffd00 2023-07-14T19:16:03.765 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:03.760+0000 7f24c9a9f700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24c419c690 con 0x7f24c4100690 2023-07-14T19:16:03.765 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:03.760+0000 7f24c37fe700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f24c40feb10 0x7f24c4194840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:16:03.765 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:03.760+0000 7f24c37fe700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f24c40feb10 0x7f24c4194840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:32940/0 (socket says 172.21.15.40:32940) 2023-07-14T19:16:03.765 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:03.760+0000 7f24c37fe700 1 -- 172.21.15.40:0/2212747029 learned_addr learned my addr 172.21.15.40:0/2212747029 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-07-14T19:16:03.765 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:03.760+0000 7f24c2ffd700 1 --2- 172.21.15.40:0/2212747029 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f24c40ffd00 0x7f24c4194d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:16:03.766 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:03.761+0000 7f24c3fff700 1 --2- 172.21.15.40:0/2212747029 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f24c4100690 0x7f24c419bdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:16:03.766 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:03.761+0000 7f24c37fe700 1 -- 172.21.15.40:0/2212747029 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f24c4100690 msgr2=0x7f24c419bdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:16:03.766 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:03.761+0000 7f24c37fe700 1 --2- 172.21.15.40:0/2212747029 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f24c4100690 0x7f24c419bdf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:03.766 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:03.761+0000 7f24c37fe700 1 -- 172.21.15.40:0/2212747029 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f24c40ffd00 msgr2=0x7f24c4194d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:16:03.766 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:03.761+0000 7f24c37fe700 1 --2- 172.21.15.40:0/2212747029 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f24c40ffd00 0x7f24c4194d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:03.766 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:03.761+0000 7f24c37fe700 1 -- 172.21.15.40:0/2212747029 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24b40095f0 con 0x7f24c40feb10 2023-07-14T19:16:03.766 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:03.761+0000 7f24c37fe700 1 --2- 172.21.15.40:0/2212747029 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f24c40feb10 0x7f24c4194840 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f24b4000c00 tx=0x7f24b40047b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:16:03.767 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:03.761+0000 7f24c0ff9700 1 -- 172.21.15.40:0/2212747029 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24b401d070 con 0x7f24c40feb10 2023-07-14T19:16:03.767 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:03.761+0000 7f24c9a9f700 1 -- 172.21.15.40:0/2212747029 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f24c419c910 con 0x7f24c40feb10 2023-07-14T19:16:03.767 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:03.761+0000 7f24c0ff9700 1 -- 172.21.15.40:0/2212747029 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f24b4004060 con 0x7f24c40feb10 2023-07-14T19:16:03.767 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:03.761+0000 7f24c0ff9700 1 -- 172.21.15.40:0/2212747029 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24b4022d30 con 0x7f24c40feb10 2023-07-14T19:16:03.767 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:03.761+0000 7f24c9a9f700 1 -- 172.21.15.40:0/2212747029 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f24c419ce10 con 0x7f24c40feb10 2023-07-14T19:16:03.768 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:03.762+0000 7f24c9a9f700 1 -- 172.21.15.40:0/2212747029 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f24c4105a90 con 0x7f24c40feb10 2023-07-14T19:16:03.768 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:03.763+0000 7f24c0ff9700 1 -- 172.21.15.40:0/2212747029 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f24b4003ba0 con 0x7f24c40feb10 2023-07-14T19:16:03.768 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:03.764+0000 7f24c0ff9700 1 --2- 172.21.15.40:0/2212747029 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f24ac0a1150 0x7f24ac0a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:16:03.768 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:03.764+0000 7f24c2ffd700 1 --2- 172.21.15.40:0/2212747029 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f24ac0a1150 0x7f24ac0a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:16:03.768 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:03.764+0000 7f24c0ff9700 1 -- 172.21.15.40:0/2212747029 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f24b40cdee0 con 0x7f24c40feb10 2023-07-14T19:16:03.769 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:03.764+0000 7f24c2ffd700 1 --2- 172.21.15.40:0/2212747029 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f24ac0a1150 0x7f24ac0a3600 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f24b8005950 tx=0x7f24b800b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:16:03.769 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:03.767+0000 7f24c0ff9700 1 -- 172.21.15.40:0/2212747029 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f24b4097770 con 0x7f24c40feb10 2023-07-14T19:16:03.856 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:16:03 smithi130 ceph-mon[97832]: mds.? [v2:172.21.15.191:6838/1977507302,v1:172.21.15.191:6839/1977507302] up:active 2023-07-14T19:16:03.856 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:16:03 smithi130 ceph-mon[97832]: from='client.? 172.21.15.40:0/3571769864' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2023-07-14T19:16:03.856 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:16:03 smithi130 ceph-mon[97832]: daemon mds.h assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2023-07-14T19:16:03.857 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:16:03 smithi130 ceph-mon[97832]: fsmap cephfs:2 {0=b=up:active,1=i=up:active} 10 up:standby 2023-07-14T19:16:03.857 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:16:03 smithi130 ceph-mon[97832]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:creating} 9 up:standby 2023-07-14T19:16:03.857 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:16:03 smithi130 ceph-mon[97832]: daemon mds.h is now active in filesystem cephfs as rank 2 2023-07-14T19:16:03.857 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:16:03 smithi130 ceph-mon[97832]: from='client.? 172.21.15.40:0/1485056403' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2023-07-14T19:16:03.971 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:16:03 smithi040 ceph-mon[96518]: mds.? [v2:172.21.15.191:6838/1977507302,v1:172.21.15.191:6839/1977507302] up:active 2023-07-14T19:16:03.971 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:16:03 smithi040 ceph-mon[96518]: from='client.? 172.21.15.40:0/3571769864' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2023-07-14T19:16:03.971 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:16:03 smithi040 ceph-mon[96518]: daemon mds.h assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2023-07-14T19:16:03.971 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:16:03 smithi040 ceph-mon[96518]: fsmap cephfs:2 {0=b=up:active,1=i=up:active} 10 up:standby 2023-07-14T19:16:03.971 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:16:03 smithi040 ceph-mon[96518]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:creating} 9 up:standby 2023-07-14T19:16:03.971 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:16:03 smithi040 ceph-mon[96518]: daemon mds.h is now active in filesystem cephfs as rank 2 2023-07-14T19:16:03.972 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:16:03 smithi040 ceph-mon[96518]: from='client.? 172.21.15.40:0/1485056403' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2023-07-14T19:16:03.972 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:16:03 smithi191 ceph-mon[98311]: mds.? [v2:172.21.15.191:6838/1977507302,v1:172.21.15.191:6839/1977507302] up:active 2023-07-14T19:16:03.972 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:16:03 smithi191 ceph-mon[98311]: from='client.? 172.21.15.40:0/3571769864' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2023-07-14T19:16:03.972 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:16:03 smithi191 ceph-mon[98311]: daemon mds.h assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2023-07-14T19:16:03.972 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:16:03 smithi191 ceph-mon[98311]: fsmap cephfs:2 {0=b=up:active,1=i=up:active} 10 up:standby 2023-07-14T19:16:03.973 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:16:03 smithi191 ceph-mon[98311]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:creating} 9 up:standby 2023-07-14T19:16:03.973 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:16:03 smithi191 ceph-mon[98311]: daemon mds.h is now active in filesystem cephfs as rank 2 2023-07-14T19:16:03.973 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:16:03 smithi191 ceph-mon[98311]: from='client.? 172.21.15.40:0/1485056403' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2023-07-14T19:16:04.001 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:03.999+0000 7f24c9a9f700 1 -- 172.21.15.40:0/2212747029 --> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "", "target": ["mon-mgr", ""]}) v1 -- 0x7f24c4060b10 con 0x7f24ac0a1150 2023-07-14T19:16:04.036 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:04.035+0000 7f24c0ff9700 1 -- 172.21.15.40:0/2212747029 <== mgr.14152 v2:172.21.15.40:6800/1693027118 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f24c4060b10 con 0x7f24ac0a1150 2023-07-14T19:16:04.039 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:04.037+0000 7f24c9a9f700 1 -- 172.21.15.40:0/2212747029 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f24ac0a1150 msgr2=0x7f24ac0a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:16:04.039 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:04.037+0000 7f24c9a9f700 1 --2- 172.21.15.40:0/2212747029 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f24ac0a1150 0x7f24ac0a3600 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f24b8005950 tx=0x7f24b800b410 comp rx=0 tx=0).stop 2023-07-14T19:16:04.039 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:04.037+0000 7f24c9a9f700 1 -- 172.21.15.40:0/2212747029 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f24c40feb10 msgr2=0x7f24c4194840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:16:04.039 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:04.037+0000 7f24c9a9f700 1 --2- 172.21.15.40:0/2212747029 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f24c40feb10 0x7f24c4194840 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f24b4000c00 tx=0x7f24b40047b0 comp rx=0 tx=0).stop 2023-07-14T19:16:04.040 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:04.037+0000 7f24c9a9f700 1 -- 172.21.15.40:0/2212747029 shutdown_connections 2023-07-14T19:16:04.040 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:04.037+0000 7f24c9a9f700 1 --2- 172.21.15.40:0/2212747029 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f24ac0a1150 0x7f24ac0a3600 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:04.040 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:04.037+0000 7f24c9a9f700 1 --2- 172.21.15.40:0/2212747029 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f24c4100690 0x7f24c419bdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:04.040 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:04.037+0000 7f24c9a9f700 1 --2- 172.21.15.40:0/2212747029 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f24c40feb10 0x7f24c4194840 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:04.040 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:04.037+0000 7f24c9a9f700 1 --2- 172.21.15.40:0/2212747029 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f24c40ffd00 0x7f24c4194d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:04.040 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:04.037+0000 7f24c9a9f700 1 -- 172.21.15.40:0/2212747029 >> 172.21.15.40:0/2212747029 conn(0x7f24c40fa130 msgr2=0x7f24c40fc3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:16:04.040 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:04.037+0000 7f24c9a9f700 1 -- 172.21.15.40:0/2212747029 shutdown_connections 2023-07-14T19:16:04.041 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:04.037+0000 7f24c9a9f700 1 -- 172.21.15.40:0/2212747029 wait complete. 2023-07-14T19:16:04.051 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs sv_0 2023-07-14T19:16:04.180 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:04.178+0000 7f3edba3c700 1 -- 172.21.15.40:0/435578990 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3ed4102fb0 msgr2=0x7f3ed4105390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:16:04.181 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:04.178+0000 7f3edba3c700 1 --2- 172.21.15.40:0/435578990 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3ed4102fb0 0x7f3ed4105390 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f3ec4009990 tx=0x7f3ec4009ca0 comp rx=0 tx=0).stop 2023-07-14T19:16:04.181 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:04.179+0000 7f3edba3c700 1 -- 172.21.15.40:0/435578990 shutdown_connections 2023-07-14T19:16:04.181 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:04.179+0000 7f3edba3c700 1 --2- 172.21.15.40:0/435578990 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3ed41058d0 0x7f3ed4107d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:04.181 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:04.179+0000 7f3edba3c700 1 --2- 172.21.15.40:0/435578990 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3ed4100690 0x7f3ed4102a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:04.181 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:04.179+0000 7f3edba3c700 1 --2- 172.21.15.40:0/435578990 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3ed4102fb0 0x7f3ed4105390 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:04.182 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:04.179+0000 7f3edba3c700 1 -- 172.21.15.40:0/435578990 >> 172.21.15.40:0/435578990 conn(0x7f3ed40fa130 msgr2=0x7f3ed40fc560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:16:04.182 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:04.179+0000 7f3edba3c700 1 -- 172.21.15.40:0/435578990 shutdown_connections 2023-07-14T19:16:04.182 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:04.179+0000 7f3edba3c700 1 -- 172.21.15.40:0/435578990 wait complete. 2023-07-14T19:16:04.183 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:04.179+0000 7f3edba3c700 1 Processor -- start 2023-07-14T19:16:04.184 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:04.179+0000 7f3edba3c700 1 -- start start 2023-07-14T19:16:04.184 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:04.180+0000 7f3edba3c700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3ed4100690 0x7f3ed41958f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:16:04.184 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:04.180+0000 7f3edba3c700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3ed4102fb0 0x7f3ed4195e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:16:04.184 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:04.180+0000 7f3edba3c700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3ed41058d0 0x7f3ed419cea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:16:04.184 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:04.180+0000 7f3edba3c700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ed419d470 con 0x7f3ed41058d0 2023-07-14T19:16:04.184 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:04.180+0000 7f3edba3c700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ed419d5b0 con 0x7f3ed4102fb0 2023-07-14T19:16:04.184 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:04.180+0000 7f3edba3c700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ed419d740 con 0x7f3ed4100690 2023-07-14T19:16:04.185 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:04.180+0000 7f3ed9fd9700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3ed41058d0 0x7f3ed419cea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:16:04.185 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:04.180+0000 7f3ed9fd9700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3ed41058d0 0x7f3ed419cea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:32974/0 (socket says 172.21.15.40:32974) 2023-07-14T19:16:04.185 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:04.180+0000 7f3ed9fd9700 1 -- 172.21.15.40:0/1174040528 learned_addr learned my addr 172.21.15.40:0/1174040528 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-07-14T19:16:04.185 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:04.180+0000 7f3ed8fd7700 1 --2- 172.21.15.40:0/1174040528 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3ed4102fb0 0x7f3ed4195e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:16:04.185 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:04.180+0000 7f3ed97d8700 1 --2- 172.21.15.40:0/1174040528 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3ed4100690 0x7f3ed41958f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:16:04.185 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:04.180+0000 7f3ed9fd9700 1 -- 172.21.15.40:0/1174040528 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3ed4100690 msgr2=0x7f3ed41958f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:16:04.185 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:04.180+0000 7f3ed9fd9700 1 --2- 172.21.15.40:0/1174040528 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3ed4100690 0x7f3ed41958f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:04.186 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:04.180+0000 7f3ed9fd9700 1 -- 172.21.15.40:0/1174040528 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3ed4102fb0 msgr2=0x7f3ed4195e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:16:04.186 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:04.180+0000 7f3ed9fd9700 1 --2- 172.21.15.40:0/1174040528 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3ed4102fb0 0x7f3ed4195e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:04.186 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:04.180+0000 7f3ed9fd9700 1 -- 172.21.15.40:0/1174040528 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ec40095f0 con 0x7f3ed41058d0 2023-07-14T19:16:04.186 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:04.180+0000 7f3ed9fd9700 1 --2- 172.21.15.40:0/1174040528 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3ed41058d0 0x7f3ed419cea0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f3ecc007ee0 tx=0x7f3ecc00d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:16:04.186 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:04.180+0000 7f3eca7fc700 1 -- 172.21.15.40:0/1174040528 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ecc00dd20 con 0x7f3ed41058d0 2023-07-14T19:16:04.186 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:04.180+0000 7f3eca7fc700 1 -- 172.21.15.40:0/1174040528 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3ecc00f040 con 0x7f3ed41058d0 2023-07-14T19:16:04.186 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:04.180+0000 7f3eca7fc700 1 -- 172.21.15.40:0/1174040528 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ecc0048a0 con 0x7f3ed41058d0 2023-07-14T19:16:04.187 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:04.180+0000 7f3edba3c700 1 -- 172.21.15.40:0/1174040528 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3ed419da20 con 0x7f3ed41058d0 2023-07-14T19:16:04.187 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:04.181+0000 7f3edba3c700 1 -- 172.21.15.40:0/1174040528 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3ed419dee0 con 0x7f3ed41058d0 2023-07-14T19:16:04.187 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:04.181+0000 7f3edba3c700 1 -- 172.21.15.40:0/1174040528 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3ed40fbca0 con 0x7f3ed41058d0 2023-07-14T19:16:04.187 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:04.183+0000 7f3eca7fc700 1 -- 172.21.15.40:0/1174040528 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f3ecc003a00 con 0x7f3ed41058d0 2023-07-14T19:16:04.188 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:04.183+0000 7f3eca7fc700 1 --2- 172.21.15.40:0/1174040528 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f3ec00a1150 0x7f3ec00a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:16:04.188 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:04.183+0000 7f3ed97d8700 1 --2- 172.21.15.40:0/1174040528 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f3ec00a1150 0x7f3ec00a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:16:04.188 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:04.183+0000 7f3eca7fc700 1 -- 172.21.15.40:0/1174040528 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f3ecc0cc860 con 0x7f3ed41058d0 2023-07-14T19:16:04.188 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:04.184+0000 7f3ed97d8700 1 --2- 172.21.15.40:0/1174040528 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f3ec00a1150 0x7f3ec00a3600 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f3ed000ac50 tx=0x7f3ed000a380 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:16:04.188 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:04.186+0000 7f3eca7fc700 1 -- 172.21.15.40:0/1174040528 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f3ecc096100 con 0x7f3ed41058d0 2023-07-14T19:16:04.242 INFO:journalctl@ceph.mds.b.smithi130.stdout:Jul 14 19:16:03 smithi130 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-mds-b[122843]: 2023-07-14T19:16:03.854+0000 7fc0c3b89700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-07-14T19:16:04.242 INFO:journalctl@ceph.mds.b.smithi130.stdout:Jul 14 19:16:03 smithi130 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-mds-b[122843]: 2023-07-14T19:16:03.854+0000 7fc0c3b89700 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2023-07-14T19:16:04.243 INFO:journalctl@ceph.mds.b.smithi130.stdout:Jul 14 19:16:03 smithi130 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-mds-b[122843]: 2023-07-14T19:16:03.854+0000 7fc0c3b89700 -1 mds.pinger is_rank_lagging: rank=2 was never sent ping request. 2023-07-14T19:16:04.424 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:04.423+0000 7f3edba3c700 1 -- 172.21.15.40:0/1174040528 --> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "sv_0", "target": ["mon-mgr", ""]}) v1 -- 0x7f3ed4060a80 con 0x7f3ec00a1150 2023-07-14T19:16:04.429 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:04.427+0000 7f3eca7fc700 1 -- 172.21.15.40:0/1174040528 <== mgr.14152 v2:172.21.15.40:6800/1693027118 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+60 (secure 0 0 0) 0x7f3ed4060a80 con 0x7f3ec00a1150 2023-07-14T19:16:04.429 INFO:teuthology.orchestra.run.smithi040.stdout:/volumes/_nogroup/sv_0/9139460b-fb4b-4bbb-9a32-6b77d015f87b 2023-07-14T19:16:04.431 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:04.430+0000 7f3edba3c700 1 -- 172.21.15.40:0/1174040528 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f3ec00a1150 msgr2=0x7f3ec00a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:16:04.432 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:04.430+0000 7f3edba3c700 1 --2- 172.21.15.40:0/1174040528 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f3ec00a1150 0x7f3ec00a3600 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f3ed000ac50 tx=0x7f3ed000a380 comp rx=0 tx=0).stop 2023-07-14T19:16:04.432 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:04.430+0000 7f3edba3c700 1 -- 172.21.15.40:0/1174040528 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3ed41058d0 msgr2=0x7f3ed419cea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:16:04.432 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:04.430+0000 7f3edba3c700 1 --2- 172.21.15.40:0/1174040528 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3ed41058d0 0x7f3ed419cea0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f3ecc007ee0 tx=0x7f3ecc00d3b0 comp rx=0 tx=0).stop 2023-07-14T19:16:04.432 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:04.430+0000 7f3edba3c700 1 -- 172.21.15.40:0/1174040528 shutdown_connections 2023-07-14T19:16:04.432 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:04.430+0000 7f3edba3c700 1 --2- 172.21.15.40:0/1174040528 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f3ec00a1150 0x7f3ec00a3600 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:04.432 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:04.430+0000 7f3edba3c700 1 --2- 172.21.15.40:0/1174040528 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3ed4100690 0x7f3ed41958f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:04.432 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:04.430+0000 7f3edba3c700 1 --2- 172.21.15.40:0/1174040528 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3ed41058d0 0x7f3ed419cea0 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:04.433 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:04.430+0000 7f3edba3c700 1 --2- 172.21.15.40:0/1174040528 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3ed4102fb0 0x7f3ed4195e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:04.433 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:04.430+0000 7f3edba3c700 1 -- 172.21.15.40:0/1174040528 >> 172.21.15.40:0/1174040528 conn(0x7f3ed40fa130 msgr2=0x7f3ed41012f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:16:04.433 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:04.430+0000 7f3edba3c700 1 -- 172.21.15.40:0/1174040528 shutdown_connections 2023-07-14T19:16:04.433 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:04.430+0000 7f3edba3c700 1 -- 172.21.15.40:0/1174040528 wait complete. 2023-07-14T19:16:04.443 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Jul 14 19:16:04 smithi040 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-mgr-x[96762]: 2023-07-14T19:16:04.005+0000 7f2f93b57700 -1 client.0 error registering admin socket command: (17) File exists 2023-07-14T19:16:04.443 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Jul 14 19:16:04 smithi040 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-mgr-x[96762]: 2023-07-14T19:16:04.005+0000 7f2f93b57700 -1 client.0 error registering admin socket command: (17) File exists 2023-07-14T19:16:04.443 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Jul 14 19:16:04 smithi040 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-mgr-x[96762]: 2023-07-14T19:16:04.005+0000 7f2f93b57700 -1 client.0 error registering admin socket command: (17) File exists 2023-07-14T19:16:04.443 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Jul 14 19:16:04 smithi040 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-mgr-x[96762]: 2023-07-14T19:16:04.005+0000 7f2f93b57700 -1 client.0 error registering admin socket command: (17) File exists 2023-07-14T19:16:04.443 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Jul 14 19:16:04 smithi040 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-mgr-x[96762]: 2023-07-14T19:16:04.005+0000 7f2f93b57700 -1 client.0 error registering admin socket command: (17) File exists 2023-07-14T19:16:04.444 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume create cephfs sv_1 '' 2023-07-14T19:16:04.576 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:04.572+0000 7f7e07138700 1 -- 172.21.15.40:0/836528271 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7e00100e00 msgr2=0x7f7e00101250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:16:04.576 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:04.572+0000 7f7e07138700 1 --2- 172.21.15.40:0/836528271 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7e00100e00 0x7f7e00101250 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f7dec009990 tx=0x7f7dec009ca0 comp rx=0 tx=0).stop 2023-07-14T19:16:04.576 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:04.572+0000 7f7e07138700 1 -- 172.21.15.40:0/836528271 shutdown_connections 2023-07-14T19:16:04.576 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:04.572+0000 7f7e07138700 1 --2- 172.21.15.40:0/836528271 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7e00101790 0x7f7e00108050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:04.576 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:04.572+0000 7f7e07138700 1 --2- 172.21.15.40:0/836528271 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7e000ffc10 0x7f7e00100020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:04.577 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:04.572+0000 7f7e07138700 1 --2- 172.21.15.40:0/836528271 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7e00100e00 0x7f7e00101250 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:04.577 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:04.572+0000 7f7e07138700 1 -- 172.21.15.40:0/836528271 >> 172.21.15.40:0/836528271 conn(0x7f7e000fb250 msgr2=0x7f7e000fd660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:16:04.577 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:04.573+0000 7f7e07138700 1 -- 172.21.15.40:0/836528271 shutdown_connections 2023-07-14T19:16:04.577 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:04.573+0000 7f7e07138700 1 -- 172.21.15.40:0/836528271 wait complete. 2023-07-14T19:16:04.577 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:04.573+0000 7f7e07138700 1 Processor -- start 2023-07-14T19:16:04.577 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:04.573+0000 7f7e07138700 1 -- start start 2023-07-14T19:16:04.577 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:04.573+0000 7f7e07138700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7e000ffc10 0x7f7e00195890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:16:04.578 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:04.573+0000 7f7e07138700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7e00100e00 0x7f7e00195dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:16:04.578 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:04.573+0000 7f7e07138700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7e00101790 0x7f7e0019ce40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:16:04.578 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:04.573+0000 7f7e07138700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e0019d410 con 0x7f7e000ffc10 2023-07-14T19:16:04.578 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:04.573+0000 7f7e07138700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e0019d550 con 0x7f7e00101790 2023-07-14T19:16:04.578 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:04.573+0000 7f7e07138700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e0019d6e0 con 0x7f7e00100e00 2023-07-14T19:16:04.578 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:04.573+0000 7f7e04ed4700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7e000ffc10 0x7f7e00195890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:16:04.578 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:04.573+0000 7f7e04ed4700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7e000ffc10 0x7f7e00195890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:32984/0 (socket says 172.21.15.40:32984) 2023-07-14T19:16:04.579 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:04.573+0000 7f7e04ed4700 1 -- 172.21.15.40:0/2530070426 learned_addr learned my addr 172.21.15.40:0/2530070426 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-07-14T19:16:04.579 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:04.574+0000 7f7e056d5700 1 --2- 172.21.15.40:0/2530070426 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7e00101790 0x7f7e0019ce40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:16:04.579 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:04.574+0000 7f7e04ed4700 1 -- 172.21.15.40:0/2530070426 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7e00100e00 msgr2=0x7f7e00195dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:16:04.579 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:04.574+0000 7f7e04ed4700 1 --2- 172.21.15.40:0/2530070426 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7e00100e00 0x7f7e00195dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:04.579 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:04.574+0000 7f7e04ed4700 1 -- 172.21.15.40:0/2530070426 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7e00101790 msgr2=0x7f7e0019ce40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:16:04.579 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:04.574+0000 7f7e04ed4700 1 --2- 172.21.15.40:0/2530070426 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7e00101790 0x7f7e0019ce40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:04.579 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:04.574+0000 7f7e04ed4700 1 -- 172.21.15.40:0/2530070426 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7dec0095f0 con 0x7f7e000ffc10 2023-07-14T19:16:04.579 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:04.574+0000 7f7e056d5700 1 --2- 172.21.15.40:0/2530070426 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7e00101790 0x7f7e0019ce40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T19:16:04.580 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:04.574+0000 7f7e04ed4700 1 --2- 172.21.15.40:0/2530070426 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7e000ffc10 0x7f7e00195890 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f7df400c990 tx=0x7f7df400cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:16:04.580 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:04.574+0000 7f7dfdffb700 1 -- 172.21.15.40:0/2530070426 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7df4007b00 con 0x7f7e000ffc10 2023-07-14T19:16:04.580 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:04.574+0000 7f7e07138700 1 -- 172.21.15.40:0/2530070426 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e0019d9c0 con 0x7f7e000ffc10 2023-07-14T19:16:04.580 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:04.574+0000 7f7dfdffb700 1 -- 172.21.15.40:0/2530070426 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f7df4007c60 con 0x7f7e000ffc10 2023-07-14T19:16:04.580 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:04.574+0000 7f7dfdffb700 1 -- 172.21.15.40:0/2530070426 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7df4018660 con 0x7f7e000ffc10 2023-07-14T19:16:04.580 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:04.574+0000 7f7e07138700 1 -- 172.21.15.40:0/2530070426 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e0019ded0 con 0x7f7e000ffc10 2023-07-14T19:16:04.581 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:04.575+0000 7f7e07138700 1 -- 172.21.15.40:0/2530070426 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7e00106b40 con 0x7f7e000ffc10 2023-07-14T19:16:04.581 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:04.576+0000 7f7dfdffb700 1 -- 172.21.15.40:0/2530070426 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f7df400f450 con 0x7f7e000ffc10 2023-07-14T19:16:04.581 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:04.577+0000 7f7dfdffb700 1 --2- 172.21.15.40:0/2530070426 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f7df00a1150 0x7f7df00a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:16:04.582 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:04.577+0000 7f7dfdffb700 1 -- 172.21.15.40:0/2530070426 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f7df40cc7e0 con 0x7f7e000ffc10 2023-07-14T19:16:04.582 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:04.577+0000 7f7dfffff700 1 --2- 172.21.15.40:0/2530070426 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f7df00a1150 0x7f7df00a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:16:04.582 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:04.577+0000 7f7dfffff700 1 --2- 172.21.15.40:0/2530070426 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f7df00a1150 0x7f7df00a3600 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f7dec00b5c0 tx=0x7f7dec005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:16:04.583 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:04.581+0000 7f7dfdffb700 1 -- 172.21.15.40:0/2530070426 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f7df4096080 con 0x7f7e000ffc10 2023-07-14T19:16:04.721 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:16:04 smithi040 ceph-mon[96518]: pgmap v257: 97 pgs: 97 active+clean; 579 KiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s wr, 9 op/s 2023-07-14T19:16:04.721 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:16:04 smithi040 ceph-mon[96518]: mds.? [v2:172.21.15.130:6836/2604019369,v1:172.21.15.130:6837/2604019369] up:active 2023-07-14T19:16:04.721 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:16:04 smithi040 ceph-mon[96518]: from='client.? 172.21.15.40:0/1485056403' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2023-07-14T19:16:04.722 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:16:04 smithi040 ceph-mon[96518]: daemon mds.c assigned to filesystem cephfs as rank 3 (now has 4 ranks) 2023-07-14T19:16:04.722 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:16:04 smithi040 ceph-mon[96518]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 9 up:standby 2023-07-14T19:16:04.722 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:16:04 smithi040 ceph-mon[96518]: fsmap cephfs:4 {0=b=up:active,1=i=up:active,2=h=up:active,3=c=up:creating} 8 up:standby 2023-07-14T19:16:04.722 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:16:04 smithi040 ceph-mon[96518]: daemon mds.c is now active in filesystem cephfs as rank 3 2023-07-14T19:16:04.826 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:04.824+0000 7f7e07138700 1 -- 172.21.15.40:0/2530070426 --> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "", "target": ["mon-mgr", ""]}) v1 -- 0x7f7e00060a80 con 0x7f7df00a1150 2023-07-14T19:16:04.849 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:04.847+0000 7f7dfdffb700 1 -- 172.21.15.40:0/2530070426 <== mgr.14152 v2:172.21.15.40:6800/1693027118 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f7e00060a80 con 0x7f7df00a1150 2023-07-14T19:16:04.851 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:04.849+0000 7f7e07138700 1 -- 172.21.15.40:0/2530070426 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f7df00a1150 msgr2=0x7f7df00a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:16:04.851 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:04.849+0000 7f7e07138700 1 --2- 172.21.15.40:0/2530070426 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f7df00a1150 0x7f7df00a3600 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f7dec00b5c0 tx=0x7f7dec005fb0 comp rx=0 tx=0).stop 2023-07-14T19:16:04.852 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:04.849+0000 7f7e07138700 1 -- 172.21.15.40:0/2530070426 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7e000ffc10 msgr2=0x7f7e00195890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:16:04.852 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:04.849+0000 7f7e07138700 1 --2- 172.21.15.40:0/2530070426 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7e000ffc10 0x7f7e00195890 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f7df400c990 tx=0x7f7df400cd50 comp rx=0 tx=0).stop 2023-07-14T19:16:04.852 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:04.850+0000 7f7e07138700 1 -- 172.21.15.40:0/2530070426 shutdown_connections 2023-07-14T19:16:04.852 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:04.850+0000 7f7e07138700 1 --2- 172.21.15.40:0/2530070426 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f7df00a1150 0x7f7df00a3600 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:04.852 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:04.850+0000 7f7e07138700 1 --2- 172.21.15.40:0/2530070426 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7e00101790 0x7f7e0019ce40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:04.852 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:04.850+0000 7f7e07138700 1 --2- 172.21.15.40:0/2530070426 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7e000ffc10 0x7f7e00195890 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:04.853 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:04.850+0000 7f7e07138700 1 --2- 172.21.15.40:0/2530070426 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7e00100e00 0x7f7e00195dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:04.853 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:04.850+0000 7f7e07138700 1 -- 172.21.15.40:0/2530070426 >> 172.21.15.40:0/2530070426 conn(0x7f7e000fb250 msgr2=0x7f7e000fd470 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:16:04.853 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:04.850+0000 7f7e07138700 1 -- 172.21.15.40:0/2530070426 shutdown_connections 2023-07-14T19:16:04.853 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:04.850+0000 7f7e07138700 1 -- 172.21.15.40:0/2530070426 wait complete. 2023-07-14T19:16:04.863 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs sv_1 2023-07-14T19:16:04.972 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:16:04 smithi191 ceph-mon[98311]: pgmap v257: 97 pgs: 97 active+clean; 579 KiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s wr, 9 op/s 2023-07-14T19:16:04.972 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:16:04 smithi191 ceph-mon[98311]: mds.? [v2:172.21.15.130:6836/2604019369,v1:172.21.15.130:6837/2604019369] up:active 2023-07-14T19:16:04.972 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:16:04 smithi191 ceph-mon[98311]: from='client.? 172.21.15.40:0/1485056403' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2023-07-14T19:16:04.972 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:16:04 smithi191 ceph-mon[98311]: daemon mds.c assigned to filesystem cephfs as rank 3 (now has 4 ranks) 2023-07-14T19:16:04.973 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:16:04 smithi191 ceph-mon[98311]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 9 up:standby 2023-07-14T19:16:04.973 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:16:04 smithi191 ceph-mon[98311]: fsmap cephfs:4 {0=b=up:active,1=i=up:active,2=h=up:active,3=c=up:creating} 8 up:standby 2023-07-14T19:16:04.973 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:16:04 smithi191 ceph-mon[98311]: daemon mds.c is now active in filesystem cephfs as rank 3 2023-07-14T19:16:04.992 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:16:04 smithi130 ceph-mon[97832]: pgmap v257: 97 pgs: 97 active+clean; 579 KiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s wr, 9 op/s 2023-07-14T19:16:04.992 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:16:04 smithi130 ceph-mon[97832]: mds.? [v2:172.21.15.130:6836/2604019369,v1:172.21.15.130:6837/2604019369] up:active 2023-07-14T19:16:04.992 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:16:04 smithi130 ceph-mon[97832]: from='client.? 172.21.15.40:0/1485056403' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2023-07-14T19:16:04.993 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:16:04 smithi130 ceph-mon[97832]: daemon mds.c assigned to filesystem cephfs as rank 3 (now has 4 ranks) 2023-07-14T19:16:04.993 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:16:04 smithi130 ceph-mon[97832]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 9 up:standby 2023-07-14T19:16:04.993 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:16:04 smithi130 ceph-mon[97832]: fsmap cephfs:4 {0=b=up:active,1=i=up:active,2=h=up:active,3=c=up:creating} 8 up:standby 2023-07-14T19:16:04.993 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:16:04 smithi130 ceph-mon[97832]: daemon mds.c is now active in filesystem cephfs as rank 3 2023-07-14T19:16:04.995 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:04.990+0000 7f5d73465700 1 -- 172.21.15.40:0/2292388158 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5d6c0ffd00 msgr2=0x7f5d6c100150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:16:04.995 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:04.990+0000 7f5d73465700 1 --2- 172.21.15.40:0/2292388158 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5d6c0ffd00 0x7f5d6c100150 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f5d5c009990 tx=0x7f5d5c009ca0 comp rx=0 tx=0).stop 2023-07-14T19:16:04.995 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:04.990+0000 7f5d73465700 1 -- 172.21.15.40:0/2292388158 shutdown_connections 2023-07-14T19:16:04.995 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:04.990+0000 7f5d73465700 1 --2- 172.21.15.40:0/2292388158 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5d6c0feb10 0x7f5d6c0fef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:04.996 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:04.990+0000 7f5d73465700 1 --2- 172.21.15.40:0/2292388158 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5d6c100690 0x7f5d6c106f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:04.996 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:04.990+0000 7f5d73465700 1 --2- 172.21.15.40:0/2292388158 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5d6c0ffd00 0x7f5d6c100150 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:04.996 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:04.990+0000 7f5d73465700 1 -- 172.21.15.40:0/2292388158 >> 172.21.15.40:0/2292388158 conn(0x7f5d6c0fa130 msgr2=0x7f5d6c0fc560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:16:04.996 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:04.991+0000 7f5d73465700 1 -- 172.21.15.40:0/2292388158 shutdown_connections 2023-07-14T19:16:04.996 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:04.991+0000 7f5d73465700 1 -- 172.21.15.40:0/2292388158 wait complete. 2023-07-14T19:16:04.996 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:04.991+0000 7f5d73465700 1 Processor -- start 2023-07-14T19:16:04.996 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:04.991+0000 7f5d73465700 1 -- start start 2023-07-14T19:16:04.997 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:04.991+0000 7f5d73465700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5d6c0feb10 0x7f5d6c194890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:16:04.997 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:04.991+0000 7f5d73465700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5d6c0ffd00 0x7f5d6c194dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:16:04.997 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:04.991+0000 7f5d73465700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5d6c100690 0x7f5d6c19be40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:16:04.997 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:04.991+0000 7f5d73465700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d6c19c410 con 0x7f5d6c100690 2023-07-14T19:16:04.997 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:04.991+0000 7f5d73465700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d6c19c550 con 0x7f5d6c0ffd00 2023-07-14T19:16:04.997 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:04.991+0000 7f5d73465700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d6c19c6e0 con 0x7f5d6c0feb10 2023-07-14T19:16:04.997 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:04.991+0000 7f5d71a02700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5d6c100690 0x7f5d6c19be40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:16:04.997 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:04.992+0000 7f5d71a02700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5d6c100690 0x7f5d6c19be40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:33004/0 (socket says 172.21.15.40:33004) 2023-07-14T19:16:04.998 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:04.992+0000 7f5d71a02700 1 -- 172.21.15.40:0/2979824486 learned_addr learned my addr 172.21.15.40:0/2979824486 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-07-14T19:16:04.998 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:04.992+0000 7f5d70a00700 1 --2- 172.21.15.40:0/2979824486 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5d6c0ffd00 0x7f5d6c194dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:16:04.998 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:04.992+0000 7f5d71201700 1 --2- 172.21.15.40:0/2979824486 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5d6c0feb10 0x7f5d6c194890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:16:04.998 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:04.992+0000 7f5d71a02700 1 -- 172.21.15.40:0/2979824486 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5d6c0feb10 msgr2=0x7f5d6c194890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:16:04.998 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:04.992+0000 7f5d71a02700 1 --2- 172.21.15.40:0/2979824486 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5d6c0feb10 0x7f5d6c194890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:04.998 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:04.992+0000 7f5d71a02700 1 -- 172.21.15.40:0/2979824486 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5d6c0ffd00 msgr2=0x7f5d6c194dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:16:04.998 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:04.992+0000 7f5d71a02700 1 --2- 172.21.15.40:0/2979824486 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5d6c0ffd00 0x7f5d6c194dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:04.999 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:04.992+0000 7f5d71a02700 1 -- 172.21.15.40:0/2979824486 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d5c0095f0 con 0x7f5d6c100690 2023-07-14T19:16:04.999 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:04.992+0000 7f5d70a00700 1 --2- 172.21.15.40:0/2979824486 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5d6c0ffd00 0x7f5d6c194dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T19:16:04.999 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:04.992+0000 7f5d71201700 1 --2- 172.21.15.40:0/2979824486 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5d6c0feb10 0x7f5d6c194890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T19:16:04.999 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:04.992+0000 7f5d71a02700 1 --2- 172.21.15.40:0/2979824486 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5d6c100690 0x7f5d6c19be40 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f5d64007ee0 tx=0x7f5d6400d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:16:04.999 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:04.992+0000 7f5d627fc700 1 -- 172.21.15.40:0/2979824486 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d64017070 con 0x7f5d6c100690 2023-07-14T19:16:04.999 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:04.992+0000 7f5d627fc700 1 -- 172.21.15.40:0/2979824486 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f5d6400f040 con 0x7f5d6c100690 2023-07-14T19:16:04.999 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:04.992+0000 7f5d627fc700 1 -- 172.21.15.40:0/2979824486 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d64005740 con 0x7f5d6c100690 2023-07-14T19:16:04.999 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:04.992+0000 7f5d73465700 1 -- 172.21.15.40:0/2979824486 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5d6c19c9c0 con 0x7f5d6c100690 2023-07-14T19:16:05.000 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:04.992+0000 7f5d73465700 1 -- 172.21.15.40:0/2979824486 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5d6c19ce80 con 0x7f5d6c100690 2023-07-14T19:16:05.000 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:04.993+0000 7f5d73465700 1 -- 172.21.15.40:0/2979824486 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5d6c1060f0 con 0x7f5d6c100690 2023-07-14T19:16:05.000 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:04.994+0000 7f5d627fc700 1 -- 172.21.15.40:0/2979824486 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f5d640058a0 con 0x7f5d6c100690 2023-07-14T19:16:05.001 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:04.995+0000 7f5d627fc700 1 --2- 172.21.15.40:0/2979824486 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f5d580a11d0 0x7f5d580a3680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:16:05.001 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:04.995+0000 7f5d71201700 1 --2- 172.21.15.40:0/2979824486 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f5d580a11d0 0x7f5d580a3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:16:05.001 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:04.995+0000 7f5d627fc700 1 -- 172.21.15.40:0/2979824486 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f5d640cc0a0 con 0x7f5d6c100690 2023-07-14T19:16:05.001 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:04.995+0000 7f5d71201700 1 --2- 172.21.15.40:0/2979824486 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f5d580a11d0 0x7f5d580a3680 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f5d68005950 tx=0x7f5d680058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:16:05.001 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:04.999+0000 7f5d627fc700 1 -- 172.21.15.40:0/2979824486 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f5d640958a0 con 0x7f5d6c100690 2023-07-14T19:16:05.239 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:05.238+0000 7f5d73465700 1 -- 172.21.15.40:0/2979824486 --> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "sv_1", "target": ["mon-mgr", ""]}) v1 -- 0x7f5d6c060a80 con 0x7f5d580a11d0 2023-07-14T19:16:05.243 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:05.242+0000 7f5d627fc700 1 -- 172.21.15.40:0/2979824486 <== mgr.14152 v2:172.21.15.40:6800/1693027118 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+60 (secure 0 0 0) 0x7f5d6c060a80 con 0x7f5d580a11d0 2023-07-14T19:16:05.244 INFO:teuthology.orchestra.run.smithi040.stdout:/volumes/_nogroup/sv_1/0ab40c84-70ae-4034-9a8f-8c6c6627b87a 2023-07-14T19:16:05.246 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:05.244+0000 7f5d73465700 1 -- 172.21.15.40:0/2979824486 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f5d580a11d0 msgr2=0x7f5d580a3680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:16:05.246 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:05.244+0000 7f5d73465700 1 --2- 172.21.15.40:0/2979824486 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f5d580a11d0 0x7f5d580a3680 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f5d68005950 tx=0x7f5d680058e0 comp rx=0 tx=0).stop 2023-07-14T19:16:05.246 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:05.244+0000 7f5d73465700 1 -- 172.21.15.40:0/2979824486 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5d6c100690 msgr2=0x7f5d6c19be40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:16:05.247 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:05.244+0000 7f5d73465700 1 --2- 172.21.15.40:0/2979824486 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5d6c100690 0x7f5d6c19be40 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f5d64007ee0 tx=0x7f5d6400d3b0 comp rx=0 tx=0).stop 2023-07-14T19:16:05.247 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:05.244+0000 7f5d73465700 1 -- 172.21.15.40:0/2979824486 shutdown_connections 2023-07-14T19:16:05.247 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:05.244+0000 7f5d73465700 1 --2- 172.21.15.40:0/2979824486 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f5d580a11d0 0x7f5d580a3680 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:05.247 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:05.244+0000 7f5d73465700 1 --2- 172.21.15.40:0/2979824486 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5d6c0feb10 0x7f5d6c194890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:05.247 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:05.244+0000 7f5d73465700 1 --2- 172.21.15.40:0/2979824486 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5d6c100690 0x7f5d6c19be40 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:05.248 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:05.244+0000 7f5d73465700 1 --2- 172.21.15.40:0/2979824486 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5d6c0ffd00 0x7f5d6c194dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:05.248 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:05.244+0000 7f5d73465700 1 -- 172.21.15.40:0/2979824486 >> 172.21.15.40:0/2979824486 conn(0x7f5d6c0fa130 msgr2=0x7f5d6c103230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:16:05.248 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:05.244+0000 7f5d73465700 1 -- 172.21.15.40:0/2979824486 shutdown_connections 2023-07-14T19:16:05.249 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:05.244+0000 7f5d73465700 1 -- 172.21.15.40:0/2979824486 wait complete. 2023-07-14T19:16:05.258 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-07-14T19:16:05.385 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:05.382+0000 7f7807cdb700 1 -- 172.21.15.40:0/846229265 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f78000ffcf0 msgr2=0x7f7800100140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:16:05.386 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:05.382+0000 7f7807cdb700 1 --2- 172.21.15.40:0/846229265 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f78000ffcf0 0x7f7800100140 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f77f0009990 tx=0x7f77f0009ca0 comp rx=0 tx=0).stop 2023-07-14T19:16:05.386 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:05.382+0000 7f7807cdb700 1 -- 172.21.15.40:0/846229265 shutdown_connections 2023-07-14T19:16:05.386 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:05.382+0000 7f7807cdb700 1 --2- 172.21.15.40:0/846229265 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f78000feb00 0x7f78000fef10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:05.386 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:05.382+0000 7f7807cdb700 1 --2- 172.21.15.40:0/846229265 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7800100680 0x7f7800106f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:05.387 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:05.382+0000 7f7807cdb700 1 --2- 172.21.15.40:0/846229265 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f78000ffcf0 0x7f7800100140 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:05.387 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:05.382+0000 7f7807cdb700 1 -- 172.21.15.40:0/846229265 >> 172.21.15.40:0/846229265 conn(0x7f78000fa100 msgr2=0x7f78000fc550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:16:05.387 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:05.382+0000 7f7807cdb700 1 -- 172.21.15.40:0/846229265 shutdown_connections 2023-07-14T19:16:05.387 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:05.382+0000 7f7807cdb700 1 -- 172.21.15.40:0/846229265 wait complete. 2023-07-14T19:16:05.388 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:05.382+0000 7f7807cdb700 1 Processor -- start 2023-07-14T19:16:05.388 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:05.382+0000 7f7807cdb700 1 -- start start 2023-07-14T19:16:05.388 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:05.382+0000 7f7807cdb700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f78000feb00 0x7f7800102a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:16:05.389 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:05.382+0000 7f7807cdb700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f78000ffcf0 0x7f78001010d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:16:05.389 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:05.382+0000 7f7807cdb700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7800100680 0x7f7800101610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:16:05.389 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:05.383+0000 7f7807cdb700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7800101db0 con 0x7f78000feb00 2023-07-14T19:16:05.389 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:05.383+0000 7f7807cdb700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7800101f20 con 0x7f78000ffcf0 2023-07-14T19:16:05.390 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:05.383+0000 7f7807cdb700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f780019f940 con 0x7f7800100680 2023-07-14T19:16:05.390 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:05.383+0000 7f7805a77700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f78000feb00 0x7f7800102a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:16:05.390 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:05.383+0000 7f7805a77700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f78000feb00 0x7f7800102a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:33012/0 (socket says 172.21.15.40:33012) 2023-07-14T19:16:05.390 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:05.383+0000 7f7805a77700 1 -- 172.21.15.40:0/208544988 learned_addr learned my addr 172.21.15.40:0/208544988 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-07-14T19:16:05.397 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:05.383+0000 7f7805276700 1 --2- 172.21.15.40:0/208544988 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f78000ffcf0 0x7f78001010d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:16:05.397 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:05.383+0000 7f7806278700 1 --2- 172.21.15.40:0/208544988 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7800100680 0x7f7800101610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:16:05.397 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:05.383+0000 7f7805a77700 1 -- 172.21.15.40:0/208544988 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7800100680 msgr2=0x7f7800101610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:16:05.397 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:05.383+0000 7f7805a77700 1 --2- 172.21.15.40:0/208544988 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7800100680 0x7f7800101610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:05.398 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:05.383+0000 7f7805a77700 1 -- 172.21.15.40:0/208544988 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f78000ffcf0 msgr2=0x7f78001010d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:16:05.398 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:05.383+0000 7f7805a77700 1 --2- 172.21.15.40:0/208544988 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f78000ffcf0 0x7f78001010d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:05.398 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:05.383+0000 7f7805a77700 1 -- 172.21.15.40:0/208544988 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77f00095f0 con 0x7f78000feb00 2023-07-14T19:16:05.399 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:05.383+0000 7f7805276700 1 --2- 172.21.15.40:0/208544988 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f78000ffcf0 0x7f78001010d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T19:16:05.399 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:05.383+0000 7f7805a77700 1 --2- 172.21.15.40:0/208544988 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f78000feb00 0x7f7800102a90 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f77fc00d980 tx=0x7f77fc00dd40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:16:05.399 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:05.383+0000 7f77f6ffd700 1 -- 172.21.15.40:0/208544988 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77fc00de60 con 0x7f78000feb00 2023-07-14T19:16:05.399 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:05.383+0000 7f77f6ffd700 1 -- 172.21.15.40:0/208544988 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f77fc0044e0 con 0x7f78000feb00 2023-07-14T19:16:05.400 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:05.383+0000 7f7807cdb700 1 -- 172.21.15.40:0/208544988 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f780019fc20 con 0x7f78000feb00 2023-07-14T19:16:05.400 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:05.383+0000 7f77f6ffd700 1 -- 172.21.15.40:0/208544988 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77fc003be0 con 0x7f78000feb00 2023-07-14T19:16:05.400 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:05.383+0000 7f7807cdb700 1 -- 172.21.15.40:0/208544988 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f78001a0130 con 0x7f78000feb00 2023-07-14T19:16:05.400 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:05.384+0000 7f7807cdb700 1 -- 172.21.15.40:0/208544988 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f780010a2d0 con 0x7f78000feb00 2023-07-14T19:16:05.401 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:05.385+0000 7f77f6ffd700 1 -- 172.21.15.40:0/208544988 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f77fc009bb0 con 0x7f78000feb00 2023-07-14T19:16:05.401 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:05.386+0000 7f77f6ffd700 1 --2- 172.21.15.40:0/208544988 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f77ec0a11a0 0x7f77ec0a3650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:16:05.401 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:05.386+0000 7f7805276700 1 --2- 172.21.15.40:0/208544988 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f77ec0a11a0 0x7f77ec0a3650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:16:05.401 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:05.386+0000 7f77f6ffd700 1 -- 172.21.15.40:0/208544988 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f77fc0cc570 con 0x7f78000feb00 2023-07-14T19:16:05.402 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:05.386+0000 7f7805276700 1 --2- 172.21.15.40:0/208544988 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f77ec0a11a0 0x7f77ec0a3650 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f77f0006010 tx=0x7f77f000b540 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:16:05.402 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:05.390+0000 7f77f6ffd700 1 -- 172.21.15.40:0/208544988 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f77fc095e10 con 0x7f78000feb00 2023-07-14T19:16:05.624 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:05.622+0000 7f77f6ffd700 1 -- 172.21.15.40:0/208544988 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7f77fc0961e0 con 0x7f78000feb00 2023-07-14T19:16:05.645 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:05.643+0000 7f7807cdb700 1 -- 172.21.15.40:0/208544988 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f78001a08f0 con 0x7f78000feb00 2023-07-14T19:16:05.645 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:05.644+0000 7f77f6ffd700 1 -- 172.21.15.40:0/208544988 <== mon.0 v2:172.21.15.40:3300/0 8 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) v1 ==== 94+0+10877 (secure 0 0 0) 0x7f77fc0a5a90 con 0x7f78000feb00 2023-07-14T19:16:05.646 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-07-14T19:16:05.646 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14523,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6835/501143718","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6834","nonce":501143718},{"type":"v1","addr":"172.21.15.40:6835","nonce":501143718}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14535,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6837/2634476117","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6836","nonce":2634476117},{"type":"v1","addr":"172.21.15.40:6837","nonce":2634476117}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":26,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2023-07-14T19:15:57.650710+0000","modified":"2023-07-14T19:16:05.624568+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24539,"mds_1":24508,"mds_2":24506,"mds_3":24490,"mds_4":24482},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14586":{"gid":14586,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.40:6839/959937545","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6838","nonce":959937545},{"type":"v1","addr":"172.21.15.40:6839","nonce":959937545}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_14598":{"gid":14598,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.40:6841/1590486077","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6840","nonce":1590486077},{"type":"v1","addr":"172.21.15.40:6841","nonce":1590486077}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24433":{"gid":24433,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.191:6833/31509862","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":31509862},{"type":"v1","addr":"172.21.15.191:6833","nonce":31509862}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24452":{"gid":24452,"name":"k","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.130:6833/3677452688","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6832","nonce":3677452688},{"type":"v1","addr":"172.21.15.130:6833","nonce":3677452688}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24478":{"gid":24478,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.191:6835/3712674634","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":3712674634},{"type":"v1","addr":"172.21.15.191:6835","nonce":3712674634}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24482":{"gid":24482,"name":"e","rank":4,"incarnation":24,"state":"up:active","state_seq":8,"addr":"172.21.15.130:6835/3088490808","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6834","nonce":3088490808},{"type":"v1","addr":"172.21.15.130:6835","nonce":3088490808}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24490":{"gid":24490,"name":"c","rank":3,"incarnation":22,"state":"up:active","state_seq":7,"addr":"172.21.15.191:6837/2406892975","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6836","nonce":2406892975},{"type":"v1","addr":"172.21.15.191:6837","nonce":2406892975}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24506":{"gid":24506,"name":"h","rank":2,"incarnation":20,"state":"up:active","state_seq":6,"addr":"172.21.15.130:6837/2604019369","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6836","nonce":2604019369},{"type":"v1","addr":"172.21.15.130:6837","nonce":2604019369}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24508":{"gid":24508,"name":"i","rank":1,"incarnation":18,"state":"up:active","state_seq":5,"addr":"172.21.15.191:6839/1977507302","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":1977507302},{"type":"v1","addr":"172.21.15.191:6839","nonce":1977507302}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24539":{"gid":24539,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.130:6839/2145479535","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6838","nonce":2145479535},{"type":"v1","addr":"172.21.15.130:6839","nonce":2145479535}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-07-14T19:16:05.647 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:05.646+0000 7f7807cdb700 1 -- 172.21.15.40:0/208544988 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f77ec0a11a0 msgr2=0x7f77ec0a3650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:16:05.648 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:05.646+0000 7f7807cdb700 1 --2- 172.21.15.40:0/208544988 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f77ec0a11a0 0x7f77ec0a3650 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f77f0006010 tx=0x7f77f000b540 comp rx=0 tx=0).stop 2023-07-14T19:16:05.648 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:05.646+0000 7f7807cdb700 1 -- 172.21.15.40:0/208544988 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f78000feb00 msgr2=0x7f7800102a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:16:05.648 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:05.646+0000 7f7807cdb700 1 --2- 172.21.15.40:0/208544988 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f78000feb00 0x7f7800102a90 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f77fc00d980 tx=0x7f77fc00dd40 comp rx=0 tx=0).stop 2023-07-14T19:16:05.648 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:05.646+0000 7f7807cdb700 1 -- 172.21.15.40:0/208544988 shutdown_connections 2023-07-14T19:16:05.648 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:05.646+0000 7f7807cdb700 1 --2- 172.21.15.40:0/208544988 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f77ec0a11a0 0x7f77ec0a3650 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:05.648 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:05.646+0000 7f7807cdb700 1 --2- 172.21.15.40:0/208544988 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7800100680 0x7f7800101610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:05.649 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:05.646+0000 7f7807cdb700 1 --2- 172.21.15.40:0/208544988 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f78000feb00 0x7f7800102a90 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:05.649 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:05.646+0000 7f7807cdb700 1 --2- 172.21.15.40:0/208544988 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f78000ffcf0 0x7f78001010d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:05.649 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:05.646+0000 7f7807cdb700 1 -- 172.21.15.40:0/208544988 >> 172.21.15.40:0/208544988 conn(0x7f78000fa100 msgr2=0x7f7800067820 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:16:05.649 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:05.646+0000 7f7807cdb700 1 -- 172.21.15.40:0/208544988 shutdown_connections 2023-07-14T19:16:05.649 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:05.646+0000 7f7807cdb700 1 -- 172.21.15.40:0/208544988 wait complete. 2023-07-14T19:16:05.649 INFO:teuthology.orchestra.run.smithi040.stderr:dumped fsmap epoch 26 2023-07-14T19:16:05.660 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-07-14T19:16:05.789 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:05.784+0000 7f74f594f700 1 -- 172.21.15.40:0/11684625 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f74f00feb50 msgr2=0x7f74f00fef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:16:05.789 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:05.784+0000 7f74f594f700 1 --2- 172.21.15.40:0/11684625 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f74f00feb50 0x7f74f00fef60 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f74e0009990 tx=0x7f74e0009ca0 comp rx=0 tx=0).stop 2023-07-14T19:16:05.790 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:05.784+0000 7f74f594f700 1 -- 172.21.15.40:0/11684625 shutdown_connections 2023-07-14T19:16:05.790 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:05.784+0000 7f74f594f700 1 --2- 172.21.15.40:0/11684625 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f74f01006d0 0x7f74f0106f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:05.790 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:05.784+0000 7f74f594f700 1 --2- 172.21.15.40:0/11684625 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f74f00feb50 0x7f74f00fef60 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:05.790 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:05.784+0000 7f74f594f700 1 --2- 172.21.15.40:0/11684625 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f74f00ffd40 0x7f74f0100190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:05.790 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:05.784+0000 7f74f594f700 1 -- 172.21.15.40:0/11684625 >> 172.21.15.40:0/11684625 conn(0x7f74f00fa150 msgr2=0x7f74f00fc5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:16:05.791 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:05.784+0000 7f74f594f700 1 -- 172.21.15.40:0/11684625 shutdown_connections 2023-07-14T19:16:05.791 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:05.784+0000 7f74f594f700 1 -- 172.21.15.40:0/11684625 wait complete. 2023-07-14T19:16:05.791 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:05.785+0000 7f74f594f700 1 Processor -- start 2023-07-14T19:16:05.791 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:05.785+0000 7f74f594f700 1 -- start start 2023-07-14T19:16:05.791 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:05.785+0000 7f74f594f700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f74f00feb50 0x7f74f00695f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:16:05.791 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:05.785+0000 7f74f594f700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f74f00ffd40 0x7f74f019a640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:16:05.791 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:05.785+0000 7f74f594f700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f74f01006d0 0x7f74f019cc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:16:05.792 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:05.785+0000 7f74f594f700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74f019d460 con 0x7f74f01006d0 2023-07-14T19:16:05.792 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:05.785+0000 7f74f594f700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74f019d5d0 con 0x7f74f00feb50 2023-07-14T19:16:05.792 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:05.785+0000 7f74f594f700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74f019d790 con 0x7f74f00ffd40 2023-07-14T19:16:05.792 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:05.785+0000 7f74ef7fe700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f74f01006d0 0x7f74f019cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:16:05.792 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:05.785+0000 7f74ef7fe700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f74f01006d0 0x7f74f019cc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:33052/0 (socket says 172.21.15.40:33052) 2023-07-14T19:16:05.792 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:05.785+0000 7f74ef7fe700 1 -- 172.21.15.40:0/128476494 learned_addr learned my addr 172.21.15.40:0/128476494 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-07-14T19:16:05.792 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:05.785+0000 7f74ee7fc700 1 --2- 172.21.15.40:0/128476494 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f74f00ffd40 0x7f74f019a640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:16:05.793 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:05.785+0000 7f74eeffd700 1 --2- 172.21.15.40:0/128476494 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f74f00feb50 0x7f74f00695f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:16:05.793 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:05.785+0000 7f74ef7fe700 1 -- 172.21.15.40:0/128476494 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f74f00ffd40 msgr2=0x7f74f019a640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:16:05.793 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:05.785+0000 7f74ef7fe700 1 --2- 172.21.15.40:0/128476494 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f74f00ffd40 0x7f74f019a640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:05.793 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:05.785+0000 7f74ef7fe700 1 -- 172.21.15.40:0/128476494 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f74f00feb50 msgr2=0x7f74f00695f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:16:05.793 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:05.785+0000 7f74ef7fe700 1 --2- 172.21.15.40:0/128476494 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f74f00feb50 0x7f74f00695f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:05.793 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:05.785+0000 7f74ef7fe700 1 -- 172.21.15.40:0/128476494 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74e00095f0 con 0x7f74f01006d0 2023-07-14T19:16:05.793 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:05.786+0000 7f74ef7fe700 1 --2- 172.21.15.40:0/128476494 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f74f01006d0 0x7f74f019cc90 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f74e8007970 tx=0x7f74e8007c80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:16:05.794 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:05.786+0000 7f74f494d700 1 -- 172.21.15.40:0/128476494 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74e8017070 con 0x7f74f01006d0 2023-07-14T19:16:05.794 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:05.786+0000 7f74f594f700 1 -- 172.21.15.40:0/128476494 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f74f019da70 con 0x7f74f01006d0 2023-07-14T19:16:05.794 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:05.786+0000 7f74f594f700 1 -- 172.21.15.40:0/128476494 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f74f019dfd0 con 0x7f74f01006d0 2023-07-14T19:16:05.794 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:05.786+0000 7f74f494d700 1 -- 172.21.15.40:0/128476494 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f74e800f040 con 0x7f74f01006d0 2023-07-14T19:16:05.794 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:05.786+0000 7f74f494d700 1 -- 172.21.15.40:0/128476494 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74e801cb50 con 0x7f74f01006d0 2023-07-14T19:16:05.794 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:05.787+0000 7f74f594f700 1 -- 172.21.15.40:0/128476494 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f74dc005320 con 0x7f74f01006d0 2023-07-14T19:16:05.795 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:05.788+0000 7f74f494d700 1 -- 172.21.15.40:0/128476494 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7f74e800d7d0 con 0x7f74f01006d0 2023-07-14T19:16:05.795 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:05.788+0000 7f74f494d700 1 --2- 172.21.15.40:0/128476494 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f74d80a1220 0x7f74d80a36d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:16:05.796 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:05.788+0000 7f74eeffd700 1 --2- 172.21.15.40:0/128476494 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f74d80a1220 0x7f74d80a36d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:16:05.796 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:05.789+0000 7f74f494d700 1 -- 172.21.15.40:0/128476494 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f74e80cc2d0 con 0x7f74f01006d0 2023-07-14T19:16:05.796 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:05.789+0000 7f74eeffd700 1 --2- 172.21.15.40:0/128476494 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f74d80a1220 0x7f74d80a36d0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f74e0006010 tx=0x7f74e000b540 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:16:05.796 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:05.792+0000 7f74f494d700 1 -- 172.21.15.40:0/128476494 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f74e8095b00 con 0x7f74f01006d0 2023-07-14T19:16:05.971 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:16:05 smithi040 ceph-mon[96518]: from='client.14688 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T19:16:05.971 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:16:05 smithi040 ceph-mon[96518]: from='client.14700 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T19:16:05.971 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:16:05 smithi040 ceph-mon[96518]: mds.? [v2:172.21.15.191:6836/2406892975,v1:172.21.15.191:6837/2406892975] up:active 2023-07-14T19:16:05.971 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:16:05 smithi040 ceph-mon[96518]: daemon mds.e assigned to filesystem cephfs as rank 4 (now has 5 ranks) 2023-07-14T19:16:05.971 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:16:05 smithi040 ceph-mon[96518]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2023-07-14T19:16:05.972 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:16:05 smithi040 ceph-mon[96518]: Cluster is now healthy 2023-07-14T19:16:05.972 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:16:05 smithi040 ceph-mon[96518]: fsmap cephfs:4 {0=b=up:active,1=i=up:active,2=h=up:active,3=c=up:active} 8 up:standby 2023-07-14T19:16:05.972 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:16:05 smithi040 ceph-mon[96518]: fsmap cephfs:5 1 up:creating 7 up:standby 4 up:active 2023-07-14T19:16:05.972 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:16:05 smithi040 ceph-mon[96518]: daemon mds.e is now active in filesystem cephfs as rank 4 2023-07-14T19:16:05.973 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:16:05 smithi191 ceph-mon[98311]: from='client.14688 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T19:16:05.973 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:16:05 smithi191 ceph-mon[98311]: from='client.14700 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T19:16:05.973 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:16:05 smithi191 ceph-mon[98311]: mds.? [v2:172.21.15.191:6836/2406892975,v1:172.21.15.191:6837/2406892975] up:active 2023-07-14T19:16:05.973 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:16:05 smithi191 ceph-mon[98311]: daemon mds.e assigned to filesystem cephfs as rank 4 (now has 5 ranks) 2023-07-14T19:16:05.973 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:16:05 smithi191 ceph-mon[98311]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2023-07-14T19:16:05.973 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:16:05 smithi191 ceph-mon[98311]: Cluster is now healthy 2023-07-14T19:16:05.973 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:16:05 smithi191 ceph-mon[98311]: fsmap cephfs:4 {0=b=up:active,1=i=up:active,2=h=up:active,3=c=up:active} 8 up:standby 2023-07-14T19:16:05.974 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:16:05 smithi191 ceph-mon[98311]: fsmap cephfs:5 1 up:creating 7 up:standby 4 up:active 2023-07-14T19:16:05.974 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:16:05 smithi191 ceph-mon[98311]: daemon mds.e is now active in filesystem cephfs as rank 4 2023-07-14T19:16:05.992 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:16:05 smithi130 ceph-mon[97832]: from='client.14688 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T19:16:05.992 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:16:05 smithi130 ceph-mon[97832]: from='client.14700 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T19:16:05.992 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:16:05 smithi130 ceph-mon[97832]: mds.? [v2:172.21.15.191:6836/2406892975,v1:172.21.15.191:6837/2406892975] up:active 2023-07-14T19:16:05.993 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:16:05 smithi130 ceph-mon[97832]: daemon mds.e assigned to filesystem cephfs as rank 4 (now has 5 ranks) 2023-07-14T19:16:05.993 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:16:05 smithi130 ceph-mon[97832]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2023-07-14T19:16:05.993 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:16:05 smithi130 ceph-mon[97832]: Cluster is now healthy 2023-07-14T19:16:05.993 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:16:05 smithi130 ceph-mon[97832]: fsmap cephfs:4 {0=b=up:active,1=i=up:active,2=h=up:active,3=c=up:active} 8 up:standby 2023-07-14T19:16:05.993 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:16:05 smithi130 ceph-mon[97832]: fsmap cephfs:5 1 up:creating 7 up:standby 4 up:active 2023-07-14T19:16:05.993 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:16:05 smithi130 ceph-mon[97832]: daemon mds.e is now active in filesystem cephfs as rank 4 2023-07-14T19:16:05.994 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:05.993+0000 7f74f594f700 1 -- 172.21.15.40:0/128476494 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f74dc005cc0 con 0x7f74f01006d0 2023-07-14T19:16:05.996 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:05.994+0000 7f74f494d700 1 -- 172.21.15.40:0/128476494 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v81) v1 ==== 74+0+22121 (secure 0 0 0) 0x7f74e8099120 con 0x7f74f01006d0 2023-07-14T19:16:05.996 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-07-14T19:16:05.996 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":81,"fsid":"adc52f04-2279-11ee-9b33-001a4aab830c","created":"2023-07-14T19:08:56.893730+0000","modified":"2023-07-14T19:15:59.225912+0000","last_up_change":"2023-07-14T19:15:16.632979+0000","last_in_change":"2023-07-14T19:14:58.961702+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-07-14T19:12:03.271333+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-07-14T19:15:56.073464+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-07-14T19:15:57.018726+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"0d79356a-95d2-49ad-857b-2092ae9f7e13","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":3599703125},{"type":"v1","addr":"172.21.15.40:6803","nonce":3599703125}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":3599703125},{"type":"v1","addr":"172.21.15.40:6805","nonce":3599703125}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":3599703125},{"type":"v1","addr":"172.21.15.40:6809","nonce":3599703125}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":3599703125},{"type":"v1","addr":"172.21.15.40:6807","nonce":3599703125}]},"public_addr":"172.21.15.40:6803/3599703125","cluster_addr":"172.21.15.40:6805/3599703125","heartbeat_back_addr":"172.21.15.40:6809/3599703125","heartbeat_front_addr":"172.21.15.40:6807/3599703125","state":["exists","up"]},{"osd":1,"uuid":"40f8a3d1-a5e4-473c-b9f2-7600ec22be0d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6800","nonce":2512760702},{"type":"v1","addr":"172.21.15.130:6801","nonce":2512760702}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6802","nonce":2512760702},{"type":"v1","addr":"172.21.15.130:6803","nonce":2512760702}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6806","nonce":2512760702},{"type":"v1","addr":"172.21.15.130:6807","nonce":2512760702}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6804","nonce":2512760702},{"type":"v1","addr":"172.21.15.130:6805","nonce":2512760702}]},"public_addr":"172.21.15.130:6801/2512760702","cluster_addr":"172.21.15.130:6803/2512760702","heartbeat_back_addr":"172.21.15.130:6807/2512760702","heartbeat_front_addr":"172.21.15.130:6805/2512760702","state":["exists","up"]},{"osd":2,"uuid":"5a157891-cb7a-4ea6-9239-f48a7c67c003","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6800","nonce":589569135},{"type":"v1","addr":"172.21.15.191:6801","nonce":589569135}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6802","nonce":589569135},{"type":"v1","addr":"172.21.15.191:6803","nonce":589569135}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6806","nonce":589569135},{"type":"v1","addr":"172.21.15.191:6807","nonce":589569135}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6804","nonce":589569135},{"type":"v1","addr":"172.21.15.191:6805","nonce":589569135}]},"public_addr":"172.21.15.191:6801/589569135","cluster_addr":"172.21.15.191:6803/589569135","heartbeat_back_addr":"172.21.15.191:6807/589569135","heartbeat_front_addr":"172.21.15.191:6805/589569135","state":["exists","up"]},{"osd":3,"uuid":"8020e7dc-ac40-487f-857a-8da40bccb459","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":996287860},{"type":"v1","addr":"172.21.15.40:6811","nonce":996287860}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6812","nonce":996287860},{"type":"v1","addr":"172.21.15.40:6813","nonce":996287860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6816","nonce":996287860},{"type":"v1","addr":"172.21.15.40:6817","nonce":996287860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":996287860},{"type":"v1","addr":"172.21.15.40:6815","nonce":996287860}]},"public_addr":"172.21.15.40:6811/996287860","cluster_addr":"172.21.15.40:6813/996287860","heartbeat_back_addr":"172.21.15.40:6817/996287860","heartbeat_front_addr":"172.21.15.40:6815/996287860","state":["exists","up"]},{"osd":4,"uuid":"c5501b35-f62d-482d-8d22-115c88f0298e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6808","nonce":3507448720},{"type":"v1","addr":"172.21.15.130:6809","nonce":3507448720}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6810","nonce":3507448720},{"type":"v1","addr":"172.21.15.130:6811","nonce":3507448720}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6814","nonce":3507448720},{"type":"v1","addr":"172.21.15.130:6815","nonce":3507448720}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6812","nonce":3507448720},{"type":"v1","addr":"172.21.15.130:6813","nonce":3507448720}]},"public_addr":"172.21.15.130:6809/3507448720","cluster_addr":"172.21.15.130:6811/3507448720","heartbeat_back_addr":"172.21.15.130:6815/3507448720","heartbeat_front_addr":"172.21.15.130:6813/3507448720","state":["exists","up"]},{"osd":5,"uuid":"7770b6d6-5874-493a-986a-84db482cee42","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6808","nonce":1781944663},{"type":"v1","addr":"172.21.15.191:6809","nonce":1781944663}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6810","nonce":1781944663},{"type":"v1","addr":"172.21.15.191:6811","nonce":1781944663}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6814","nonce":1781944663},{"type":"v1","addr":"172.21.15.191:6815","nonce":1781944663}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6812","nonce":1781944663},{"type":"v1","addr":"172.21.15.191:6813","nonce":1781944663}]},"public_addr":"172.21.15.191:6809/1781944663","cluster_addr":"172.21.15.191:6811/1781944663","heartbeat_back_addr":"172.21.15.191:6815/1781944663","heartbeat_front_addr":"172.21.15.191:6813/1781944663","state":["exists","up"]},{"osd":6,"uuid":"ef65ad00-dfee-4300-9961-dfc8573aab06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6818","nonce":693186773},{"type":"v1","addr":"172.21.15.40:6819","nonce":693186773}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6820","nonce":693186773},{"type":"v1","addr":"172.21.15.40:6821","nonce":693186773}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6824","nonce":693186773},{"type":"v1","addr":"172.21.15.40:6825","nonce":693186773}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6822","nonce":693186773},{"type":"v1","addr":"172.21.15.40:6823","nonce":693186773}]},"public_addr":"172.21.15.40:6819/693186773","cluster_addr":"172.21.15.40:6821/693186773","heartbeat_back_addr":"172.21.15.40:6825/693186773","heartbeat_front_addr":"172.21.15.40:6823/693186773","state":["exists","up"]},{"osd":7,"uuid":"1700fbfa-cbfc-4f76-97be-727a3b100740","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":50,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6816","nonce":2384019528},{"type":"v1","addr":"172.21.15.130:6817","nonce":2384019528}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6818","nonce":2384019528},{"type":"v1","addr":"172.21.15.130:6819","nonce":2384019528}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6822","nonce":2384019528},{"type":"v1","addr":"172.21.15.130:6823","nonce":2384019528}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6820","nonce":2384019528},{"type":"v1","addr":"172.21.15.130:6821","nonce":2384019528}]},"public_addr":"172.21.15.130:6817/2384019528","cluster_addr":"172.21.15.130:6819/2384019528","heartbeat_back_addr":"172.21.15.130:6823/2384019528","heartbeat_front_addr":"172.21.15.130:6821/2384019528","state":["exists","up"]},{"osd":8,"uuid":"b76d4033-c1e2-457b-8571-c8a49fe443f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":57,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6816","nonce":2386818417},{"type":"v1","addr":"172.21.15.191:6817","nonce":2386818417}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6818","nonce":2386818417},{"type":"v1","addr":"172.21.15.191:6819","nonce":2386818417}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6822","nonce":2386818417},{"type":"v1","addr":"172.21.15.191:6823","nonce":2386818417}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6820","nonce":2386818417},{"type":"v1","addr":"172.21.15.191:6821","nonce":2386818417}]},"public_addr":"172.21.15.191:6817/2386818417","cluster_addr":"172.21.15.191:6819/2386818417","heartbeat_back_addr":"172.21.15.191:6823/2386818417","heartbeat_front_addr":"172.21.15.191:6821/2386818417","state":["exists","up"]},{"osd":9,"uuid":"df31bf2f-e30e-41b6-b2b9-0e26a07a88bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":64,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6826","nonce":2228380263},{"type":"v1","addr":"172.21.15.40:6827","nonce":2228380263}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6828","nonce":2228380263},{"type":"v1","addr":"172.21.15.40:6829","nonce":2228380263}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":2228380263},{"type":"v1","addr":"172.21.15.40:6833","nonce":2228380263}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6830","nonce":2228380263},{"type":"v1","addr":"172.21.15.40:6831","nonce":2228380263}]},"public_addr":"172.21.15.40:6827/2228380263","cluster_addr":"172.21.15.40:6829/2228380263","heartbeat_back_addr":"172.21.15.40:6833/2228380263","heartbeat_front_addr":"172.21.15.40:6831/2228380263","state":["exists","up"]},{"osd":10,"uuid":"4a93d621-49ee-4ad7-9e49-750614faf912","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6824","nonce":2417067798},{"type":"v1","addr":"172.21.15.130:6825","nonce":2417067798}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6826","nonce":2417067798},{"type":"v1","addr":"172.21.15.130:6827","nonce":2417067798}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6830","nonce":2417067798},{"type":"v1","addr":"172.21.15.130:6831","nonce":2417067798}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6828","nonce":2417067798},{"type":"v1","addr":"172.21.15.130:6829","nonce":2417067798}]},"public_addr":"172.21.15.130:6825/2417067798","cluster_addr":"172.21.15.130:6827/2417067798","heartbeat_back_addr":"172.21.15.130:6831/2417067798","heartbeat_front_addr":"172.21.15.130:6829/2417067798","state":["exists","up"]},{"osd":11,"uuid":"f43a2a6f-82f3-42ac-b77c-4f1b34da6918","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":76,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6824","nonce":2886059446},{"type":"v1","addr":"172.21.15.191:6825","nonce":2886059446}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6826","nonce":2886059446},{"type":"v1","addr":"172.21.15.191:6827","nonce":2886059446}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6830","nonce":2886059446},{"type":"v1","addr":"172.21.15.191:6831","nonce":2886059446}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6828","nonce":2886059446},{"type":"v1","addr":"172.21.15.191:6829","nonce":2886059446}]},"public_addr":"172.21.15.191:6825/2886059446","cluster_addr":"172.21.15.191:6827/2886059446","heartbeat_back_addr":"172.21.15.191:6831/2886059446","heartbeat_front_addr":"172.21.15.191:6829/2886059446","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:11:18.320495+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:11:39.901199+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:12:00.869042+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:12:23.292742+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:12:44.587019+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:13:06.494893+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:13:28.378374+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:13:50.033491+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:14:10.937062+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:14:32.358498+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:14:53.666584+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:15:14.837566+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.40:6801/3698757264":"2023-07-15T19:09:50.732458+0000","172.21.15.40:6800/3698757264":"2023-07-15T19:09:50.732458+0000","172.21.15.40:6800/2569015667":"2023-07-15T19:09:24.441543+0000","172.21.15.40:0/3653961003":"2023-07-15T19:09:24.441543+0000","172.21.15.40:0/3568861680":"2023-07-15T19:09:24.441543+0000","172.21.15.40:0/3249773172":"2023-07-15T19:09:50.732458+0000","172.21.15.40:0/4084215480":"2023-07-15T19:09:24.441543+0000","172.21.15.40:0/1163941760":"2023-07-15T19:09:50.732458+0000","172.21.15.40:6801/2569015667":"2023-07-15T19:09:24.441543+0000","172.21.15.40:0/2326639960":"2023-07-15T19:09:50.732458+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-07-14T19:16:05.998 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:05.996+0000 7f74f594f700 1 -- 172.21.15.40:0/128476494 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f74d80a1220 msgr2=0x7f74d80a36d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:16:05.998 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:05.996+0000 7f74f594f700 1 --2- 172.21.15.40:0/128476494 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f74d80a1220 0x7f74d80a36d0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f74e0006010 tx=0x7f74e000b540 comp rx=0 tx=0).stop 2023-07-14T19:16:05.999 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:05.996+0000 7f74f594f700 1 -- 172.21.15.40:0/128476494 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f74f01006d0 msgr2=0x7f74f019cc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:16:05.999 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:05.996+0000 7f74f594f700 1 --2- 172.21.15.40:0/128476494 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f74f01006d0 0x7f74f019cc90 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f74e8007970 tx=0x7f74e8007c80 comp rx=0 tx=0).stop 2023-07-14T19:16:05.999 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:05.996+0000 7f74f594f700 1 -- 172.21.15.40:0/128476494 shutdown_connections 2023-07-14T19:16:05.999 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:05.997+0000 7f74f594f700 1 --2- 172.21.15.40:0/128476494 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f74d80a1220 0x7f74d80a36d0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:05.999 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:05.997+0000 7f74f594f700 1 --2- 172.21.15.40:0/128476494 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f74f01006d0 0x7f74f019cc90 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:05.999 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:05.997+0000 7f74f594f700 1 --2- 172.21.15.40:0/128476494 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f74f00feb50 0x7f74f00695f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:05.999 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:05.997+0000 7f74f594f700 1 --2- 172.21.15.40:0/128476494 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f74f00ffd40 0x7f74f019a640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:05.999 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:05.997+0000 7f74f594f700 1 -- 172.21.15.40:0/128476494 >> 172.21.15.40:0/128476494 conn(0x7f74f00fa150 msgr2=0x7f74f01031e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:16:06.000 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:05.997+0000 7f74f594f700 1 -- 172.21.15.40:0/128476494 shutdown_connections 2023-07-14T19:16:06.000 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:05.997+0000 7f74f594f700 1 -- 172.21.15.40:0/128476494 wait complete. 2023-07-14T19:16:06.011 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2023-07-14T19:16:06.012 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-07-14T19:16:06.137 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:06.135+0000 7f0d136cd700 1 -- 172.21.15.40:0/1687899498 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0d0c0feb00 msgr2=0x7f0d0c0fef10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:16:06.138 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:06.135+0000 7f0d136cd700 1 --2- 172.21.15.40:0/1687899498 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0d0c0feb00 0x7f0d0c0fef10 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f0d08009990 tx=0x7f0d08009ca0 comp rx=0 tx=0).stop 2023-07-14T19:16:06.138 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:06.135+0000 7f0d136cd700 1 -- 172.21.15.40:0/1687899498 shutdown_connections 2023-07-14T19:16:06.138 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:06.135+0000 7f0d136cd700 1 --2- 172.21.15.40:0/1687899498 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0d0c100680 0x7f0d0c106f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:06.138 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:06.135+0000 7f0d136cd700 1 --2- 172.21.15.40:0/1687899498 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0d0c0feb00 0x7f0d0c0fef10 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:06.139 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:06.135+0000 7f0d136cd700 1 --2- 172.21.15.40:0/1687899498 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0d0c0ffcf0 0x7f0d0c100140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:06.139 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:06.135+0000 7f0d136cd700 1 -- 172.21.15.40:0/1687899498 >> 172.21.15.40:0/1687899498 conn(0x7f0d0c0fa100 msgr2=0x7f0d0c0fc550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:16:06.139 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:06.136+0000 7f0d136cd700 1 -- 172.21.15.40:0/1687899498 shutdown_connections 2023-07-14T19:16:06.139 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:06.136+0000 7f0d136cd700 1 -- 172.21.15.40:0/1687899498 wait complete. 2023-07-14T19:16:06.143 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:06.136+0000 7f0d136cd700 1 Processor -- start 2023-07-14T19:16:06.144 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:06.136+0000 7f0d136cd700 1 -- start start 2023-07-14T19:16:06.144 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:06.136+0000 7f0d136cd700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0d0c0feb00 0x7f0d0c195970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:16:06.144 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:06.136+0000 7f0d136cd700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0d0c0ffcf0 0x7f0d0c195eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:16:06.145 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:06.136+0000 7f0d136cd700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0d0c100680 0x7f0d0c19cf20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:16:06.145 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:06.136+0000 7f0d136cd700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d0c19d4f0 con 0x7f0d0c0feb00 2023-07-14T19:16:06.145 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:06.136+0000 7f0d136cd700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d0c19d630 con 0x7f0d0c100680 2023-07-14T19:16:06.146 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:06.136+0000 7f0d136cd700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d0c19d7c0 con 0x7f0d0c0ffcf0 2023-07-14T19:16:06.146 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:06.136+0000 7f0d11469700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0d0c0feb00 0x7f0d0c195970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:16:06.146 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:06.136+0000 7f0d11469700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0d0c0feb00 0x7f0d0c195970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:33062/0 (socket says 172.21.15.40:33062) 2023-07-14T19:16:06.146 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:06.136+0000 7f0d11469700 1 -- 172.21.15.40:0/946289386 learned_addr learned my addr 172.21.15.40:0/946289386 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-07-14T19:16:06.147 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:06.137+0000 7f0d10c68700 1 --2- 172.21.15.40:0/946289386 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0d0c0ffcf0 0x7f0d0c195eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:16:06.147 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:06.137+0000 7f0d11c6a700 1 --2- 172.21.15.40:0/946289386 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0d0c100680 0x7f0d0c19cf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:16:06.147 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:06.137+0000 7f0d11469700 1 -- 172.21.15.40:0/946289386 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0d0c0ffcf0 msgr2=0x7f0d0c195eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:16:06.147 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:06.137+0000 7f0d11469700 1 --2- 172.21.15.40:0/946289386 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0d0c0ffcf0 0x7f0d0c195eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:06.148 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:06.137+0000 7f0d11469700 1 -- 172.21.15.40:0/946289386 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0d0c100680 msgr2=0x7f0d0c19cf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:16:06.148 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:06.137+0000 7f0d11469700 1 --2- 172.21.15.40:0/946289386 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0d0c100680 0x7f0d0c19cf20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:06.148 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:06.137+0000 7f0d11469700 1 -- 172.21.15.40:0/946289386 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d080095f0 con 0x7f0d0c0feb00 2023-07-14T19:16:06.148 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:06.137+0000 7f0d11c6a700 1 --2- 172.21.15.40:0/946289386 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0d0c100680 0x7f0d0c19cf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T19:16:06.149 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:06.137+0000 7f0d11469700 1 --2- 172.21.15.40:0/946289386 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0d0c0feb00 0x7f0d0c195970 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f0d08004760 tx=0x7f0d08009580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:16:06.149 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:06.137+0000 7f0d027fc700 1 -- 172.21.15.40:0/946289386 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d0801d070 con 0x7f0d0c0feb00 2023-07-14T19:16:06.149 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:06.137+0000 7f0d027fc700 1 -- 172.21.15.40:0/946289386 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f0d08004120 con 0x7f0d0c0feb00 2023-07-14T19:16:06.149 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:06.137+0000 7f0d027fc700 1 -- 172.21.15.40:0/946289386 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d08005290 con 0x7f0d0c0feb00 2023-07-14T19:16:06.150 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:06.137+0000 7f0d136cd700 1 -- 172.21.15.40:0/946289386 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0d0c19da40 con 0x7f0d0c0feb00 2023-07-14T19:16:06.150 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:06.137+0000 7f0d136cd700 1 -- 172.21.15.40:0/946289386 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0d0c19de60 con 0x7f0d0c0feb00 2023-07-14T19:16:06.150 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:06.138+0000 7f0d136cd700 1 -- 172.21.15.40:0/946289386 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0d0c10b4a0 con 0x7f0d0c0feb00 2023-07-14T19:16:06.150 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:06.139+0000 7f0d027fc700 1 -- 172.21.15.40:0/946289386 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7f0d08004810 con 0x7f0d0c0feb00 2023-07-14T19:16:06.150 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:06.140+0000 7f0d027fc700 1 --2- 172.21.15.40:0/946289386 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f0cf80a12a0 0x7f0cf80a3750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:16:06.150 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:06.140+0000 7f0d10c68700 1 --2- 172.21.15.40:0/946289386 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f0cf80a12a0 0x7f0cf80a3750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:16:06.150 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:06.140+0000 7f0d027fc700 1 -- 172.21.15.40:0/946289386 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f0d080cd9d0 con 0x7f0d0c0feb00 2023-07-14T19:16:06.151 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:06.140+0000 7f0d10c68700 1 --2- 172.21.15.40:0/946289386 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f0cf80a12a0 0x7f0cf80a3750 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f0cfc005950 tx=0x7f0cfc0058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:16:06.151 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:06.144+0000 7f0d027fc700 1 -- 172.21.15.40:0/946289386 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f0d08097100 con 0x7f0d0c0feb00 2023-07-14T19:16:06.350 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:06.348+0000 7f0d136cd700 1 -- 172.21.15.40:0/946289386 --> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f0d0c060a80 con 0x7f0cf80a12a0 2023-07-14T19:16:06.363 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:06.361+0000 7f0d027fc700 1 -- 172.21.15.40:0/946289386 <== mgr.14152 v2:172.21.15.40:6800/1693027118 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+335894 (secure 0 0 0) 0x7f0d0c060a80 con 0x7f0cf80a12a0 2023-07-14T19:16:06.367 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-07-14T19:16:06.369 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:06.365+0000 7f0d136cd700 1 -- 172.21.15.40:0/946289386 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f0cf80a12a0 msgr2=0x7f0cf80a3750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:16:06.369 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:06.365+0000 7f0d136cd700 1 --2- 172.21.15.40:0/946289386 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f0cf80a12a0 0x7f0cf80a3750 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f0cfc005950 tx=0x7f0cfc0058e0 comp rx=0 tx=0).stop 2023-07-14T19:16:06.370 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:06.365+0000 7f0d136cd700 1 -- 172.21.15.40:0/946289386 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0d0c0feb00 msgr2=0x7f0d0c195970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:16:06.370 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:06.365+0000 7f0d136cd700 1 --2- 172.21.15.40:0/946289386 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0d0c0feb00 0x7f0d0c195970 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f0d08004760 tx=0x7f0d08009580 comp rx=0 tx=0).stop 2023-07-14T19:16:06.370 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:06.365+0000 7f0d136cd700 1 -- 172.21.15.40:0/946289386 shutdown_connections 2023-07-14T19:16:06.370 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:06.365+0000 7f0d136cd700 1 --2- 172.21.15.40:0/946289386 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f0cf80a12a0 0x7f0cf80a3750 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:06.370 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:06.365+0000 7f0d136cd700 1 --2- 172.21.15.40:0/946289386 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0d0c100680 0x7f0d0c19cf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:06.370 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:06.365+0000 7f0d136cd700 1 --2- 172.21.15.40:0/946289386 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0d0c0feb00 0x7f0d0c195970 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:06.370 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:06.365+0000 7f0d136cd700 1 --2- 172.21.15.40:0/946289386 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0d0c0ffcf0 0x7f0d0c195eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:06.371 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:06.365+0000 7f0d136cd700 1 -- 172.21.15.40:0/946289386 >> 172.21.15.40:0/946289386 conn(0x7f0d0c0fa100 msgr2=0x7f0d0c0fc320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:16:06.371 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:06.365+0000 7f0d136cd700 1 -- 172.21.15.40:0/946289386 shutdown_connections 2023-07-14T19:16:06.371 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:06.365+0000 7f0d136cd700 1 -- 172.21.15.40:0/946289386 wait complete. 2023-07-14T19:16:06.371 INFO:teuthology.orchestra.run.smithi040.stderr:dumped all 2023-07-14T19:16:06.379 INFO:teuthology.orchestra.run.smithi040.stdout:{"pg_ready":true,"pg_map":{"version":258,"stamp":"2023-07-14T19:16:04.809979+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":594602,"num_objects":53,"num_object_clones":0,"num_object_copies":159,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":293,"num_write_kb":2971,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":36,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":233,"ondisk_log_size":233,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":251,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":334696,"kb_used_data":10468,"kb_used_omap":20,"kb_used_meta":324075,"kb_avail":1124607128,"statfs":{"total":1151940427776,"available":1151597699072,"internally_reserved":0,"allocated":10719232,"data_stored":6770996,"data_compressed":107142,"data_compressed_allocated":135168,"data_compressed_original":1769472,"omap_allocated":21372,"internal_metadata":331852932},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":4234,"num_objects":51,"num_object_clones":0,"num_object_copies":153,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":51,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":112,"num_write_kb":33,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":36,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.585197"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-07-14T19:15:59.938963+0000","last_change":"2023-07-14T19:15:58.250082+0000","last_active":"2023-07-14T19:15:59.938963+0000","last_peered":"2023-07-14T19:15:59.938963+0000","last_clean":"2023-07-14T19:15:59.938963+0000","last_became_active":"2023-07-14T19:15:58.249783+0000","last_became_peered":"2023-07-14T19:15:58.249783+0000","last_unstale":"2023-07-14T19:15:59.938963+0000","last_undegraded":"2023-07-14T19:15:59.938963+0000","last_fullsized":"2023-07-14T19:15:59.938963+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_clean_scrub_stamp":"2023-07-14T19:15:57.219223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T03:38:35.618742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-07-14T19:15:59.573145+0000","last_change":"2023-07-14T19:15:58.240082+0000","last_active":"2023-07-14T19:15:59.573145+0000","last_peered":"2023-07-14T19:15:59.573145+0000","last_clean":"2023-07-14T19:15:59.573145+0000","last_became_active":"2023-07-14T19:15:58.239788+0000","last_became_peered":"2023-07-14T19:15:58.239788+0000","last_unstale":"2023-07-14T19:15:59.573145+0000","last_undegraded":"2023-07-14T19:15:59.573145+0000","last_fullsized":"2023-07-14T19:15:59.573145+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_clean_scrub_stamp":"2023-07-14T19:15:57.219223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T05:20:02.889590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-07-14T19:15:59.573092+0000","last_change":"2023-07-14T19:15:58.248010+0000","last_active":"2023-07-14T19:15:59.573092+0000","last_peered":"2023-07-14T19:15:59.573092+0000","last_clean":"2023-07-14T19:15:59.573092+0000","last_became_active":"2023-07-14T19:15:58.247755+0000","last_became_peered":"2023-07-14T19:15:58.247755+0000","last_unstale":"2023-07-14T19:15:59.573092+0000","last_undegraded":"2023-07-14T19:15:59.573092+0000","last_fullsized":"2023-07-14T19:15:59.573092+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_clean_scrub_stamp":"2023-07-14T19:15:57.219223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T19:47:43.289654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-07-14T19:15:59.229952+0000","last_change":"2023-07-14T19:15:58.246946+0000","last_active":"2023-07-14T19:15:59.229952+0000","last_peered":"2023-07-14T19:15:59.229952+0000","last_clean":"2023-07-14T19:15:59.229952+0000","last_became_active":"2023-07-14T19:15:58.246598+0000","last_became_peered":"2023-07-14T19:15:58.246598+0000","last_unstale":"2023-07-14T19:15:59.229952+0000","last_undegraded":"2023-07-14T19:15:59.229952+0000","last_fullsized":"2023-07-14T19:15:59.229952+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_clean_scrub_stamp":"2023-07-14T19:15:57.219223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T06:05:42.069121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-07-14T19:15:59.232238+0000","last_change":"2023-07-14T19:15:58.252632+0000","last_active":"2023-07-14T19:15:59.232238+0000","last_peered":"2023-07-14T19:15:59.232238+0000","last_clean":"2023-07-14T19:15:59.232238+0000","last_became_active":"2023-07-14T19:15:58.252319+0000","last_became_peered":"2023-07-14T19:15:58.252319+0000","last_unstale":"2023-07-14T19:15:59.232238+0000","last_undegraded":"2023-07-14T19:15:59.232238+0000","last_fullsized":"2023-07-14T19:15:59.232238+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_clean_scrub_stamp":"2023-07-14T19:15:57.219223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T06:07:06.243539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-07-14T19:15:59.722095+0000","last_change":"2023-07-14T19:15:58.248305+0000","last_active":"2023-07-14T19:15:59.722095+0000","last_peered":"2023-07-14T19:15:59.722095+0000","last_clean":"2023-07-14T19:15:59.722095+0000","last_became_active":"2023-07-14T19:15:58.247347+0000","last_became_peered":"2023-07-14T19:15:58.247347+0000","last_unstale":"2023-07-14T19:15:59.722095+0000","last_undegraded":"2023-07-14T19:15:59.722095+0000","last_fullsized":"2023-07-14T19:15:59.722095+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_clean_scrub_stamp":"2023-07-14T19:15:57.219223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T05:16:26.859500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-07-14T19:15:59.721610+0000","last_change":"2023-07-14T19:15:58.240871+0000","last_active":"2023-07-14T19:15:59.721610+0000","last_peered":"2023-07-14T19:15:59.721610+0000","last_clean":"2023-07-14T19:15:59.721610+0000","last_became_active":"2023-07-14T19:15:58.240574+0000","last_became_peered":"2023-07-14T19:15:58.240574+0000","last_unstale":"2023-07-14T19:15:59.721610+0000","last_undegraded":"2023-07-14T19:15:59.721610+0000","last_fullsized":"2023-07-14T19:15:59.721610+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_clean_scrub_stamp":"2023-07-14T19:15:57.219223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T00:07:11.508517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-07-14T19:15:59.721987+0000","last_change":"2023-07-14T19:15:58.241979+0000","last_active":"2023-07-14T19:15:59.721987+0000","last_peered":"2023-07-14T19:15:59.721987+0000","last_clean":"2023-07-14T19:15:59.721987+0000","last_became_active":"2023-07-14T19:15:58.241111+0000","last_became_peered":"2023-07-14T19:15:58.241111+0000","last_unstale":"2023-07-14T19:15:59.721987+0000","last_undegraded":"2023-07-14T19:15:59.721987+0000","last_fullsized":"2023-07-14T19:15:59.721987+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_clean_scrub_stamp":"2023-07-14T19:15:57.219223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T07:04:12.177825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-07-14T19:15:59.721751+0000","last_change":"2023-07-14T19:15:58.248866+0000","last_active":"2023-07-14T19:15:59.721751+0000","last_peered":"2023-07-14T19:15:59.721751+0000","last_clean":"2023-07-14T19:15:59.721751+0000","last_became_active":"2023-07-14T19:15:58.248456+0000","last_became_peered":"2023-07-14T19:15:58.248456+0000","last_unstale":"2023-07-14T19:15:59.721751+0000","last_undegraded":"2023-07-14T19:15:59.721751+0000","last_fullsized":"2023-07-14T19:15:59.721751+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_clean_scrub_stamp":"2023-07-14T19:15:57.219223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:28:00.287289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-07-14T19:15:59.722575+0000","last_change":"2023-07-14T19:15:58.248296+0000","last_active":"2023-07-14T19:15:59.722575+0000","last_peered":"2023-07-14T19:15:59.722575+0000","last_clean":"2023-07-14T19:15:59.722575+0000","last_became_active":"2023-07-14T19:15:58.247810+0000","last_became_peered":"2023-07-14T19:15:58.247810+0000","last_unstale":"2023-07-14T19:15:59.722575+0000","last_undegraded":"2023-07-14T19:15:59.722575+0000","last_fullsized":"2023-07-14T19:15:59.722575+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_clean_scrub_stamp":"2023-07-14T19:15:57.219223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T03:06:56.929583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-07-14T19:15:59.722644+0000","last_change":"2023-07-14T19:15:58.252057+0000","last_active":"2023-07-14T19:15:59.722644+0000","last_peered":"2023-07-14T19:15:59.722644+0000","last_clean":"2023-07-14T19:15:59.722644+0000","last_became_active":"2023-07-14T19:15:58.251425+0000","last_became_peered":"2023-07-14T19:15:58.251425+0000","last_unstale":"2023-07-14T19:15:59.722644+0000","last_undegraded":"2023-07-14T19:15:59.722644+0000","last_fullsized":"2023-07-14T19:15:59.722644+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_clean_scrub_stamp":"2023-07-14T19:15:57.219223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T00:23:30.017734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-07-14T19:15:59.229817+0000","last_change":"2023-07-14T19:15:58.241009+0000","last_active":"2023-07-14T19:15:59.229817+0000","last_peered":"2023-07-14T19:15:59.229817+0000","last_clean":"2023-07-14T19:15:59.229817+0000","last_became_active":"2023-07-14T19:15:58.240622+0000","last_became_peered":"2023-07-14T19:15:58.240622+0000","last_unstale":"2023-07-14T19:15:59.229817+0000","last_undegraded":"2023-07-14T19:15:59.229817+0000","last_fullsized":"2023-07-14T19:15:59.229817+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_clean_scrub_stamp":"2023-07-14T19:15:57.219223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T05:28:57.780030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-07-14T19:15:59.722108+0000","last_change":"2023-07-14T19:15:58.243377+0000","last_active":"2023-07-14T19:15:59.722108+0000","last_peered":"2023-07-14T19:15:59.722108+0000","last_clean":"2023-07-14T19:15:59.722108+0000","last_became_active":"2023-07-14T19:15:58.242225+0000","last_became_peered":"2023-07-14T19:15:58.242225+0000","last_unstale":"2023-07-14T19:15:59.722108+0000","last_undegraded":"2023-07-14T19:15:59.722108+0000","last_fullsized":"2023-07-14T19:15:59.722108+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_clean_scrub_stamp":"2023-07-14T19:15:57.219223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:21:59.269854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-07-14T19:15:59.573307+0000","last_change":"2023-07-14T19:15:58.244556+0000","last_active":"2023-07-14T19:15:59.573307+0000","last_peered":"2023-07-14T19:15:59.573307+0000","last_clean":"2023-07-14T19:15:59.573307+0000","last_became_active":"2023-07-14T19:15:58.244228+0000","last_became_peered":"2023-07-14T19:15:58.244228+0000","last_unstale":"2023-07-14T19:15:59.573307+0000","last_undegraded":"2023-07-14T19:15:59.573307+0000","last_fullsized":"2023-07-14T19:15:59.573307+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_clean_scrub_stamp":"2023-07-14T19:15:57.219223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:11:22.589361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-07-14T19:15:59.572875+0000","last_change":"2023-07-14T19:15:58.250176+0000","last_active":"2023-07-14T19:15:59.572875+0000","last_peered":"2023-07-14T19:15:59.572875+0000","last_clean":"2023-07-14T19:15:59.572875+0000","last_became_active":"2023-07-14T19:15:58.249708+0000","last_became_peered":"2023-07-14T19:15:58.249708+0000","last_unstale":"2023-07-14T19:15:59.572875+0000","last_undegraded":"2023-07-14T19:15:59.572875+0000","last_fullsized":"2023-07-14T19:15:59.572875+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_clean_scrub_stamp":"2023-07-14T19:15:57.219223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T22:33:15.826269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-07-14T19:15:59.721419+0000","last_change":"2023-07-14T19:15:58.246859+0000","last_active":"2023-07-14T19:15:59.721419+0000","last_peered":"2023-07-14T19:15:59.721419+0000","last_clean":"2023-07-14T19:15:59.721419+0000","last_became_active":"2023-07-14T19:15:58.246541+0000","last_became_peered":"2023-07-14T19:15:58.246541+0000","last_unstale":"2023-07-14T19:15:59.721419+0000","last_undegraded":"2023-07-14T19:15:59.721419+0000","last_fullsized":"2023-07-14T19:15:59.721419+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_clean_scrub_stamp":"2023-07-14T19:15:57.219223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T03:07:22.056544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-07-14T19:15:59.939090+0000","last_change":"2023-07-14T19:15:58.250112+0000","last_active":"2023-07-14T19:15:59.939090+0000","last_peered":"2023-07-14T19:15:59.939090+0000","last_clean":"2023-07-14T19:15:59.939090+0000","last_became_active":"2023-07-14T19:15:58.249824+0000","last_became_peered":"2023-07-14T19:15:58.249824+0000","last_unstale":"2023-07-14T19:15:59.939090+0000","last_undegraded":"2023-07-14T19:15:59.939090+0000","last_fullsized":"2023-07-14T19:15:59.939090+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_clean_scrub_stamp":"2023-07-14T19:15:57.219223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T22:34:26.596011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-07-14T19:15:59.723229+0000","last_change":"2023-07-14T19:15:58.243011+0000","last_active":"2023-07-14T19:15:59.723229+0000","last_peered":"2023-07-14T19:15:59.723229+0000","last_clean":"2023-07-14T19:15:59.723229+0000","last_became_active":"2023-07-14T19:15:58.242693+0000","last_became_peered":"2023-07-14T19:15:58.242693+0000","last_unstale":"2023-07-14T19:15:59.723229+0000","last_undegraded":"2023-07-14T19:15:59.723229+0000","last_fullsized":"2023-07-14T19:15:59.723229+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_clean_scrub_stamp":"2023-07-14T19:15:57.219223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:04:37.187798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-07-14T19:15:59.721093+0000","last_change":"2023-07-14T19:15:58.251298+0000","last_active":"2023-07-14T19:15:59.721093+0000","last_peered":"2023-07-14T19:15:59.721093+0000","last_clean":"2023-07-14T19:15:59.721093+0000","last_became_active":"2023-07-14T19:15:58.250984+0000","last_became_peered":"2023-07-14T19:15:58.250984+0000","last_unstale":"2023-07-14T19:15:59.721093+0000","last_undegraded":"2023-07-14T19:15:59.721093+0000","last_fullsized":"2023-07-14T19:15:59.721093+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_clean_scrub_stamp":"2023-07-14T19:15:57.219223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T20:29:32.350678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"81'11","reported_seq":25,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-07-14T19:16:03.613085+0000","last_change":"2023-07-14T19:15:57.235142+0000","last_active":"2023-07-14T19:16:03.613085+0000","last_peered":"2023-07-14T19:16:03.613085+0000","last_clean":"2023-07-14T19:16:03.613085+0000","last_became_active":"2023-07-14T19:15:57.234803+0000","last_became_peered":"2023-07-14T19:15:57.234803+0000","last_unstale":"2023-07-14T19:16:03.613085+0000","last_undegraded":"2023-07-14T19:16:03.613085+0000","last_fullsized":"2023-07-14T19:16:03.613085+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:15:56.210166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:15:56.210166+0000","last_clean_scrub_stamp":"2023-07-14T19:15:56.210166+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:27:04.191229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-07-14T19:15:59.722681+0000","last_change":"2023-07-14T19:15:58.247688+0000","last_active":"2023-07-14T19:15:59.722681+0000","last_peered":"2023-07-14T19:15:59.722681+0000","last_clean":"2023-07-14T19:15:59.722681+0000","last_became_active":"2023-07-14T19:15:58.247400+0000","last_became_peered":"2023-07-14T19:15:58.247400+0000","last_unstale":"2023-07-14T19:15:59.722681+0000","last_undegraded":"2023-07-14T19:15:59.722681+0000","last_fullsized":"2023-07-14T19:15:59.722681+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_clean_scrub_stamp":"2023-07-14T19:15:57.219223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T21:28:57.315894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"80'1","reported_seq":15,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-07-14T19:15:59.722989+0000","last_change":"2023-07-14T19:15:57.240026+0000","last_active":"2023-07-14T19:15:59.722989+0000","last_peered":"2023-07-14T19:15:59.722989+0000","last_clean":"2023-07-14T19:15:59.722989+0000","last_became_active":"2023-07-14T19:15:57.239687+0000","last_became_peered":"2023-07-14T19:15:57.239687+0000","last_unstale":"2023-07-14T19:15:59.722989+0000","last_undegraded":"2023-07-14T19:15:59.722989+0000","last_fullsized":"2023-07-14T19:15:59.722989+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:15:56.210166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:15:56.210166+0000","last_clean_scrub_stamp":"2023-07-14T19:15:56.210166+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:18:22.780240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-07-14T19:15:59.573195+0000","last_change":"2023-07-14T19:15:58.247039+0000","last_active":"2023-07-14T19:15:59.573195+0000","last_peered":"2023-07-14T19:15:59.573195+0000","last_clean":"2023-07-14T19:15:59.573195+0000","last_became_active":"2023-07-14T19:15:58.246685+0000","last_became_peered":"2023-07-14T19:15:58.246685+0000","last_unstale":"2023-07-14T19:15:59.573195+0000","last_undegraded":"2023-07-14T19:15:59.573195+0000","last_fullsized":"2023-07-14T19:15:59.573195+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_clean_scrub_stamp":"2023-07-14T19:15:57.219223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T03:20:32.272596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"81'7","reported_seq":21,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-07-14T19:16:03.612329+0000","last_change":"2023-07-14T19:15:57.244612+0000","last_active":"2023-07-14T19:16:03.612329+0000","last_peered":"2023-07-14T19:16:03.612329+0000","last_clean":"2023-07-14T19:16:03.612329+0000","last_became_active":"2023-07-14T19:15:57.243761+0000","last_became_peered":"2023-07-14T19:15:57.243761+0000","last_unstale":"2023-07-14T19:16:03.612329+0000","last_undegraded":"2023-07-14T19:16:03.612329+0000","last_fullsized":"2023-07-14T19:16:03.612329+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:15:56.210166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:15:56.210166+0000","last_clean_scrub_stamp":"2023-07-14T19:15:56.210166+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T21:44:07.276569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-07-14T19:15:59.721874+0000","last_change":"2023-07-14T19:15:58.239903+0000","last_active":"2023-07-14T19:15:59.721874+0000","last_peered":"2023-07-14T19:15:59.721874+0000","last_clean":"2023-07-14T19:15:59.721874+0000","last_became_active":"2023-07-14T19:15:58.239627+0000","last_became_peered":"2023-07-14T19:15:58.239627+0000","last_unstale":"2023-07-14T19:15:59.721874+0000","last_undegraded":"2023-07-14T19:15:59.721874+0000","last_fullsized":"2023-07-14T19:15:59.721874+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_clean_scrub_stamp":"2023-07-14T19:15:57.219223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:56:30.086686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"80'2","reported_seq":16,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-07-14T19:15:59.232454+0000","last_change":"2023-07-14T19:15:57.241773+0000","last_active":"2023-07-14T19:15:59.232454+0000","last_peered":"2023-07-14T19:15:59.232454+0000","last_clean":"2023-07-14T19:15:59.232454+0000","last_became_active":"2023-07-14T19:15:57.241451+0000","last_became_peered":"2023-07-14T19:15:57.241451+0000","last_unstale":"2023-07-14T19:15:59.232454+0000","last_undegraded":"2023-07-14T19:15:59.232454+0000","last_fullsized":"2023-07-14T19:15:59.232454+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:15:56.210166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:15:56.210166+0000","last_clean_scrub_stamp":"2023-07-14T19:15:56.210166+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T00:53:37.977462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-07-14T19:15:59.231297+0000","last_change":"2023-07-14T19:15:58.250564+0000","last_active":"2023-07-14T19:15:59.231297+0000","last_peered":"2023-07-14T19:15:59.231297+0000","last_clean":"2023-07-14T19:15:59.231297+0000","last_became_active":"2023-07-14T19:15:58.250277+0000","last_became_peered":"2023-07-14T19:15:58.250277+0000","last_unstale":"2023-07-14T19:15:59.231297+0000","last_undegraded":"2023-07-14T19:15:59.231297+0000","last_fullsized":"2023-07-14T19:15:59.231297+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_clean_scrub_stamp":"2023-07-14T19:15:57.219223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:37:56.001097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"81'5","reported_seq":19,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-07-14T19:16:01.574166+0000","last_change":"2023-07-14T19:15:57.245586+0000","last_active":"2023-07-14T19:16:01.574166+0000","last_peered":"2023-07-14T19:16:01.574166+0000","last_clean":"2023-07-14T19:16:01.574166+0000","last_became_active":"2023-07-14T19:15:57.244767+0000","last_became_peered":"2023-07-14T19:15:57.244767+0000","last_unstale":"2023-07-14T19:16:01.574166+0000","last_undegraded":"2023-07-14T19:16:01.574166+0000","last_fullsized":"2023-07-14T19:16:01.574166+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:15:56.210166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:15:56.210166+0000","last_clean_scrub_stamp":"2023-07-14T19:15:56.210166+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T06:43:35.975084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":574,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-07-14T19:15:59.232307+0000","last_change":"2023-07-14T19:15:58.240509+0000","last_active":"2023-07-14T19:15:59.232307+0000","last_peered":"2023-07-14T19:15:59.232307+0000","last_clean":"2023-07-14T19:15:59.232307+0000","last_became_active":"2023-07-14T19:15:58.240198+0000","last_became_peered":"2023-07-14T19:15:58.240198+0000","last_unstale":"2023-07-14T19:15:59.232307+0000","last_undegraded":"2023-07-14T19:15:59.232307+0000","last_fullsized":"2023-07-14T19:15:59.232307+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_clean_scrub_stamp":"2023-07-14T19:15:57.219223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T19:29:25.962372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"80'1","reported_seq":15,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-07-14T19:15:59.721966+0000","last_change":"2023-07-14T19:15:57.239582+0000","last_active":"2023-07-14T19:15:59.721966+0000","last_peered":"2023-07-14T19:15:59.721966+0000","last_clean":"2023-07-14T19:15:59.721966+0000","last_became_active":"2023-07-14T19:15:57.239244+0000","last_became_peered":"2023-07-14T19:15:57.239244+0000","last_unstale":"2023-07-14T19:15:59.721966+0000","last_undegraded":"2023-07-14T19:15:59.721966+0000","last_fullsized":"2023-07-14T19:15:59.721966+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:15:56.210166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:15:56.210166+0000","last_clean_scrub_stamp":"2023-07-14T19:15:56.210166+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T03:16:58.719650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-07-14T19:15:59.231791+0000","last_change":"2023-07-14T19:15:58.249443+0000","last_active":"2023-07-14T19:15:59.231791+0000","last_peered":"2023-07-14T19:15:59.231791+0000","last_clean":"2023-07-14T19:15:59.231791+0000","last_became_active":"2023-07-14T19:15:58.249131+0000","last_became_peered":"2023-07-14T19:15:58.249131+0000","last_unstale":"2023-07-14T19:15:59.231791+0000","last_undegraded":"2023-07-14T19:15:59.231791+0000","last_fullsized":"2023-07-14T19:15:59.231791+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_clean_scrub_stamp":"2023-07-14T19:15:57.219223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T03:45:01.049663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":14,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-07-14T19:15:59.722732+0000","last_change":"2023-07-14T19:15:57.243725+0000","last_active":"2023-07-14T19:15:59.722732+0000","last_peered":"2023-07-14T19:15:59.722732+0000","last_clean":"2023-07-14T19:15:59.722732+0000","last_became_active":"2023-07-14T19:15:57.243431+0000","last_became_peered":"2023-07-14T19:15:57.243431+0000","last_unstale":"2023-07-14T19:15:59.722732+0000","last_undegraded":"2023-07-14T19:15:59.722732+0000","last_fullsized":"2023-07-14T19:15:59.722732+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:15:56.210166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:15:56.210166+0000","last_clean_scrub_stamp":"2023-07-14T19:15:56.210166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:41:49.174581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-07-14T19:15:59.721836+0000","last_change":"2023-07-14T19:15:58.239460+0000","last_active":"2023-07-14T19:15:59.721836+0000","last_peered":"2023-07-14T19:15:59.721836+0000","last_clean":"2023-07-14T19:15:59.721836+0000","last_became_active":"2023-07-14T19:15:58.239086+0000","last_became_peered":"2023-07-14T19:15:58.239086+0000","last_unstale":"2023-07-14T19:15:59.721836+0000","last_undegraded":"2023-07-14T19:15:59.721836+0000","last_fullsized":"2023-07-14T19:15:59.721836+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_clean_scrub_stamp":"2023-07-14T19:15:57.219223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T23:25:07.452583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"81'1","reported_seq":15,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-07-14T19:15:59.231044+0000","last_change":"2023-07-14T19:15:57.240173+0000","last_active":"2023-07-14T19:15:59.231044+0000","last_peered":"2023-07-14T19:15:59.231044+0000","last_clean":"2023-07-14T19:15:59.231044+0000","last_became_active":"2023-07-14T19:15:57.239822+0000","last_became_peered":"2023-07-14T19:15:57.239822+0000","last_unstale":"2023-07-14T19:15:59.231044+0000","last_undegraded":"2023-07-14T19:15:59.231044+0000","last_fullsized":"2023-07-14T19:15:59.231044+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:15:56.210166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:15:56.210166+0000","last_clean_scrub_stamp":"2023-07-14T19:15:56.210166+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T20:22:32.066346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-07-14T19:15:59.722607+0000","last_change":"2023-07-14T19:15:58.241533+0000","last_active":"2023-07-14T19:15:59.722607+0000","last_peered":"2023-07-14T19:15:59.722607+0000","last_clean":"2023-07-14T19:15:59.722607+0000","last_became_active":"2023-07-14T19:15:58.241216+0000","last_became_peered":"2023-07-14T19:15:58.241216+0000","last_unstale":"2023-07-14T19:15:59.722607+0000","last_undegraded":"2023-07-14T19:15:59.722607+0000","last_fullsized":"2023-07-14T19:15:59.722607+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_clean_scrub_stamp":"2023-07-14T19:15:57.219223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:40:27.802949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":14,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-07-14T19:15:59.722052+0000","last_change":"2023-07-14T19:15:57.236808+0000","last_active":"2023-07-14T19:15:59.722052+0000","last_peered":"2023-07-14T19:15:59.722052+0000","last_clean":"2023-07-14T19:15:59.722052+0000","last_became_active":"2023-07-14T19:15:57.236480+0000","last_became_peered":"2023-07-14T19:15:57.236480+0000","last_unstale":"2023-07-14T19:15:59.722052+0000","last_undegraded":"2023-07-14T19:15:59.722052+0000","last_fullsized":"2023-07-14T19:15:59.722052+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:15:56.210166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:15:56.210166+0000","last_clean_scrub_stamp":"2023-07-14T19:15:56.210166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T20:50:18.427213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-07-14T19:15:59.721741+0000","last_change":"2023-07-14T19:15:58.243236+0000","last_active":"2023-07-14T19:15:59.721741+0000","last_peered":"2023-07-14T19:15:59.721741+0000","last_clean":"2023-07-14T19:15:59.721741+0000","last_became_active":"2023-07-14T19:15:58.242951+0000","last_became_peered":"2023-07-14T19:15:58.242951+0000","last_unstale":"2023-07-14T19:15:59.721741+0000","last_undegraded":"2023-07-14T19:15:59.721741+0000","last_fullsized":"2023-07-14T19:15:59.721741+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_clean_scrub_stamp":"2023-07-14T19:15:57.219223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T05:20:59.047342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"81'2","reported_seq":16,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-07-14T19:16:01.572601+0000","last_change":"2023-07-14T19:15:57.242251+0000","last_active":"2023-07-14T19:16:01.572601+0000","last_peered":"2023-07-14T19:16:01.572601+0000","last_clean":"2023-07-14T19:16:01.572601+0000","last_became_active":"2023-07-14T19:15:57.241907+0000","last_became_peered":"2023-07-14T19:15:57.241907+0000","last_unstale":"2023-07-14T19:16:01.572601+0000","last_undegraded":"2023-07-14T19:16:01.572601+0000","last_fullsized":"2023-07-14T19:16:01.572601+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:15:56.210166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:15:56.210166+0000","last_clean_scrub_stamp":"2023-07-14T19:15:56.210166+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:52:53.425429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-07-14T19:15:59.573106+0000","last_change":"2023-07-14T19:15:58.248412+0000","last_active":"2023-07-14T19:15:59.573106+0000","last_peered":"2023-07-14T19:15:59.573106+0000","last_clean":"2023-07-14T19:15:59.573106+0000","last_became_active":"2023-07-14T19:15:58.247619+0000","last_became_peered":"2023-07-14T19:15:58.247619+0000","last_unstale":"2023-07-14T19:15:59.573106+0000","last_undegraded":"2023-07-14T19:15:59.573106+0000","last_fullsized":"2023-07-14T19:15:59.573106+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_clean_scrub_stamp":"2023-07-14T19:15:57.219223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T00:15:52.855579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"81'4","reported_seq":18,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-07-14T19:15:59.231404+0000","last_change":"2023-07-14T19:15:57.240436+0000","last_active":"2023-07-14T19:15:59.231404+0000","last_peered":"2023-07-14T19:15:59.231404+0000","last_clean":"2023-07-14T19:15:59.231404+0000","last_became_active":"2023-07-14T19:15:57.240148+0000","last_became_peered":"2023-07-14T19:15:57.240148+0000","last_unstale":"2023-07-14T19:15:59.231404+0000","last_undegraded":"2023-07-14T19:15:59.231404+0000","last_fullsized":"2023-07-14T19:15:59.231404+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:15:56.210166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:15:56.210166+0000","last_clean_scrub_stamp":"2023-07-14T19:15:56.210166+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T03:15:40.160793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-07-14T19:15:59.722062+0000","last_change":"2023-07-14T19:15:58.246127+0000","last_active":"2023-07-14T19:15:59.722062+0000","last_peered":"2023-07-14T19:15:59.722062+0000","last_clean":"2023-07-14T19:15:59.722062+0000","last_became_active":"2023-07-14T19:15:58.245778+0000","last_became_peered":"2023-07-14T19:15:58.245778+0000","last_unstale":"2023-07-14T19:15:59.722062+0000","last_undegraded":"2023-07-14T19:15:59.722062+0000","last_fullsized":"2023-07-14T19:15:59.722062+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_clean_scrub_stamp":"2023-07-14T19:15:57.219223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T00:21:57.683936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"80'6","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-07-14T19:15:59.722153+0000","last_change":"2023-07-14T19:15:57.243660+0000","last_active":"2023-07-14T19:15:59.722153+0000","last_peered":"2023-07-14T19:15:59.722153+0000","last_clean":"2023-07-14T19:15:59.722153+0000","last_became_active":"2023-07-14T19:15:57.243338+0000","last_became_peered":"2023-07-14T19:15:57.243338+0000","last_unstale":"2023-07-14T19:15:59.722153+0000","last_undegraded":"2023-07-14T19:15:59.722153+0000","last_fullsized":"2023-07-14T19:15:59.722153+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:15:56.210166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:15:56.210166+0000","last_clean_scrub_stamp":"2023-07-14T19:15:56.210166+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T06:12:23.930218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1564,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-07-14T19:15:59.232525+0000","last_change":"2023-07-14T19:15:58.244162+0000","last_active":"2023-07-14T19:15:59.232525+0000","last_peered":"2023-07-14T19:15:59.232525+0000","last_clean":"2023-07-14T19:15:59.232525+0000","last_became_active":"2023-07-14T19:15:58.243912+0000","last_became_peered":"2023-07-14T19:15:58.243912+0000","last_unstale":"2023-07-14T19:15:59.232525+0000","last_undegraded":"2023-07-14T19:15:59.232525+0000","last_fullsized":"2023-07-14T19:15:59.232525+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_clean_scrub_stamp":"2023-07-14T19:15:57.219223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T23:05:24.992671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"81'6","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-07-14T19:16:02.596564+0000","last_change":"2023-07-14T19:15:57.246975+0000","last_active":"2023-07-14T19:16:02.596564+0000","last_peered":"2023-07-14T19:16:02.596564+0000","last_clean":"2023-07-14T19:16:02.596564+0000","last_became_active":"2023-07-14T19:15:57.246481+0000","last_became_peered":"2023-07-14T19:15:57.246481+0000","last_unstale":"2023-07-14T19:16:02.596564+0000","last_undegraded":"2023-07-14T19:16:02.596564+0000","last_fullsized":"2023-07-14T19:16:02.596564+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:15:56.210166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:15:56.210166+0000","last_clean_scrub_stamp":"2023-07-14T19:15:56.210166+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T06:50:24.165830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-07-14T19:15:59.721632+0000","last_change":"2023-07-14T19:15:58.241870+0000","last_active":"2023-07-14T19:15:59.721632+0000","last_peered":"2023-07-14T19:15:59.721632+0000","last_clean":"2023-07-14T19:15:59.721632+0000","last_became_active":"2023-07-14T19:15:58.241503+0000","last_became_peered":"2023-07-14T19:15:58.241503+0000","last_unstale":"2023-07-14T19:15:59.721632+0000","last_undegraded":"2023-07-14T19:15:59.721632+0000","last_fullsized":"2023-07-14T19:15:59.721632+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_clean_scrub_stamp":"2023-07-14T19:15:57.219223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T23:05:14.779450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"81'3","reported_seq":17,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-07-14T19:16:01.571518+0000","last_change":"2023-07-14T19:15:57.236715+0000","last_active":"2023-07-14T19:16:01.571518+0000","last_peered":"2023-07-14T19:16:01.571518+0000","last_clean":"2023-07-14T19:16:01.571518+0000","last_became_active":"2023-07-14T19:15:57.236391+0000","last_became_peered":"2023-07-14T19:15:57.236391+0000","last_unstale":"2023-07-14T19:16:01.571518+0000","last_undegraded":"2023-07-14T19:16:01.571518+0000","last_fullsized":"2023-07-14T19:16:01.571518+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:15:56.210166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:15:56.210166+0000","last_clean_scrub_stamp":"2023-07-14T19:15:56.210166+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:06:23.938675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-07-14T19:15:59.721870+0000","last_change":"2023-07-14T19:15:58.245110+0000","last_active":"2023-07-14T19:15:59.721870+0000","last_peered":"2023-07-14T19:15:59.721870+0000","last_clean":"2023-07-14T19:15:59.721870+0000","last_became_active":"2023-07-14T19:15:58.244779+0000","last_became_peered":"2023-07-14T19:15:58.244779+0000","last_unstale":"2023-07-14T19:15:59.721870+0000","last_undegraded":"2023-07-14T19:15:59.721870+0000","last_fullsized":"2023-07-14T19:15:59.721870+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_clean_scrub_stamp":"2023-07-14T19:15:57.219223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T00:24:23.826897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":14,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-07-14T19:15:59.229648+0000","last_change":"2023-07-14T19:15:57.236795+0000","last_active":"2023-07-14T19:15:59.229648+0000","last_peered":"2023-07-14T19:15:59.229648+0000","last_clean":"2023-07-14T19:15:59.229648+0000","last_became_active":"2023-07-14T19:15:57.236442+0000","last_became_peered":"2023-07-14T19:15:57.236442+0000","last_unstale":"2023-07-14T19:15:59.229648+0000","last_undegraded":"2023-07-14T19:15:59.229648+0000","last_fullsized":"2023-07-14T19:15:59.229648+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:15:56.210166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:15:56.210166+0000","last_clean_scrub_stamp":"2023-07-14T19:15:56.210166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:05:41.764871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-07-14T19:15:59.721448+0000","last_change":"2023-07-14T19:15:58.247790+0000","last_active":"2023-07-14T19:15:59.721448+0000","last_peered":"2023-07-14T19:15:59.721448+0000","last_clean":"2023-07-14T19:15:59.721448+0000","last_became_active":"2023-07-14T19:15:58.247461+0000","last_became_peered":"2023-07-14T19:15:58.247461+0000","last_unstale":"2023-07-14T19:15:59.721448+0000","last_undegraded":"2023-07-14T19:15:59.721448+0000","last_fullsized":"2023-07-14T19:15:59.721448+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_clean_scrub_stamp":"2023-07-14T19:15:57.219223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:29:24.306834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"81'16","reported_seq":30,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-07-14T19:16:02.592714+0000","last_change":"2023-07-14T19:15:57.244534+0000","last_active":"2023-07-14T19:16:02.592714+0000","last_peered":"2023-07-14T19:16:02.592714+0000","last_clean":"2023-07-14T19:16:02.592714+0000","last_became_active":"2023-07-14T19:15:57.243107+0000","last_became_peered":"2023-07-14T19:15:57.243107+0000","last_unstale":"2023-07-14T19:16:02.592714+0000","last_undegraded":"2023-07-14T19:16:02.592714+0000","last_fullsized":"2023-07-14T19:16:02.592714+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:15:56.210166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:15:56.210166+0000","last_clean_scrub_stamp":"2023-07-14T19:15:56.210166+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T21:48:40.199873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-07-14T19:15:59.231092+0000","last_change":"2023-07-14T19:15:58.250511+0000","last_active":"2023-07-14T19:15:59.231092+0000","last_peered":"2023-07-14T19:15:59.231092+0000","last_clean":"2023-07-14T19:15:59.231092+0000","last_became_active":"2023-07-14T19:15:58.250159+0000","last_became_peered":"2023-07-14T19:15:58.250159+0000","last_unstale":"2023-07-14T19:15:59.231092+0000","last_undegraded":"2023-07-14T19:15:59.231092+0000","last_fullsized":"2023-07-14T19:15:59.231092+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_clean_scrub_stamp":"2023-07-14T19:15:57.219223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T21:32:01.440745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"80'1","reported_seq":15,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-07-14T19:15:59.232676+0000","last_change":"2023-07-14T19:15:57.238719+0000","last_active":"2023-07-14T19:15:59.232676+0000","last_peered":"2023-07-14T19:15:59.232676+0000","last_clean":"2023-07-14T19:15:59.232676+0000","last_became_active":"2023-07-14T19:15:57.238404+0000","last_became_peered":"2023-07-14T19:15:57.238404+0000","last_unstale":"2023-07-14T19:15:59.232676+0000","last_undegraded":"2023-07-14T19:15:59.232676+0000","last_fullsized":"2023-07-14T19:15:59.232676+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:15:56.210166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:15:56.210166+0000","last_clean_scrub_stamp":"2023-07-14T19:15:56.210166+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T03:14:38.948894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-07-14T19:15:59.938888+0000","last_change":"2023-07-14T19:15:58.245052+0000","last_active":"2023-07-14T19:15:59.938888+0000","last_peered":"2023-07-14T19:15:59.938888+0000","last_clean":"2023-07-14T19:15:59.938888+0000","last_became_active":"2023-07-14T19:15:58.244733+0000","last_became_peered":"2023-07-14T19:15:58.244733+0000","last_unstale":"2023-07-14T19:15:59.938888+0000","last_undegraded":"2023-07-14T19:15:59.938888+0000","last_fullsized":"2023-07-14T19:15:59.938888+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_clean_scrub_stamp":"2023-07-14T19:15:57.219223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T03:43:49.689439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"81'1","reported_seq":15,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-07-14T19:15:59.722595+0000","last_change":"2023-07-14T19:15:57.245402+0000","last_active":"2023-07-14T19:15:59.722595+0000","last_peered":"2023-07-14T19:15:59.722595+0000","last_clean":"2023-07-14T19:15:59.722595+0000","last_became_active":"2023-07-14T19:15:57.245092+0000","last_became_peered":"2023-07-14T19:15:57.245092+0000","last_unstale":"2023-07-14T19:15:59.722595+0000","last_undegraded":"2023-07-14T19:15:59.722595+0000","last_fullsized":"2023-07-14T19:15:59.722595+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:15:56.210166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:15:56.210166+0000","last_clean_scrub_stamp":"2023-07-14T19:15:56.210166+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:17:28.260009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-07-14T19:15:59.722257+0000","last_change":"2023-07-14T19:15:58.247611+0000","last_active":"2023-07-14T19:15:59.722257+0000","last_peered":"2023-07-14T19:15:59.722257+0000","last_clean":"2023-07-14T19:15:59.722257+0000","last_became_active":"2023-07-14T19:15:58.247313+0000","last_became_peered":"2023-07-14T19:15:58.247313+0000","last_unstale":"2023-07-14T19:15:59.722257+0000","last_undegraded":"2023-07-14T19:15:59.722257+0000","last_fullsized":"2023-07-14T19:15:59.722257+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_clean_scrub_stamp":"2023-07-14T19:15:57.219223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T06:40:19.317380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":14,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-07-14T19:15:59.722196+0000","last_change":"2023-07-14T19:15:57.242149+0000","last_active":"2023-07-14T19:15:59.722196+0000","last_peered":"2023-07-14T19:15:59.722196+0000","last_clean":"2023-07-14T19:15:59.722196+0000","last_became_active":"2023-07-14T19:15:57.241847+0000","last_became_peered":"2023-07-14T19:15:57.241847+0000","last_unstale":"2023-07-14T19:15:59.722196+0000","last_undegraded":"2023-07-14T19:15:59.722196+0000","last_fullsized":"2023-07-14T19:15:59.722196+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:15:56.210166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:15:56.210166+0000","last_clean_scrub_stamp":"2023-07-14T19:15:56.210166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:14:47.640280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-07-14T19:15:59.722352+0000","last_change":"2023-07-14T19:15:58.244573+0000","last_active":"2023-07-14T19:15:59.722352+0000","last_peered":"2023-07-14T19:15:59.722352+0000","last_clean":"2023-07-14T19:15:59.722352+0000","last_became_active":"2023-07-14T19:15:58.244165+0000","last_became_peered":"2023-07-14T19:15:58.244165+0000","last_unstale":"2023-07-14T19:15:59.722352+0000","last_undegraded":"2023-07-14T19:15:59.722352+0000","last_fullsized":"2023-07-14T19:15:59.722352+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_clean_scrub_stamp":"2023-07-14T19:15:57.219223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T05:36:19.851400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-07-14T19:15:59.231138+0000","last_change":"2023-07-14T19:15:58.246282+0000","last_active":"2023-07-14T19:15:59.231138+0000","last_peered":"2023-07-14T19:15:59.231138+0000","last_clean":"2023-07-14T19:15:59.231138+0000","last_became_active":"2023-07-14T19:15:58.245984+0000","last_became_peered":"2023-07-14T19:15:58.245984+0000","last_unstale":"2023-07-14T19:15:59.231138+0000","last_undegraded":"2023-07-14T19:15:59.231138+0000","last_fullsized":"2023-07-14T19:15:59.231138+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_clean_scrub_stamp":"2023-07-14T19:15:57.219223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T00:42:41.109802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":14,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-07-14T19:15:59.722605+0000","last_change":"2023-07-14T19:15:57.244731+0000","last_active":"2023-07-14T19:15:59.722605+0000","last_peered":"2023-07-14T19:15:59.722605+0000","last_clean":"2023-07-14T19:15:59.722605+0000","last_became_active":"2023-07-14T19:15:57.244384+0000","last_became_peered":"2023-07-14T19:15:57.244384+0000","last_unstale":"2023-07-14T19:15:59.722605+0000","last_undegraded":"2023-07-14T19:15:59.722605+0000","last_fullsized":"2023-07-14T19:15:59.722605+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:15:56.210166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:15:56.210166+0000","last_clean_scrub_stamp":"2023-07-14T19:15:56.210166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T22:20:50.336578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-07-14T19:15:59.722302+0000","last_change":"2023-07-14T19:15:58.241457+0000","last_active":"2023-07-14T19:15:59.722302+0000","last_peered":"2023-07-14T19:15:59.722302+0000","last_clean":"2023-07-14T19:15:59.722302+0000","last_became_active":"2023-07-14T19:15:58.241113+0000","last_became_peered":"2023-07-14T19:15:58.241113+0000","last_unstale":"2023-07-14T19:15:59.722302+0000","last_undegraded":"2023-07-14T19:15:59.722302+0000","last_fullsized":"2023-07-14T19:15:59.722302+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_clean_scrub_stamp":"2023-07-14T19:15:57.219223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T21:28:59.714151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"81'2","reported_seq":16,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-07-14T19:16:01.572074+0000","last_change":"2023-07-14T19:15:57.235057+0000","last_active":"2023-07-14T19:16:01.572074+0000","last_peered":"2023-07-14T19:16:01.572074+0000","last_clean":"2023-07-14T19:16:01.572074+0000","last_became_active":"2023-07-14T19:15:57.234751+0000","last_became_peered":"2023-07-14T19:15:57.234751+0000","last_unstale":"2023-07-14T19:16:01.572074+0000","last_undegraded":"2023-07-14T19:16:01.572074+0000","last_fullsized":"2023-07-14T19:16:01.572074+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:15:56.210166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:15:56.210166+0000","last_clean_scrub_stamp":"2023-07-14T19:15:56.210166+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:12:00.277001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":336,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-07-14T19:15:59.721531+0000","last_change":"2023-07-14T19:14:58.969281+0000","last_active":"2023-07-14T19:15:59.721531+0000","last_peered":"2023-07-14T19:15:59.721531+0000","last_clean":"2023-07-14T19:15:59.721531+0000","last_became_active":"2023-07-14T19:14:58.968767+0000","last_became_peered":"2023-07-14T19:14:58.968767+0000","last_unstale":"2023-07-14T19:15:59.721531+0000","last_undegraded":"2023-07-14T19:15:59.721531+0000","last_fullsized":"2023-07-14T19:15:59.721531+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:12:03.589019+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:12:03.589019+0000","last_clean_scrub_stamp":"2023-07-14T19:12:03.589019+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T05:39:04.087637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-07-14T19:15:59.722476+0000","last_change":"2023-07-14T19:15:58.250737+0000","last_active":"2023-07-14T19:15:59.722476+0000","last_peered":"2023-07-14T19:15:59.722476+0000","last_clean":"2023-07-14T19:15:59.722476+0000","last_became_active":"2023-07-14T19:15:58.250379+0000","last_became_peered":"2023-07-14T19:15:58.250379+0000","last_unstale":"2023-07-14T19:15:59.722476+0000","last_undegraded":"2023-07-14T19:15:59.722476+0000","last_fullsized":"2023-07-14T19:15:59.722476+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_clean_scrub_stamp":"2023-07-14T19:15:57.219223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T19:57:17.147937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"81'1","reported_seq":15,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-07-14T19:15:59.721738+0000","last_change":"2023-07-14T19:15:57.236799+0000","last_active":"2023-07-14T19:15:59.721738+0000","last_peered":"2023-07-14T19:15:59.721738+0000","last_clean":"2023-07-14T19:15:59.721738+0000","last_became_active":"2023-07-14T19:15:57.236503+0000","last_became_peered":"2023-07-14T19:15:57.236503+0000","last_unstale":"2023-07-14T19:15:59.721738+0000","last_undegraded":"2023-07-14T19:15:59.721738+0000","last_fullsized":"2023-07-14T19:15:59.721738+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:15:56.210166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:15:56.210166+0000","last_clean_scrub_stamp":"2023-07-14T19:15:56.210166+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T00:26:24.967329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"81'3","reported_seq":17,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-07-14T19:16:03.614535+0000","last_change":"2023-07-14T19:15:57.245788+0000","last_active":"2023-07-14T19:16:03.614535+0000","last_peered":"2023-07-14T19:16:03.614535+0000","last_clean":"2023-07-14T19:16:03.614535+0000","last_became_active":"2023-07-14T19:15:57.244749+0000","last_became_peered":"2023-07-14T19:15:57.244749+0000","last_unstale":"2023-07-14T19:16:03.614535+0000","last_undegraded":"2023-07-14T19:16:03.614535+0000","last_fullsized":"2023-07-14T19:16:03.614535+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:15:56.210166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:15:56.210166+0000","last_clean_scrub_stamp":"2023-07-14T19:15:56.210166+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T00:56:58.078952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-07-14T19:15:59.722962+0000","last_change":"2023-07-14T19:15:58.250973+0000","last_active":"2023-07-14T19:15:59.722962+0000","last_peered":"2023-07-14T19:15:59.722962+0000","last_clean":"2023-07-14T19:15:59.722962+0000","last_became_active":"2023-07-14T19:15:58.250634+0000","last_became_peered":"2023-07-14T19:15:58.250634+0000","last_unstale":"2023-07-14T19:15:59.722962+0000","last_undegraded":"2023-07-14T19:15:59.722962+0000","last_fullsized":"2023-07-14T19:15:59.722962+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_clean_scrub_stamp":"2023-07-14T19:15:57.219223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T06:14:32.307565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"81'7","reported_seq":21,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-07-14T19:16:03.613824+0000","last_change":"2023-07-14T19:15:57.234996+0000","last_active":"2023-07-14T19:16:03.613824+0000","last_peered":"2023-07-14T19:16:03.613824+0000","last_clean":"2023-07-14T19:16:03.613824+0000","last_became_active":"2023-07-14T19:15:57.234697+0000","last_became_peered":"2023-07-14T19:15:57.234697+0000","last_unstale":"2023-07-14T19:16:03.613824+0000","last_undegraded":"2023-07-14T19:16:03.613824+0000","last_fullsized":"2023-07-14T19:16:03.613824+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:15:56.210166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:15:56.210166+0000","last_clean_scrub_stamp":"2023-07-14T19:15:56.210166+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T21:46:18.232628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-07-14T19:15:59.722957+0000","last_change":"2023-07-14T19:15:58.241753+0000","last_active":"2023-07-14T19:15:59.722957+0000","last_peered":"2023-07-14T19:15:59.722957+0000","last_clean":"2023-07-14T19:15:59.722957+0000","last_became_active":"2023-07-14T19:15:58.241485+0000","last_became_peered":"2023-07-14T19:15:58.241485+0000","last_unstale":"2023-07-14T19:15:59.722957+0000","last_undegraded":"2023-07-14T19:15:59.722957+0000","last_fullsized":"2023-07-14T19:15:59.722957+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_clean_scrub_stamp":"2023-07-14T19:15:57.219223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T03:02:48.399750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"81'4","reported_seq":18,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-07-14T19:16:01.572337+0000","last_change":"2023-07-14T19:15:57.240019+0000","last_active":"2023-07-14T19:16:01.572337+0000","last_peered":"2023-07-14T19:16:01.572337+0000","last_clean":"2023-07-14T19:16:01.572337+0000","last_became_active":"2023-07-14T19:15:57.239521+0000","last_became_peered":"2023-07-14T19:15:57.239521+0000","last_unstale":"2023-07-14T19:16:01.572337+0000","last_undegraded":"2023-07-14T19:16:01.572337+0000","last_fullsized":"2023-07-14T19:16:01.572337+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:15:56.210166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:15:56.210166+0000","last_clean_scrub_stamp":"2023-07-14T19:15:56.210166+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T23:32:40.283168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-07-14T19:15:59.573249+0000","last_change":"2023-07-14T19:15:58.248187+0000","last_active":"2023-07-14T19:15:59.573249+0000","last_peered":"2023-07-14T19:15:59.573249+0000","last_clean":"2023-07-14T19:15:59.573249+0000","last_became_active":"2023-07-14T19:15:58.247854+0000","last_became_peered":"2023-07-14T19:15:58.247854+0000","last_unstale":"2023-07-14T19:15:59.573249+0000","last_undegraded":"2023-07-14T19:15:59.573249+0000","last_fullsized":"2023-07-14T19:15:59.573249+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_clean_scrub_stamp":"2023-07-14T19:15:57.219223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T21:01:16.786236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"81'2","reported_seq":16,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-07-14T19:16:01.572678+0000","last_change":"2023-07-14T19:15:57.245012+0000","last_active":"2023-07-14T19:16:01.572678+0000","last_peered":"2023-07-14T19:16:01.572678+0000","last_clean":"2023-07-14T19:16:01.572678+0000","last_became_active":"2023-07-14T19:15:57.244776+0000","last_became_peered":"2023-07-14T19:15:57.244776+0000","last_unstale":"2023-07-14T19:16:01.572678+0000","last_undegraded":"2023-07-14T19:16:01.572678+0000","last_fullsized":"2023-07-14T19:16:01.572678+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:15:56.210166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:15:56.210166+0000","last_clean_scrub_stamp":"2023-07-14T19:15:56.210166+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T20:16:41.721374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-07-14T19:15:59.232170+0000","last_change":"2023-07-14T19:15:58.247704+0000","last_active":"2023-07-14T19:15:59.232170+0000","last_peered":"2023-07-14T19:15:59.232170+0000","last_clean":"2023-07-14T19:15:59.232170+0000","last_became_active":"2023-07-14T19:15:58.247364+0000","last_became_peered":"2023-07-14T19:15:58.247364+0000","last_unstale":"2023-07-14T19:15:59.232170+0000","last_undegraded":"2023-07-14T19:15:59.232170+0000","last_fullsized":"2023-07-14T19:15:59.232170+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_clean_scrub_stamp":"2023-07-14T19:15:57.219223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T19:44:25.096100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":14,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-07-14T19:15:59.722984+0000","last_change":"2023-07-14T19:15:57.234991+0000","last_active":"2023-07-14T19:15:59.722984+0000","last_peered":"2023-07-14T19:15:59.722984+0000","last_clean":"2023-07-14T19:15:59.722984+0000","last_became_active":"2023-07-14T19:15:57.234680+0000","last_became_peered":"2023-07-14T19:15:57.234680+0000","last_unstale":"2023-07-14T19:15:59.722984+0000","last_undegraded":"2023-07-14T19:15:59.722984+0000","last_fullsized":"2023-07-14T19:15:59.722984+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:15:56.210166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:15:56.210166+0000","last_clean_scrub_stamp":"2023-07-14T19:15:56.210166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T03:09:07.479398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-07-14T19:15:59.938707+0000","last_change":"2023-07-14T19:15:58.247224+0000","last_active":"2023-07-14T19:15:59.938707+0000","last_peered":"2023-07-14T19:15:59.938707+0000","last_clean":"2023-07-14T19:15:59.938707+0000","last_became_active":"2023-07-14T19:15:58.246889+0000","last_became_peered":"2023-07-14T19:15:58.246889+0000","last_unstale":"2023-07-14T19:15:59.938707+0000","last_undegraded":"2023-07-14T19:15:59.938707+0000","last_fullsized":"2023-07-14T19:15:59.938707+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_clean_scrub_stamp":"2023-07-14T19:15:57.219223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:25:39.860559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":14,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-07-14T19:15:59.720943+0000","last_change":"2023-07-14T19:15:57.246491+0000","last_active":"2023-07-14T19:15:59.720943+0000","last_peered":"2023-07-14T19:15:59.720943+0000","last_clean":"2023-07-14T19:15:59.720943+0000","last_became_active":"2023-07-14T19:15:57.246178+0000","last_became_peered":"2023-07-14T19:15:57.246178+0000","last_unstale":"2023-07-14T19:15:59.720943+0000","last_undegraded":"2023-07-14T19:15:59.720943+0000","last_fullsized":"2023-07-14T19:15:59.720943+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:15:56.210166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:15:56.210166+0000","last_clean_scrub_stamp":"2023-07-14T19:15:56.210166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T03:15:13.275825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-07-14T19:15:59.723335+0000","last_change":"2023-07-14T19:15:58.247005+0000","last_active":"2023-07-14T19:15:59.723335+0000","last_peered":"2023-07-14T19:15:59.723335+0000","last_clean":"2023-07-14T19:15:59.723335+0000","last_became_active":"2023-07-14T19:15:58.246156+0000","last_became_peered":"2023-07-14T19:15:58.246156+0000","last_unstale":"2023-07-14T19:15:59.723335+0000","last_undegraded":"2023-07-14T19:15:59.723335+0000","last_fullsized":"2023-07-14T19:15:59.723335+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_clean_scrub_stamp":"2023-07-14T19:15:57.219223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T20:11:25.410164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"81'4","reported_seq":18,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-07-14T19:16:01.572528+0000","last_change":"2023-07-14T19:15:57.245551+0000","last_active":"2023-07-14T19:16:01.572528+0000","last_peered":"2023-07-14T19:16:01.572528+0000","last_clean":"2023-07-14T19:16:01.572528+0000","last_became_active":"2023-07-14T19:15:57.245179+0000","last_became_peered":"2023-07-14T19:15:57.245179+0000","last_unstale":"2023-07-14T19:16:01.572528+0000","last_undegraded":"2023-07-14T19:16:01.572528+0000","last_fullsized":"2023-07-14T19:16:01.572528+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:15:56.210166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:15:56.210166+0000","last_clean_scrub_stamp":"2023-07-14T19:15:56.210166+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T00:35:19.482494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-07-14T19:15:59.722416+0000","last_change":"2023-07-14T19:15:58.250454+0000","last_active":"2023-07-14T19:15:59.722416+0000","last_peered":"2023-07-14T19:15:59.722416+0000","last_clean":"2023-07-14T19:15:59.722416+0000","last_became_active":"2023-07-14T19:15:58.250145+0000","last_became_peered":"2023-07-14T19:15:58.250145+0000","last_unstale":"2023-07-14T19:15:59.722416+0000","last_undegraded":"2023-07-14T19:15:59.722416+0000","last_fullsized":"2023-07-14T19:15:59.722416+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_clean_scrub_stamp":"2023-07-14T19:15:57.219223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T23:46:27.565214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"81'3","reported_seq":17,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-07-14T19:16:02.594567+0000","last_change":"2023-07-14T19:15:57.247545+0000","last_active":"2023-07-14T19:16:02.594567+0000","last_peered":"2023-07-14T19:16:02.594567+0000","last_clean":"2023-07-14T19:16:02.594567+0000","last_became_active":"2023-07-14T19:15:57.247201+0000","last_became_peered":"2023-07-14T19:15:57.247201+0000","last_unstale":"2023-07-14T19:16:02.594567+0000","last_undegraded":"2023-07-14T19:16:02.594567+0000","last_fullsized":"2023-07-14T19:16:02.594567+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:15:56.210166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:15:56.210166+0000","last_clean_scrub_stamp":"2023-07-14T19:15:56.210166+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T23:52:58.892773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-07-14T19:15:59.722674+0000","last_change":"2023-07-14T19:15:58.252144+0000","last_active":"2023-07-14T19:15:59.722674+0000","last_peered":"2023-07-14T19:15:59.722674+0000","last_clean":"2023-07-14T19:15:59.722674+0000","last_became_active":"2023-07-14T19:15:58.251844+0000","last_became_peered":"2023-07-14T19:15:58.251844+0000","last_unstale":"2023-07-14T19:15:59.722674+0000","last_undegraded":"2023-07-14T19:15:59.722674+0000","last_fullsized":"2023-07-14T19:15:59.722674+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_clean_scrub_stamp":"2023-07-14T19:15:57.219223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:19:09.690589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"80'1","reported_seq":15,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-07-14T19:15:59.231229+0000","last_change":"2023-07-14T19:15:57.239560+0000","last_active":"2023-07-14T19:15:59.231229+0000","last_peered":"2023-07-14T19:15:59.231229+0000","last_clean":"2023-07-14T19:15:59.231229+0000","last_became_active":"2023-07-14T19:15:57.239250+0000","last_became_peered":"2023-07-14T19:15:57.239250+0000","last_unstale":"2023-07-14T19:15:59.231229+0000","last_undegraded":"2023-07-14T19:15:59.231229+0000","last_fullsized":"2023-07-14T19:15:59.231229+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:15:56.210166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:15:56.210166+0000","last_clean_scrub_stamp":"2023-07-14T19:15:56.210166+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T21:04:10.055051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-07-14T19:15:59.722104+0000","last_change":"2023-07-14T19:15:58.242144+0000","last_active":"2023-07-14T19:15:59.722104+0000","last_peered":"2023-07-14T19:15:59.722104+0000","last_clean":"2023-07-14T19:15:59.722104+0000","last_became_active":"2023-07-14T19:15:58.241597+0000","last_became_peered":"2023-07-14T19:15:58.241597+0000","last_unstale":"2023-07-14T19:15:59.722104+0000","last_undegraded":"2023-07-14T19:15:59.722104+0000","last_fullsized":"2023-07-14T19:15:59.722104+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_clean_scrub_stamp":"2023-07-14T19:15:57.219223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T22:43:21.380478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"80'2","reported_seq":16,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-07-14T19:15:59.721650+0000","last_change":"2023-07-14T19:15:57.247059+0000","last_active":"2023-07-14T19:15:59.721650+0000","last_peered":"2023-07-14T19:15:59.721650+0000","last_clean":"2023-07-14T19:15:59.721650+0000","last_became_active":"2023-07-14T19:15:57.246736+0000","last_became_peered":"2023-07-14T19:15:57.246736+0000","last_unstale":"2023-07-14T19:15:59.721650+0000","last_undegraded":"2023-07-14T19:15:59.721650+0000","last_fullsized":"2023-07-14T19:15:59.721650+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:15:56.210166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:15:56.210166+0000","last_clean_scrub_stamp":"2023-07-14T19:15:56.210166+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T20:57:38.014630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-07-14T19:15:59.722005+0000","last_change":"2023-07-14T19:15:58.244180+0000","last_active":"2023-07-14T19:15:59.722005+0000","last_peered":"2023-07-14T19:15:59.722005+0000","last_clean":"2023-07-14T19:15:59.722005+0000","last_became_active":"2023-07-14T19:15:58.243885+0000","last_became_peered":"2023-07-14T19:15:58.243885+0000","last_unstale":"2023-07-14T19:15:59.722005+0000","last_undegraded":"2023-07-14T19:15:59.722005+0000","last_fullsized":"2023-07-14T19:15:59.722005+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_clean_scrub_stamp":"2023-07-14T19:15:57.219223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:56:30.077276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-07-14T19:15:59.938781+0000","last_change":"2023-07-14T19:15:58.246944+0000","last_active":"2023-07-14T19:15:59.938781+0000","last_peered":"2023-07-14T19:15:59.938781+0000","last_clean":"2023-07-14T19:15:59.938781+0000","last_became_active":"2023-07-14T19:15:58.246672+0000","last_became_peered":"2023-07-14T19:15:58.246672+0000","last_unstale":"2023-07-14T19:15:59.938781+0000","last_undegraded":"2023-07-14T19:15:59.938781+0000","last_fullsized":"2023-07-14T19:15:59.938781+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_clean_scrub_stamp":"2023-07-14T19:15:57.219223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:59:58.787272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-07-14T19:15:59.720933+0000","last_change":"2023-07-14T19:15:58.246761+0000","last_active":"2023-07-14T19:15:59.720933+0000","last_peered":"2023-07-14T19:15:59.720933+0000","last_clean":"2023-07-14T19:15:59.720933+0000","last_became_active":"2023-07-14T19:15:58.246318+0000","last_became_peered":"2023-07-14T19:15:58.246318+0000","last_unstale":"2023-07-14T19:15:59.720933+0000","last_undegraded":"2023-07-14T19:15:59.720933+0000","last_fullsized":"2023-07-14T19:15:59.720933+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_clean_scrub_stamp":"2023-07-14T19:15:57.219223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T22:22:00.801887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-07-14T19:15:59.721079+0000","last_change":"2023-07-14T19:15:58.251465+0000","last_active":"2023-07-14T19:15:59.721079+0000","last_peered":"2023-07-14T19:15:59.721079+0000","last_clean":"2023-07-14T19:15:59.721079+0000","last_became_active":"2023-07-14T19:15:58.251138+0000","last_became_peered":"2023-07-14T19:15:58.251138+0000","last_unstale":"2023-07-14T19:15:59.721079+0000","last_undegraded":"2023-07-14T19:15:59.721079+0000","last_fullsized":"2023-07-14T19:15:59.721079+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_clean_scrub_stamp":"2023-07-14T19:15:57.219223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:53:28.483753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-07-14T19:15:59.573084+0000","last_change":"2023-07-14T19:15:58.244335+0000","last_active":"2023-07-14T19:15:59.573084+0000","last_peered":"2023-07-14T19:15:59.573084+0000","last_clean":"2023-07-14T19:15:59.573084+0000","last_became_active":"2023-07-14T19:15:58.243955+0000","last_became_peered":"2023-07-14T19:15:58.243955+0000","last_unstale":"2023-07-14T19:15:59.573084+0000","last_undegraded":"2023-07-14T19:15:59.573084+0000","last_fullsized":"2023-07-14T19:15:59.573084+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_clean_scrub_stamp":"2023-07-14T19:15:57.219223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T06:54:43.644336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-07-14T19:15:59.723074+0000","last_change":"2023-07-14T19:15:58.244373+0000","last_active":"2023-07-14T19:15:59.723074+0000","last_peered":"2023-07-14T19:15:59.723074+0000","last_clean":"2023-07-14T19:15:59.723074+0000","last_became_active":"2023-07-14T19:15:58.244042+0000","last_became_peered":"2023-07-14T19:15:58.244042+0000","last_unstale":"2023-07-14T19:15:59.723074+0000","last_undegraded":"2023-07-14T19:15:59.723074+0000","last_fullsized":"2023-07-14T19:15:59.723074+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_clean_scrub_stamp":"2023-07-14T19:15:57.219223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T03:40:02.041528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-07-14T19:15:59.938908+0000","last_change":"2023-07-14T19:15:58.240770+0000","last_active":"2023-07-14T19:15:59.938908+0000","last_peered":"2023-07-14T19:15:59.938908+0000","last_clean":"2023-07-14T19:15:59.938908+0000","last_became_active":"2023-07-14T19:15:58.240408+0000","last_became_peered":"2023-07-14T19:15:58.240408+0000","last_unstale":"2023-07-14T19:15:59.938908+0000","last_undegraded":"2023-07-14T19:15:59.938908+0000","last_fullsized":"2023-07-14T19:15:59.938908+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_clean_scrub_stamp":"2023-07-14T19:15:57.219223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T19:45:10.482496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-07-14T19:15:59.721267+0000","last_change":"2023-07-14T19:15:58.247557+0000","last_active":"2023-07-14T19:15:59.721267+0000","last_peered":"2023-07-14T19:15:59.721267+0000","last_clean":"2023-07-14T19:15:59.721267+0000","last_became_active":"2023-07-14T19:15:58.247228+0000","last_became_peered":"2023-07-14T19:15:58.247228+0000","last_unstale":"2023-07-14T19:15:59.721267+0000","last_undegraded":"2023-07-14T19:15:59.721267+0000","last_fullsized":"2023-07-14T19:15:59.721267+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_clean_scrub_stamp":"2023-07-14T19:15:57.219223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T05:18:52.409847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-07-14T19:15:59.722818+0000","last_change":"2023-07-14T19:15:58.247013+0000","last_active":"2023-07-14T19:15:59.722818+0000","last_peered":"2023-07-14T19:15:59.722818+0000","last_clean":"2023-07-14T19:15:59.722818+0000","last_became_active":"2023-07-14T19:15:58.246729+0000","last_became_peered":"2023-07-14T19:15:58.246729+0000","last_unstale":"2023-07-14T19:15:59.722818+0000","last_undegraded":"2023-07-14T19:15:59.722818+0000","last_fullsized":"2023-07-14T19:15:59.722818+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_clean_scrub_stamp":"2023-07-14T19:15:57.219223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T22:17:27.678424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-07-14T19:15:59.231044+0000","last_change":"2023-07-14T19:15:58.243023+0000","last_active":"2023-07-14T19:15:59.231044+0000","last_peered":"2023-07-14T19:15:59.231044+0000","last_clean":"2023-07-14T19:15:59.231044+0000","last_became_active":"2023-07-14T19:15:58.242630+0000","last_became_peered":"2023-07-14T19:15:58.242630+0000","last_unstale":"2023-07-14T19:15:59.231044+0000","last_undegraded":"2023-07-14T19:15:59.231044+0000","last_fullsized":"2023-07-14T19:15:59.231044+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_clean_scrub_stamp":"2023-07-14T19:15:57.219223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T23:27:40.017638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-07-14T19:15:59.723119+0000","last_change":"2023-07-14T19:15:58.246205+0000","last_active":"2023-07-14T19:15:59.723119+0000","last_peered":"2023-07-14T19:15:59.723119+0000","last_clean":"2023-07-14T19:15:59.723119+0000","last_became_active":"2023-07-14T19:15:58.245720+0000","last_became_peered":"2023-07-14T19:15:58.245720+0000","last_unstale":"2023-07-14T19:15:59.723119+0000","last_undegraded":"2023-07-14T19:15:59.723119+0000","last_fullsized":"2023-07-14T19:15:59.723119+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_clean_scrub_stamp":"2023-07-14T19:15:57.219223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T05:22:46.730021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-07-14T19:15:59.722477+0000","last_change":"2023-07-14T19:15:58.248611+0000","last_active":"2023-07-14T19:15:59.722477+0000","last_peered":"2023-07-14T19:15:59.722477+0000","last_clean":"2023-07-14T19:15:59.722477+0000","last_became_active":"2023-07-14T19:15:58.248352+0000","last_became_peered":"2023-07-14T19:15:58.248352+0000","last_unstale":"2023-07-14T19:15:59.722477+0000","last_undegraded":"2023-07-14T19:15:59.722477+0000","last_fullsized":"2023-07-14T19:15:59.722477+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_clean_scrub_stamp":"2023-07-14T19:15:57.219223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T06:43:46.682942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-07-14T19:15:59.722246+0000","last_change":"2023-07-14T19:15:58.243167+0000","last_active":"2023-07-14T19:15:59.722246+0000","last_peered":"2023-07-14T19:15:59.722246+0000","last_clean":"2023-07-14T19:15:59.722246+0000","last_became_active":"2023-07-14T19:15:58.241770+0000","last_became_peered":"2023-07-14T19:15:58.241770+0000","last_unstale":"2023-07-14T19:15:59.722246+0000","last_undegraded":"2023-07-14T19:15:59.722246+0000","last_fullsized":"2023-07-14T19:15:59.722246+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_clean_scrub_stamp":"2023-07-14T19:15:57.219223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T05:56:34.485106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-07-14T19:15:59.938811+0000","last_change":"2023-07-14T19:15:58.243972+0000","last_active":"2023-07-14T19:15:59.938811+0000","last_peered":"2023-07-14T19:15:59.938811+0000","last_clean":"2023-07-14T19:15:59.938811+0000","last_became_active":"2023-07-14T19:15:58.243685+0000","last_became_peered":"2023-07-14T19:15:58.243685+0000","last_unstale":"2023-07-14T19:15:59.938811+0000","last_undegraded":"2023-07-14T19:15:59.938811+0000","last_fullsized":"2023-07-14T19:15:59.938811+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_clean_scrub_stamp":"2023-07-14T19:15:57.219223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:44:54.955614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":442368,"data_stored":4722944,"data_compressed":280807,"data_compressed_allocated":360448,"data_compressed_original":4669440,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":4234,"num_objects":51,"num_object_clones":0,"num_object_copies":153,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":51,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":112,"num_write_kb":33,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":36,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":180224,"data_stored":11216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":96,"ondisk_log_size":96,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":76,"seq":326417514508,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27868,"kb_used_data":848,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717284,"statfs":{"total":95995035648,"available":95966498816,"internally_reserved":0,"allocated":868352,"data_stored":415920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47699999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59899999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.44800000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.436}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61699999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52700000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.626}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65400000000000003}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60799999999999998}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70599999999999996}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69499999999999995}]}]},{"osd":10,"up_from":69,"seq":296352743439,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27940,"kb_used_data":920,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717212,"statfs":{"total":95995035648,"available":95966425088,"internally_reserved":0,"allocated":942080,"data_stored":1008046,"data_compressed":35714,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41999999999999998}]},{"osd":1,"last update":"Fri Jul 14 19:16:00 2023","interfaces":[{"interface":"back","average":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.66500000000000004,"5min":0.66500000000000004,"15min":0.66500000000000004},"last":0.47999999999999998},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.76900000000000002,"5min":0.76900000000000002,"15min":0.76900000000000002},"last":0.442}]},{"osd":2,"last update":"Fri Jul 14 19:16:00 2023","interfaces":[{"interface":"back","average":{"1min":0.67100000000000004,"5min":0.67100000000000004,"15min":0.67100000000000004},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":1.0009999999999999,"5min":1.0009999999999999,"15min":1.0009999999999999},"last":0.60999999999999999},{"interface":"front","average":{"1min":0.751,"5min":0.751,"15min":0.751},"min":{"1min":0.51500000000000001,"5min":0.51500000000000001,"15min":0.51500000000000001},"max":{"1min":1.1379999999999999,"5min":1.1379999999999999,"15min":1.1379999999999999},"last":0.70199999999999996}]},{"osd":3,"last update":"Fri Jul 14 19:16:00 2023","interfaces":[{"interface":"back","average":{"1min":0.70099999999999996,"5min":0.70099999999999996,"15min":0.70099999999999996},"min":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":0.92200000000000004,"5min":0.92200000000000004,"15min":0.92200000000000004},"last":0.53800000000000003},{"interface":"front","average":{"1min":0.71599999999999997,"5min":0.71599999999999997,"15min":0.71599999999999997},"min":{"1min":0.45300000000000001,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":0.90900000000000003,"5min":0.90900000000000003,"15min":0.90900000000000003},"last":0.55100000000000005}]},{"osd":4,"last update":"Fri Jul 14 19:16:00 2023","interfaces":[{"interface":"back","average":{"1min":0.64700000000000002,"5min":0.64700000000000002,"15min":0.64700000000000002},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.83199999999999996,"5min":0.83199999999999996,"15min":0.83199999999999996},"last":0.50800000000000001},{"interface":"front","average":{"1min":0.59999999999999998,"5min":0.59999999999999998,"15min":0.59999999999999998},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.878,"5min":0.878,"15min":0.878},"last":0.46100000000000002}]},{"osd":5,"last update":"Fri Jul 14 19:16:00 2023","interfaces":[{"interface":"back","average":{"1min":0.76000000000000001,"5min":0.76000000000000001,"15min":0.76000000000000001},"min":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"max":{"1min":1.1259999999999999,"5min":1.1259999999999999,"15min":1.1259999999999999},"last":0.77300000000000002},{"interface":"front","average":{"1min":0.77900000000000003,"5min":0.77900000000000003,"15min":0.77900000000000003},"min":{"1min":0.56299999999999994,"5min":0.56299999999999994,"15min":0.56299999999999994},"max":{"1min":1.018,"5min":1.018,"15min":1.018},"last":0.66300000000000003}]},{"osd":6,"last update":"Fri Jul 14 19:16:00 2023","interfaces":[{"interface":"back","average":{"1min":0.82799999999999996,"5min":0.82799999999999996,"15min":0.82799999999999996},"min":{"1min":0.56899999999999995,"5min":0.56899999999999995,"15min":0.56899999999999995},"max":{"1min":1.294,"5min":1.294,"15min":1.294},"last":0.56899999999999995},{"interface":"front","average":{"1min":0.80400000000000005,"5min":0.80400000000000005,"15min":0.80400000000000005},"min":{"1min":0.57999999999999996,"5min":0.57999999999999996,"15min":0.57999999999999996},"max":{"1min":1.0960000000000001,"5min":1.0960000000000001,"15min":1.0960000000000001},"last":0.57999999999999996}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.49099999999999999}]},{"osd":8,"last update":"Fri Jul 14 19:16:00 2023","interfaces":[{"interface":"back","average":{"1min":0.84599999999999997,"5min":0.84599999999999997,"15min":0.84599999999999997},"min":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"max":{"1min":1.0349999999999999,"5min":1.0349999999999999,"15min":1.0349999999999999},"last":0.74099999999999999},{"interface":"front","average":{"1min":0.88900000000000001,"5min":0.88900000000000001,"15min":0.88900000000000001},"min":{"1min":0.60499999999999998,"5min":0.60499999999999998,"15min":0.60499999999999998},"max":{"1min":1.369,"5min":1.369,"15min":1.369},"last":0.67400000000000004}]},{"osd":9,"last update":"Fri Jul 14 19:16:00 2023","interfaces":[{"interface":"back","average":{"1min":0.88600000000000001,"5min":0.88600000000000001,"15min":0.88600000000000001},"min":{"1min":0.65000000000000002,"5min":0.65000000000000002,"15min":0.65000000000000002},"max":{"1min":1.1759999999999999,"5min":1.1759999999999999,"15min":1.1759999999999999},"last":0.65000000000000002},{"interface":"front","average":{"1min":0.88200000000000001,"5min":0.88200000000000001,"15min":0.88200000000000001},"min":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":1.468,"5min":1.468,"15min":1.468},"last":0.52800000000000002}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.80900000000000005}]}]},{"osd":9,"up_from":64,"seq":274877906964,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27888,"kb_used_data":872,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717264,"statfs":{"total":95995035648,"available":95966478336,"internally_reserved":0,"allocated":892928,"data_stored":416828,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jul 14 19:15:35 2023","interfaces":[{"interface":"back","average":{"1min":0.48699999999999999,"5min":0.48699999999999999,"15min":0.48699999999999999},"min":{"1min":0.29999999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.82099999999999995,"5min":0.82099999999999995,"15min":0.82099999999999995},"last":0.30099999999999999},{"interface":"front","average":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.46100000000000002},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.65300000000000002,"5min":0.65300000000000002,"15min":0.65300000000000002},"last":0.32000000000000001}]},{"osd":1,"last update":"Fri Jul 14 19:15:35 2023","interfaces":[{"interface":"back","average":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"min":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":1.0149999999999999,"5min":1.0149999999999999,"15min":1.0149999999999999},"last":0.53800000000000003},{"interface":"front","average":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":0.65700000000000003},"min":{"1min":0.53700000000000003,"5min":0.53700000000000003,"15min":0.53700000000000003},"max":{"1min":0.85199999999999998,"5min":0.85199999999999998,"15min":0.85199999999999998},"last":0.47899999999999998}]},{"osd":2,"last update":"Fri Jul 14 19:15:35 2023","interfaces":[{"interface":"back","average":{"1min":0.65800000000000003,"5min":0.65800000000000003,"15min":0.65800000000000003},"min":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":0.92200000000000004,"5min":0.92200000000000004,"15min":0.92200000000000004},"last":0.58399999999999996},{"interface":"front","average":{"1min":0.65800000000000003,"5min":0.65800000000000003,"15min":0.65800000000000003},"min":{"1min":0.46500000000000002,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":0.91000000000000003,"5min":0.91000000000000003,"15min":0.91000000000000003},"last":0.55100000000000005}]},{"osd":3,"last update":"Fri Jul 14 19:15:35 2023","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":0.876,"5min":0.876,"15min":0.876},"last":0.41899999999999998},{"interface":"front","average":{"1min":0.61699999999999999,"5min":0.61699999999999999,"15min":0.61699999999999999},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.96499999999999997,"5min":0.96499999999999997,"15min":0.96499999999999997},"last":0.34999999999999998}]},{"osd":4,"last update":"Fri Jul 14 19:15:35 2023","interfaces":[{"interface":"back","average":{"1min":0.68500000000000005,"5min":0.68500000000000005,"15min":0.68500000000000005},"min":{"1min":0.48999999999999999,"5min":0.48999999999999999,"15min":0.48999999999999999},"max":{"1min":1.0700000000000001,"5min":1.0700000000000001,"15min":1.0700000000000001},"last":0.50600000000000001},{"interface":"front","average":{"1min":0.67000000000000004,"5min":0.67000000000000004,"15min":0.67000000000000004},"min":{"1min":0.47199999999999998,"5min":0.47199999999999998,"15min":0.47199999999999998},"max":{"1min":0.94099999999999995,"5min":0.94099999999999995,"15min":0.94099999999999995},"last":0.496}]},{"osd":5,"last update":"Fri Jul 14 19:15:35 2023","interfaces":[{"interface":"back","average":{"1min":0.76500000000000001,"5min":0.76500000000000001,"15min":0.76500000000000001},"min":{"1min":0.43099999999999999,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":1.1140000000000001,"5min":1.1140000000000001,"15min":1.1140000000000001},"last":0.59699999999999998},{"interface":"front","average":{"1min":0.72399999999999998,"5min":0.72399999999999998,"15min":0.72399999999999998},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":1.018,"5min":1.018,"15min":1.018},"last":0.60999999999999999}]},{"osd":6,"last update":"Fri Jul 14 19:15:35 2023","interfaces":[{"interface":"back","average":{"1min":0.79800000000000004,"5min":0.79800000000000004,"15min":0.79800000000000004},"min":{"1min":0.44800000000000001,"5min":0.44800000000000001,"15min":0.44800000000000001},"max":{"1min":1.6839999999999999,"5min":1.6839999999999999,"15min":1.6839999999999999},"last":0.52500000000000002},{"interface":"front","average":{"1min":0.76700000000000002,"5min":0.76700000000000002,"15min":0.76700000000000002},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":1.145,"5min":1.145,"15min":1.145},"last":0.439}]},{"osd":7,"last update":"Fri Jul 14 19:15:35 2023","interfaces":[{"interface":"back","average":{"1min":0.78300000000000003,"5min":0.78300000000000003,"15min":0.78300000000000003},"min":{"1min":0.48699999999999999,"5min":0.48699999999999999,"15min":0.48699999999999999},"max":{"1min":1.0800000000000001,"5min":1.0800000000000001,"15min":1.0800000000000001},"last":0.56399999999999995},{"interface":"front","average":{"1min":0.83099999999999996,"5min":0.83099999999999996,"15min":0.83099999999999996},"min":{"1min":0.58499999999999996,"5min":0.58499999999999996,"15min":0.58499999999999996},"max":{"1min":1.0469999999999999,"5min":1.0469999999999999,"15min":1.0469999999999999},"last":0.621}]},{"osd":8,"last update":"Fri Jul 14 19:15:35 2023","interfaces":[{"interface":"back","average":{"1min":0.871,"5min":0.871,"15min":0.871},"min":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"max":{"1min":1.242,"5min":1.242,"15min":1.242},"last":0.629},{"interface":"front","average":{"1min":0.82699999999999996,"5min":0.82699999999999996,"15min":0.82699999999999996},"min":{"1min":0.58499999999999996,"5min":0.58499999999999996,"15min":0.58499999999999996},"max":{"1min":1.173,"5min":1.173,"15min":1.173},"last":0.65100000000000002}]},{"osd":10,"last update":"Fri Jul 14 19:15:58 2023","interfaces":[{"interface":"back","average":{"1min":0.79200000000000004,"5min":0.79200000000000004,"15min":0.79200000000000004},"min":{"1min":0.54900000000000004,"5min":0.54900000000000004,"15min":0.54900000000000004},"max":{"1min":1.0820000000000001,"5min":1.0820000000000001,"15min":1.0820000000000001},"last":0.64200000000000002},{"interface":"front","average":{"1min":0.79100000000000004,"5min":0.79100000000000004,"15min":0.79100000000000004},"min":{"1min":0.59599999999999997,"5min":0.59599999999999997,"15min":0.59599999999999997},"max":{"1min":1.044,"5min":1.044,"15min":1.044},"last":0.66100000000000003}]}]},{"osd":8,"up_from":57,"seq":244813135897,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27880,"kb_used_data":864,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717272,"statfs":{"total":95995035648,"available":95966486528,"internally_reserved":0,"allocated":884736,"data_stored":416327,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jul 14 19:15:13 2023","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.52100000000000002},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.78600000000000003,"5min":0.78600000000000003,"15min":0.78600000000000003},"last":0.505},{"interface":"front","average":{"1min":0.58599999999999997,"5min":0.58599999999999997,"15min":0.58599999999999997},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.80500000000000005,"5min":0.80500000000000005,"15min":0.80500000000000005},"last":0.54800000000000004}]},{"osd":1,"last update":"Fri Jul 14 19:15:13 2023","interfaces":[{"interface":"back","average":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"min":{"1min":0.375,"5min":0.375,"15min":0.375},"max":{"1min":1.0069999999999999,"5min":1.0069999999999999,"15min":1.0069999999999999},"last":0.69099999999999995},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.59299999999999997,"15min":0.59299999999999997},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.91500000000000004,"5min":0.91500000000000004,"15min":0.91500000000000004},"last":0.78400000000000003}]},{"osd":2,"last update":"Fri Jul 14 19:15:13 2023","interfaces":[{"interface":"back","average":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.54300000000000004},"last":0.46899999999999997},{"interface":"front","average":{"1min":0.48699999999999999,"5min":0.48699999999999999,"15min":0.48699999999999999},"min":{"1min":0.31,"5min":0.31,"15min":0.31},"max":{"1min":0.69399999999999995,"5min":0.69399999999999995,"15min":0.69399999999999995},"last":0.66700000000000004}]},{"osd":3,"last update":"Fri Jul 14 19:15:13 2023","interfaces":[{"interface":"back","average":{"1min":0.60999999999999999,"5min":0.60999999999999999,"15min":0.60999999999999999},"min":{"1min":0.44,"5min":0.44,"15min":0.44},"max":{"1min":0.84799999999999998,"5min":0.84799999999999998,"15min":0.84799999999999998},"last":0.64700000000000002},{"interface":"front","average":{"1min":0.64800000000000002,"5min":0.64800000000000002,"15min":0.64800000000000002},"min":{"1min":0.49099999999999999,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":0.80700000000000005,"5min":0.80700000000000005,"15min":0.80700000000000005},"last":0.61699999999999999}]},{"osd":4,"last update":"Fri Jul 14 19:15:13 2023","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.58099999999999996,"15min":0.58099999999999996},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":1.069,"5min":1.069,"15min":1.069},"last":0.74299999999999999},{"interface":"front","average":{"1min":0.68500000000000005,"5min":0.68500000000000005,"15min":0.68500000000000005},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":1.474,"5min":1.474,"15min":1.474},"last":0.75800000000000001}]},{"osd":5,"last update":"Fri Jul 14 19:15:13 2023","interfaces":[{"interface":"back","average":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":0.75700000000000001,"5min":0.75700000000000001,"15min":0.75700000000000001},"last":0.72299999999999998},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.56899999999999995,"15min":0.56899999999999995},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":1.0349999999999999,"5min":1.0349999999999999,"15min":1.0349999999999999},"last":0.71099999999999997}]},{"osd":6,"last update":"Fri Jul 14 19:15:13 2023","interfaces":[{"interface":"back","average":{"1min":0.66400000000000003,"5min":0.66400000000000003,"15min":0.66400000000000003},"min":{"1min":0.44800000000000001,"5min":0.44800000000000001,"15min":0.44800000000000001},"max":{"1min":0.86299999999999999,"5min":0.86299999999999999,"15min":0.86299999999999999},"last":0.84299999999999997},{"interface":"front","average":{"1min":0.73199999999999998,"5min":0.73199999999999998,"15min":0.73199999999999998},"min":{"1min":0.43099999999999999,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":1.1399999999999999,"5min":1.1399999999999999,"15min":1.1399999999999999},"last":0.82999999999999996}]},{"osd":7,"last update":"Fri Jul 14 19:15:13 2023","interfaces":[{"interface":"back","average":{"1min":0.72599999999999998,"5min":0.72599999999999998,"15min":0.72599999999999998},"min":{"1min":0.48399999999999999,"5min":0.48399999999999999,"15min":0.48399999999999999},"max":{"1min":1.1879999999999999,"5min":1.1879999999999999,"15min":1.1879999999999999},"last":0.93500000000000005},{"interface":"front","average":{"1min":0.72799999999999998,"5min":0.72799999999999998,"15min":0.72799999999999998},"min":{"1min":0.5,"5min":0.5,"15min":0.5},"max":{"1min":1.093,"5min":1.093,"15min":1.093},"last":0.86399999999999999}]},{"osd":9,"last update":"Fri Jul 14 19:15:33 2023","interfaces":[{"interface":"back","average":{"1min":0.754,"5min":0.754,"15min":0.754},"min":{"1min":0.55600000000000005,"5min":0.55600000000000005,"15min":0.55600000000000005},"max":{"1min":0.97399999999999998,"5min":0.97399999999999998,"15min":0.97399999999999998},"last":0.89800000000000002},{"interface":"front","average":{"1min":0.75600000000000001,"5min":0.75600000000000001,"15min":0.75600000000000001},"min":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"max":{"1min":0.97099999999999997,"5min":0.97099999999999997,"15min":0.97099999999999997},"last":0.88700000000000001}]},{"osd":10,"last update":"Fri Jul 14 19:15:55 2023","interfaces":[{"interface":"back","average":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"min":{"1min":0.52700000000000002,"5min":0.52700000000000002,"15min":0.52700000000000002},"max":{"1min":1.006,"5min":1.006,"15min":1.006},"last":0.877},{"interface":"front","average":{"1min":0.73599999999999999,"5min":0.73599999999999999,"15min":0.73599999999999999},"min":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"max":{"1min":0.95999999999999996,"5min":0.95999999999999996,"15min":0.95999999999999996},"last":0.95399999999999996}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.91700000000000004}]}]},{"osd":7,"up_from":50,"seq":214748364828,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27916,"kb_used_data":896,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717236,"statfs":{"total":95995035648,"available":95966449664,"internally_reserved":0,"allocated":917504,"data_stored":1006597,"data_compressed":35714,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jul 14 19:15:51 2023","interfaces":[{"interface":"back","average":{"1min":0.57199999999999995,"5min":0.53300000000000003,"15min":0.52700000000000002},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.82999999999999996,"5min":0.82999999999999996,"15min":0.82999999999999996},"last":0.5},{"interface":"front","average":{"1min":0.55200000000000005,"5min":0.55600000000000005,"15min":0.55600000000000005},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.78100000000000003,"5min":0.89300000000000002,"15min":0.89300000000000002},"last":0.51800000000000002}]},{"osd":1,"last update":"Fri Jul 14 19:15:51 2023","interfaces":[{"interface":"back","average":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"min":{"1min":0.25700000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.67100000000000004,"5min":0.67100000000000004,"15min":0.67100000000000004},"last":0.44},{"interface":"front","average":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"min":{"1min":0.254,"5min":0.254,"15min":0.254},"max":{"1min":0.69099999999999995,"5min":0.91300000000000003,"15min":0.91300000000000003},"last":0.36299999999999999}]},{"osd":2,"last update":"Fri Jul 14 19:15:51 2023","interfaces":[{"interface":"back","average":{"1min":0.60199999999999998,"5min":0.59299999999999997,"15min":0.59099999999999997},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.78600000000000003,"5min":0.83999999999999997,"15min":0.83999999999999997},"last":0.58899999999999997},{"interface":"front","average":{"1min":0.65600000000000003,"5min":0.628,"15min":0.623},"min":{"1min":0.43099999999999999,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":0.96999999999999997,"5min":0.96999999999999997,"15min":0.96999999999999997},"last":0.79400000000000004}]},{"osd":3,"last update":"Fri Jul 14 19:15:51 2023","interfaces":[{"interface":"back","average":{"1min":0.61699999999999999,"5min":0.56599999999999995,"15min":0.55800000000000005},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":0.89200000000000002,"5min":0.89200000000000002,"15min":0.89200000000000002},"last":0.72299999999999998},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.80300000000000005,"5min":0.91700000000000004,"15min":0.91700000000000004},"last":0.56299999999999994}]},{"osd":4,"last update":"Fri Jul 14 19:15:51 2023","interfaces":[{"interface":"back","average":{"1min":0.53800000000000003,"5min":0.54000000000000004,"15min":0.54000000000000004},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.81499999999999995,"5min":0.86399999999999999,"15min":0.86399999999999999},"last":0.40600000000000003},{"interface":"front","average":{"1min":0.497,"5min":0.48099999999999998,"15min":0.47899999999999998},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":0.80600000000000005,"5min":0.80600000000000005,"15min":0.80600000000000005},"last":0.33100000000000002}]},{"osd":5,"last update":"Fri Jul 14 19:15:51 2023","interfaces":[{"interface":"back","average":{"1min":0.65100000000000002,"5min":0.67800000000000005,"15min":0.68200000000000005},"min":{"1min":0.43099999999999999,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":0.90600000000000003,"5min":1.355,"15min":1.355},"last":0.80600000000000005},{"interface":"front","average":{"1min":0.73599999999999999,"5min":0.71499999999999997,"15min":0.71099999999999997},"min":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":1.048,"5min":1.3009999999999999,"15min":1.3009999999999999},"last":0.625}]},{"osd":6,"last update":"Fri Jul 14 19:15:51 2023","interfaces":[{"interface":"back","average":{"1min":0.69599999999999995,"5min":0.69399999999999995,"15min":0.69399999999999995},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":0.93200000000000005,"5min":1.2310000000000001,"15min":1.2310000000000001},"last":0.61099999999999999},{"interface":"front","average":{"1min":0.70399999999999996,"5min":0.66800000000000004,"15min":0.66200000000000003},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":1.069,"5min":1.1140000000000001,"15min":1.1140000000000001},"last":0.68899999999999995}]},{"osd":8,"last update":"Fri Jul 14 19:15:13 2023","interfaces":[{"interface":"back","average":{"1min":0.74399999999999999,"5min":0.74399999999999999,"15min":0.74399999999999999},"min":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"max":{"1min":1.0620000000000001,"5min":1.0620000000000001,"15min":1.0620000000000001},"last":0.81999999999999995},{"interface":"front","average":{"1min":0.72999999999999998,"5min":0.72999999999999998,"15min":0.72999999999999998},"min":{"1min":0.495,"5min":0.495,"15min":0.495},"max":{"1min":0.93500000000000005,"5min":0.93500000000000005,"15min":0.93500000000000005},"last":0.73799999999999999}]},{"osd":9,"last update":"Fri Jul 14 19:15:35 2023","interfaces":[{"interface":"back","average":{"1min":0.754,"5min":0.754,"15min":0.754},"min":{"1min":0.48499999999999999,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":1.1599999999999999,"5min":1.1599999999999999,"15min":1.1599999999999999},"last":0.65400000000000003},{"interface":"front","average":{"1min":0.77800000000000002,"5min":0.77800000000000002,"15min":0.77800000000000002},"min":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":1.214,"5min":1.214,"15min":1.214},"last":0.75800000000000001}]},{"osd":10,"last update":"Fri Jul 14 19:15:55 2023","interfaces":[{"interface":"back","average":{"1min":0.753,"5min":0.753,"15min":0.753},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":1.0960000000000001,"5min":1.0960000000000001,"15min":1.0960000000000001},"last":0.496},{"interface":"front","average":{"1min":0.748,"5min":0.748,"15min":0.748},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":1.0980000000000001,"5min":1.0980000000000001,"15min":1.0980000000000001},"last":0.52100000000000002}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67500000000000004}]}]},{"osd":6,"up_from":45,"seq":193273528353,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27872,"kb_used_data":848,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717280,"statfs":{"total":95995035648,"available":95966494720,"internally_reserved":0,"allocated":868352,"data_stored":415920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jul 14 19:15:32 2023","interfaces":[{"interface":"back","average":{"1min":0.47699999999999998,"5min":0.42099999999999999,"15min":0.41099999999999998},"min":{"1min":0.34100000000000003,"5min":0.222,"15min":0.222},"max":{"1min":0.77100000000000002,"5min":0.77100000000000002,"15min":0.77100000000000002},"last":0.29499999999999998},{"interface":"front","average":{"1min":0.53200000000000003,"5min":0.45800000000000002,"15min":0.44600000000000001},"min":{"1min":0.32600000000000001,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":1.1459999999999999,"5min":1.1459999999999999,"15min":1.1459999999999999},"last":0.32700000000000001}]},{"osd":1,"last update":"Fri Jul 14 19:15:32 2023","interfaces":[{"interface":"back","average":{"1min":0.66300000000000003,"5min":0.57799999999999996,"15min":0.56399999999999995},"min":{"1min":0.51800000000000002,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.89600000000000002,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.371},{"interface":"front","average":{"1min":0.65700000000000003,"5min":0.69599999999999995,"15min":0.70199999999999996},"min":{"1min":0.51500000000000001,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":0.85299999999999998,"5min":3.1930000000000001,"15min":3.1930000000000001},"last":0.47099999999999997}]},{"osd":2,"last update":"Fri Jul 14 19:15:32 2023","interfaces":[{"interface":"back","average":{"1min":0.67100000000000004,"5min":0.59999999999999998,"15min":0.58799999999999997},"min":{"1min":0.436,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.93300000000000005,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.56999999999999995},{"interface":"front","average":{"1min":0.625,"5min":0.58399999999999996,"15min":0.57699999999999996},"min":{"1min":0.435,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.84699999999999998,"5min":0.85299999999999998,"15min":0.85299999999999998},"last":0.59899999999999998}]},{"osd":3,"last update":"Fri Jul 14 19:15:32 2023","interfaces":[{"interface":"back","average":{"1min":0.54200000000000004,"5min":0.504,"15min":0.498},"min":{"1min":0.35199999999999998,"5min":0.314,"15min":0.314},"max":{"1min":0.82799999999999996,"5min":0.82799999999999996,"15min":0.82799999999999996},"last":0.438},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.54700000000000004,"15min":0.54100000000000004},"min":{"1min":0.36799999999999999,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.78700000000000003,"5min":0.96199999999999997,"15min":0.96199999999999997},"last":0.42299999999999999}]},{"osd":4,"last update":"Fri Jul 14 19:15:32 2023","interfaces":[{"interface":"back","average":{"1min":0.73299999999999998,"5min":0.64200000000000002,"15min":0.627},"min":{"1min":0.51200000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.91500000000000004,"5min":0.91500000000000004,"15min":0.91500000000000004},"last":0.45900000000000002},{"interface":"front","average":{"1min":0.749,"5min":0.67000000000000004,"15min":0.65700000000000003},"min":{"1min":0.52600000000000002,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":0.97499999999999998,"5min":1.0309999999999999,"15min":1.0309999999999999},"last":0.55100000000000005}]},{"osd":5,"last update":"Fri Jul 14 19:15:32 2023","interfaces":[{"interface":"back","average":{"1min":0.73799999999999999,"5min":0.75800000000000001,"15min":0.76100000000000001},"min":{"1min":0.48799999999999999,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":1.085,"5min":3.1299999999999999,"15min":3.1299999999999999},"last":0.58199999999999996},{"interface":"front","average":{"1min":0.78200000000000003,"5min":0.68600000000000005,"15min":0.67000000000000004},"min":{"1min":0.53900000000000003,"5min":0.38,"15min":0.38},"max":{"1min":1.069,"5min":1.105,"15min":1.105},"last":0.67600000000000005}]},{"osd":7,"last update":"Fri Jul 14 19:15:56 2023","interfaces":[{"interface":"back","average":{"1min":0.79000000000000004,"5min":0.84199999999999997,"15min":0.85099999999999998},"min":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"max":{"1min":1.0489999999999999,"5min":3.25,"15min":3.25},"last":0.53500000000000003},{"interface":"front","average":{"1min":0.79600000000000004,"5min":0.749,"15min":0.74099999999999999},"min":{"1min":0.58299999999999996,"5min":0.54000000000000004,"15min":0.54000000000000004},"max":{"1min":1.0589999999999999,"5min":1.0589999999999999,"15min":1.0589999999999999},"last":0.61499999999999999}]},{"osd":8,"last update":"Fri Jul 14 19:15:14 2023","interfaces":[{"interface":"back","average":{"1min":0.74099999999999999,"5min":0.74099999999999999,"15min":0.74099999999999999},"min":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":1.044,"5min":1.044,"15min":1.044},"last":0.64700000000000002},{"interface":"front","average":{"1min":0.94999999999999996,"5min":0.94999999999999996,"15min":0.94999999999999996},"min":{"1min":0.57999999999999996,"5min":0.57999999999999996,"15min":0.57999999999999996},"max":{"1min":3.226,"5min":3.226,"15min":3.226},"last":0.66400000000000003}]},{"osd":9,"last update":"Fri Jul 14 19:15:38 2023","interfaces":[{"interface":"back","average":{"1min":0.85099999999999998,"5min":0.85099999999999998,"15min":0.85099999999999998},"min":{"1min":0.502,"5min":0.502,"15min":0.502},"max":{"1min":1.123,"5min":1.123,"15min":1.123},"last":0.51800000000000002},{"interface":"front","average":{"1min":0.83499999999999996,"5min":0.83499999999999996,"15min":0.83499999999999996},"min":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":1.135,"5min":1.135,"15min":1.135},"last":0.5}]},{"osd":10,"last update":"Fri Jul 14 19:15:56 2023","interfaces":[{"interface":"back","average":{"1min":0.85899999999999999,"5min":0.85899999999999999,"15min":0.85899999999999999},"min":{"1min":0.63,"5min":0.63,"15min":0.63},"max":{"1min":1.0840000000000001,"5min":1.0840000000000001,"15min":1.0840000000000001},"last":0.63},{"interface":"front","average":{"1min":0.85699999999999998,"5min":0.85699999999999998,"15min":0.85699999999999998},"min":{"1min":0.64100000000000001,"5min":0.64100000000000001,"15min":0.64100000000000001},"max":{"1min":1.0820000000000001,"5min":1.0820000000000001,"15min":1.0820000000000001},"last":0.70599999999999996}]}]},{"osd":1,"up_from":13,"seq":55834574902,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27872,"kb_used_data":856,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717280,"statfs":{"total":95995035648,"available":95966494720,"internally_reserved":0,"allocated":876544,"data_stored":417211,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jul 14 19:15:49 2023","interfaces":[{"interface":"back","average":{"1min":0.60199999999999998,"5min":0.51100000000000001,"15min":0.45400000000000001},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.81999999999999995,"5min":1.149,"15min":1.149},"last":0.42099999999999999},{"interface":"front","average":{"1min":0.63500000000000001,"5min":0.52500000000000002,"15min":0.46999999999999997},"min":{"1min":0.36099999999999999,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.97099999999999997,"5min":1.0740000000000001,"15min":1.0740000000000001},"last":0.51200000000000001}]},{"osd":2,"last update":"Fri Jul 14 19:15:09 2023","interfaces":[{"interface":"back","average":{"1min":0.51400000000000001,"5min":0.51300000000000001,"15min":0.51200000000000001},"min":{"1min":0.39300000000000002,"5min":0.372,"15min":0.372},"max":{"1min":0.70699999999999996,"5min":0.875,"15min":0.875},"last":0.35799999999999998},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.51700000000000002,"15min":0.505},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.79500000000000004,"5min":0.79500000000000004,"15min":0.79500000000000004},"last":0.45500000000000002}]},{"osd":3,"last update":"Fri Jul 14 19:15:28 2023","interfaces":[{"interface":"back","average":{"1min":0.65400000000000003,"5min":0.58199999999999996,"15min":0.56499999999999995},"min":{"1min":0.45500000000000002,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":1.1659999999999999,"5min":1.1659999999999999,"15min":1.1659999999999999},"last":0.436},{"interface":"front","average":{"1min":0.629,"5min":0.57799999999999996,"15min":0.56399999999999995},"min":{"1min":0.41599999999999998,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":1.177,"5min":1.177,"15min":1.177},"last":0.46899999999999997}]},{"osd":4,"last update":"Fri Jul 14 19:15:49 2023","interfaces":[{"interface":"back","average":{"1min":0.52800000000000002,"5min":0.435,"15min":0.42299999999999999},"min":{"1min":0.32100000000000001,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.77300000000000002,"5min":0.77300000000000002,"15min":0.77300000000000002},"last":0.38300000000000001},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.47099999999999997,"15min":0.45500000000000002},"min":{"1min":0.38500000000000001,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.80300000000000005,"5min":0.80300000000000005,"15min":0.80300000000000005},"last":0.48899999999999999}]},{"osd":5,"last update":"Fri Jul 14 19:15:09 2023","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.52600000000000002,"15min":0.52300000000000002},"min":{"1min":0.41799999999999998,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.71199999999999997,"5min":0.71199999999999997,"15min":0.71199999999999997},"last":0.40500000000000003},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.58499999999999996,"15min":0.58599999999999997},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.82699999999999996,"5min":0.82699999999999996,"15min":0.82699999999999996},"last":0.52300000000000002}]},{"osd":6,"last update":"Fri Jul 14 19:15:36 2023","interfaces":[{"interface":"back","average":{"1min":0.65400000000000003,"5min":0.63900000000000001,"15min":0.63700000000000001},"min":{"1min":0.42699999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.84699999999999998,"5min":1.266,"15min":1.266},"last":0.54000000000000004},{"interface":"front","average":{"1min":0.64800000000000002,"5min":0.624,"15min":0.62},"min":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":0.93500000000000005,"5min":1.083,"15min":1.083},"last":0.56299999999999994}]},{"osd":7,"last update":"Fri Jul 14 19:15:56 2023","interfaces":[{"interface":"back","average":{"1min":0.65600000000000003,"5min":0.55000000000000004,"15min":0.53200000000000003},"min":{"1min":0.378,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.86799999999999999,"5min":0.86799999999999999,"15min":0.86799999999999999},"last":0.55200000000000005},{"interface":"front","average":{"1min":0.68799999999999994,"5min":0.61199999999999999,"15min":0.59899999999999998},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.93300000000000005,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.58099999999999996}]},{"osd":8,"last update":"Fri Jul 14 19:15:15 2023","interfaces":[{"interface":"back","average":{"1min":0.65800000000000003,"5min":0.65800000000000003,"15min":0.65800000000000003},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.89100000000000001,"5min":0.89100000000000001,"15min":0.89100000000000001},"last":0.61899999999999999},{"interface":"front","average":{"1min":0.63500000000000001,"5min":0.63500000000000001,"15min":0.63500000000000001},"min":{"1min":0.47999999999999998,"5min":0.47999999999999998,"15min":0.47999999999999998},"max":{"1min":0.84099999999999997,"5min":0.84099999999999997,"15min":0.84099999999999997},"last":0.60799999999999998}]},{"osd":9,"last update":"Fri Jul 14 19:15:36 2023","interfaces":[{"interface":"back","average":{"1min":0.72799999999999998,"5min":0.72799999999999998,"15min":0.72799999999999998},"min":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":0.94899999999999995,"5min":0.94899999999999995,"15min":0.94899999999999995},"last":0.63200000000000001},{"interface":"front","average":{"1min":0.747,"5min":0.747,"15min":0.747},"min":{"1min":0.53900000000000003,"5min":0.53900000000000003,"15min":0.53900000000000003},"max":{"1min":0.93400000000000005,"5min":0.93400000000000005,"15min":0.93400000000000005},"last":0.66000000000000003}]},{"osd":10,"last update":"Fri Jul 14 19:15:56 2023","interfaces":[{"interface":"back","average":{"1min":0.65800000000000003,"5min":0.65800000000000003,"15min":0.65800000000000003},"min":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":0.86799999999999999,"5min":0.86799999999999999,"15min":0.86799999999999999},"last":0.64000000000000001},{"interface":"front","average":{"1min":0.72599999999999998,"5min":0.72599999999999998,"15min":0.72599999999999998},"min":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"max":{"1min":0.88300000000000001,"5min":0.88300000000000001,"15min":0.88300000000000001},"last":0.59799999999999998}]}]},{"osd":0,"up_from":8,"seq":34359738427,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27876,"kb_used_data":852,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717276,"statfs":{"total":95995035648,"available":95966490624,"internally_reserved":0,"allocated":872448,"data_stored":416010,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Jul 14 19:15:50 2023","interfaces":[{"interface":"back","average":{"1min":0.627,"5min":0.55900000000000005,"15min":0.52900000000000003},"min":{"1min":0.49099999999999999,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.88,"5min":0.95799999999999996,"15min":0.95799999999999996},"last":0.44600000000000001},{"interface":"front","average":{"1min":0.64800000000000002,"5min":0.56399999999999995,"15min":0.54200000000000004},"min":{"1min":0.40600000000000003,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.91700000000000004,"5min":0.99099999999999999,"15min":0.99099999999999999},"last":0.495}]},{"osd":2,"last update":"Fri Jul 14 19:15:04 2023","interfaces":[{"interface":"back","average":{"1min":0.58699999999999997,"5min":0.52200000000000002,"15min":0.50600000000000001},"min":{"1min":0.35699999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.90500000000000003,"5min":0.90500000000000003,"15min":0.90500000000000003},"last":0.41099999999999998},{"interface":"front","average":{"1min":0.63300000000000001,"5min":0.53900000000000003,"15min":0.51700000000000002},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":1.044,"5min":1.044,"15min":1.044},"last":0.51800000000000002}]},{"osd":3,"last update":"Fri Jul 14 19:14:30 2023","interfaces":[{"interface":"back","average":{"1min":0.437,"5min":0.40100000000000002,"15min":0.39500000000000002},"min":{"1min":0.315,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.623,"5min":0.623,"15min":0.623},"last":0.35799999999999998},{"interface":"front","average":{"1min":0.497,"5min":0.47299999999999998,"15min":0.46899999999999997},"min":{"1min":0.34100000000000003,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.59999999999999998,"5min":0.63200000000000001,"15min":0.63200000000000001},"last":0.375}]},{"osd":4,"last update":"Fri Jul 14 19:15:50 2023","interfaces":[{"interface":"back","average":{"1min":0.65900000000000003,"5min":0.57899999999999996,"15min":0.55100000000000005},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":0.92900000000000005,"5min":0.95399999999999996,"15min":0.95399999999999996},"last":0.59499999999999997},{"interface":"front","average":{"1min":0.63200000000000001,"5min":0.59199999999999997,"15min":0.56999999999999995},"min":{"1min":0.44,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.86099999999999999,"5min":0.93400000000000005,"15min":0.93400000000000005},"last":0.623}]},{"osd":5,"last update":"Fri Jul 14 19:15:14 2023","interfaces":[{"interface":"back","average":{"1min":0.71099999999999997,"5min":0.63100000000000001,"15min":0.61799999999999999},"min":{"1min":0.48099999999999998,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":1.0089999999999999,"5min":1.0169999999999999,"15min":1.0169999999999999},"last":0.66400000000000003},{"interface":"front","average":{"1min":0.72099999999999997,"5min":0.60699999999999998,"15min":0.58799999999999997},"min":{"1min":0.55100000000000005,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":1.073,"5min":1.073,"15min":1.073},"last":0.67700000000000005}]},{"osd":6,"last update":"Fri Jul 14 19:15:33 2023","interfaces":[{"interface":"back","average":{"1min":0.53800000000000003,"5min":0.53200000000000003,"15min":0.53100000000000003},"min":{"1min":0.30499999999999999,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.79600000000000004,"5min":0.79600000000000004,"15min":0.79600000000000004},"last":0.53600000000000003},{"interface":"front","average":{"1min":0.70599999999999996,"5min":0.61599999999999999,"15min":0.60099999999999998},"min":{"1min":0.432,"5min":0.312,"15min":0.312},"max":{"1min":1.0169999999999999,"5min":1.105,"15min":1.105},"last":0.55200000000000005}]},{"osd":7,"last update":"Fri Jul 14 19:15:55 2023","interfaces":[{"interface":"back","average":{"1min":0.77400000000000002,"5min":0.72499999999999998,"15min":0.71699999999999997},"min":{"1min":0.55700000000000005,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":1.0629999999999999,"5min":1.0629999999999999,"15min":1.0629999999999999},"last":0.81299999999999994},{"interface":"front","average":{"1min":0.68700000000000006,"5min":0.69999999999999996,"15min":0.70199999999999996},"min":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":0.99299999999999999,"5min":1.0229999999999999,"15min":1.0229999999999999},"last":0.77200000000000002}]},{"osd":8,"last update":"Fri Jul 14 19:15:14 2023","interfaces":[{"interface":"back","average":{"1min":0.69299999999999995,"5min":0.69299999999999995,"15min":0.69299999999999995},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.91100000000000003,"5min":0.91100000000000003,"15min":0.91100000000000003},"last":0.58299999999999996},{"interface":"front","average":{"1min":0.84599999999999997,"5min":0.84599999999999997,"15min":0.84599999999999997},"min":{"1min":0.623,"5min":0.623,"15min":0.623},"max":{"1min":1.097,"5min":1.097,"15min":1.097},"last":0.70299999999999996}]},{"osd":9,"last update":"Fri Jul 14 19:15:37 2023","interfaces":[{"interface":"back","average":{"1min":0.73099999999999998,"5min":0.73099999999999998,"15min":0.73099999999999998},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":1.111,"5min":1.111,"15min":1.111},"last":0.56699999999999995},{"interface":"front","average":{"1min":0.70399999999999996,"5min":0.70399999999999996,"15min":0.70399999999999996},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.98099999999999998,"5min":0.98099999999999998,"15min":0.98099999999999998},"last":0.64900000000000002}]},{"osd":10,"last update":"Fri Jul 14 19:15:59 2023","interfaces":[{"interface":"back","average":{"1min":0.82799999999999996,"5min":0.82799999999999996,"15min":0.82799999999999996},"min":{"1min":0.66200000000000003,"5min":0.66200000000000003,"15min":0.66200000000000003},"max":{"1min":1.1379999999999999,"5min":1.1379999999999999,"15min":1.1379999999999999},"last":0.745},{"interface":"front","average":{"1min":0.77800000000000002,"5min":0.77800000000000002,"15min":0.77800000000000002},"min":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"max":{"1min":0.99199999999999999,"5min":0.99199999999999999,"15min":0.99199999999999999},"last":0.72699999999999998}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69399999999999995}]}]},{"osd":2,"up_from":18,"seq":77309411379,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27900,"kb_used_data":884,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717252,"statfs":{"total":95995035648,"available":95966466048,"internally_reserved":0,"allocated":905216,"data_stored":418786,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jul 14 19:15:08 2023","interfaces":[{"interface":"back","average":{"1min":0.52800000000000002,"5min":0.48799999999999999,"15min":0.47899999999999998},"min":{"1min":0.32600000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.76600000000000001,"5min":0.76600000000000001,"15min":0.76600000000000001},"last":0.36399999999999999},{"interface":"front","average":{"1min":0.58099999999999996,"5min":0.505,"15min":0.49299999999999999},"min":{"1min":0.373,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.79500000000000004,"5min":0.79500000000000004,"15min":0.79500000000000004},"last":0.38600000000000001}]},{"osd":1,"last update":"Fri Jul 14 19:15:08 2023","interfaces":[{"interface":"back","average":{"1min":0.58699999999999997,"5min":0.505,"15min":0.48499999999999999},"min":{"1min":0.38400000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.84599999999999997,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.56699999999999995},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.51500000000000001,"15min":0.497},"min":{"1min":0.29199999999999998,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":0.77700000000000002,"5min":0.90700000000000003,"15min":0.90700000000000003},"last":0.71299999999999997}]},{"osd":3,"last update":"Fri Jul 14 19:15:27 2023","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.53200000000000003,"15min":0.51300000000000001},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.74399999999999999,"5min":0.77700000000000002,"15min":0.77700000000000002},"last":0.44},{"interface":"front","average":{"1min":0.64800000000000002,"5min":0.56599999999999995,"15min":0.54900000000000004},"min":{"1min":0.432,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.80800000000000005,"5min":0.80800000000000005,"15min":0.80800000000000005},"last":0.501}]},{"osd":4,"last update":"Fri Jul 14 19:15:53 2023","interfaces":[{"interface":"back","average":{"1min":0.57899999999999996,"5min":0.53400000000000003,"15min":0.51500000000000001},"min":{"1min":0.45200000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.78700000000000003,"5min":0.78700000000000003,"15min":0.78700000000000003},"last":0.55100000000000005},{"interface":"front","average":{"1min":0.60599999999999998,"5min":0.54700000000000004,"15min":0.52700000000000002},"min":{"1min":0.436,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.81299999999999994,"5min":0.81299999999999994,"15min":0.81299999999999994},"last":0.69499999999999995}]},{"osd":5,"last update":"Fri Jul 14 19:15:13 2023","interfaces":[{"interface":"back","average":{"1min":0.432,"5min":0.40600000000000003,"15min":0.40200000000000002},"min":{"1min":0.32600000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.59499999999999997,"5min":0.59499999999999997,"15min":0.59499999999999997},"last":0.46600000000000003},{"interface":"front","average":{"1min":0.48799999999999999,"5min":0.432,"15min":0.42299999999999999},"min":{"1min":0.313,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.82599999999999996,"5min":0.82599999999999996,"15min":0.82599999999999996},"last":0.32100000000000001}]},{"osd":6,"last update":"Fri Jul 14 19:15:33 2023","interfaces":[{"interface":"back","average":{"1min":0.68799999999999994,"5min":0.627,"15min":0.61699999999999999},"min":{"1min":0.501,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.97799999999999998,"5min":0.97799999999999998,"15min":0.97799999999999998},"last":0.41699999999999998},{"interface":"front","average":{"1min":0.68799999999999994,"5min":0.62,"15min":0.60799999999999998},"min":{"1min":0.45800000000000002,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.97299999999999998,"5min":0.97299999999999998,"15min":0.97299999999999998},"last":0.67000000000000004}]},{"osd":7,"last update":"Fri Jul 14 19:15:53 2023","interfaces":[{"interface":"back","average":{"1min":0.64700000000000002,"5min":0.65100000000000002,"15min":0.65200000000000002},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.96299999999999997,"5min":0.96299999999999997,"15min":0.96299999999999997},"last":0.56999999999999995},{"interface":"front","average":{"1min":0.67100000000000004,"5min":0.66700000000000004,"15min":0.66700000000000004},"min":{"1min":0.48799999999999999,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.94099999999999995,"5min":0.94099999999999995,"15min":0.94099999999999995},"last":0.628}]},{"osd":8,"last update":"Fri Jul 14 19:15:13 2023","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.58299999999999996,"15min":0.58299999999999996},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.90600000000000003,"5min":0.90600000000000003,"15min":0.90600000000000003},"last":0.57899999999999996},{"interface":"front","average":{"1min":0.59399999999999997,"5min":0.59399999999999997,"15min":0.59399999999999997},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.90800000000000003,"5min":0.90800000000000003,"15min":0.90800000000000003},"last":0.60999999999999999}]},{"osd":9,"last update":"Fri Jul 14 19:15:39 2023","interfaces":[{"interface":"back","average":{"1min":0.73299999999999998,"5min":0.73299999999999998,"15min":0.73299999999999998},"min":{"1min":0.56299999999999994,"5min":0.56299999999999994,"15min":0.56299999999999994},"max":{"1min":1.046,"5min":1.046,"15min":1.046},"last":0.52900000000000003},{"interface":"front","average":{"1min":0.72799999999999998,"5min":0.72799999999999998,"15min":0.72799999999999998},"min":{"1min":0.49199999999999999,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":1.0289999999999999,"5min":1.0289999999999999,"15min":1.0289999999999999},"last":0.51600000000000001}]},{"osd":10,"last update":"Fri Jul 14 19:15:56 2023","interfaces":[{"interface":"back","average":{"1min":0.63900000000000001,"5min":0.63900000000000001,"15min":0.63900000000000001},"min":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":0.88500000000000001,"5min":0.88500000000000001,"15min":0.88500000000000001},"last":0.625},{"interface":"front","average":{"1min":0.71799999999999997,"5min":0.71799999999999997,"15min":0.71799999999999997},"min":{"1min":0.56200000000000006,"5min":0.56200000000000006,"15min":0.56200000000000006},"max":{"1min":1.073,"5min":1.073,"15min":1.073},"last":0.65100000000000002}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59499999999999997}]}]},{"osd":3,"up_from":25,"seq":107374182446,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27872,"kb_used_data":852,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717280,"statfs":{"total":95995035648,"available":95966494720,"internally_reserved":0,"allocated":872448,"data_stored":415693,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jul 14 19:15:29 2023","interfaces":[{"interface":"back","average":{"1min":0.40600000000000003,"5min":0.36199999999999999,"15min":0.35099999999999998},"min":{"1min":0.27400000000000002,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":0.59599999999999997,"5min":0.59599999999999997,"15min":0.59599999999999997},"last":0.35299999999999998},{"interface":"front","average":{"1min":0.44500000000000001,"5min":0.39200000000000002,"15min":0.378},"min":{"1min":0.28699999999999998,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.85899999999999999,"5min":0.85899999999999999,"15min":0.85899999999999999},"last":0.375}]},{"osd":1,"last update":"Fri Jul 14 19:15:29 2023","interfaces":[{"interface":"back","average":{"1min":0.64200000000000002,"5min":0.55700000000000005,"15min":0.53000000000000003},"min":{"1min":0.42399999999999999,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.94699999999999995,"5min":0.94699999999999995,"15min":0.94699999999999995},"last":0.59299999999999997},{"interface":"front","average":{"1min":0.63500000000000001,"5min":0.56999999999999995,"15min":0.55200000000000005},"min":{"1min":0.41799999999999998,"5min":0.372,"15min":0.372},"max":{"1min":0.79100000000000004,"5min":0.79100000000000004,"15min":0.79100000000000004},"last":0.54400000000000004}]},{"osd":2,"last update":"Fri Jul 14 19:15:29 2023","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.56999999999999995,"15min":0.55800000000000005},"min":{"1min":0.373,"5min":0.372,"15min":0.372},"max":{"1min":0.77500000000000002,"5min":0.871,"15min":0.871},"last":0.48599999999999999},{"interface":"front","average":{"1min":0.63300000000000001,"5min":0.57399999999999995,"15min":0.55600000000000005},"min":{"1min":0.434,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.93700000000000006,"5min":0.93700000000000006,"15min":0.93700000000000006},"last":0.502}]},{"osd":4,"last update":"Fri Jul 14 19:15:56 2023","interfaces":[{"interface":"back","average":{"1min":0.64500000000000002,"5min":0.58399999999999996,"15min":0.56200000000000006},"min":{"1min":0.47499999999999998,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.98599999999999999,"5min":0.98599999999999999,"15min":0.98599999999999999},"last":0.46100000000000002},{"interface":"front","average":{"1min":0.66800000000000004,"5min":0.60199999999999998,"15min":0.58999999999999997},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":0.86799999999999999,"5min":0.86799999999999999,"15min":0.86799999999999999},"last":0.55400000000000005}]},{"osd":5,"last update":"Fri Jul 14 19:15:15 2023","interfaces":[{"interface":"back","average":{"1min":0.64700000000000002,"5min":0.58899999999999997,"15min":0.57899999999999996},"min":{"1min":0.38600000000000001,"5min":0.374,"15min":0.374},"max":{"1min":0.84499999999999997,"5min":0.84499999999999997,"15min":0.84499999999999997},"last":0.51900000000000002},{"interface":"front","average":{"1min":0.63500000000000001,"5min":0.61299999999999999,"15min":0.60899999999999999},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.91700000000000004,"5min":0.91700000000000004,"15min":0.91700000000000004},"last":0.52900000000000003}]},{"osd":6,"last update":"Fri Jul 14 19:15:36 2023","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.51200000000000001,"15min":0.50600000000000001},"min":{"1min":0.35499999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.86399999999999999,"5min":0.86399999999999999,"15min":0.86399999999999999},"last":0.67600000000000005},{"interface":"front","average":{"1min":0.66300000000000003,"5min":0.56699999999999995,"15min":0.55100000000000005},"min":{"1min":0.378,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.99199999999999999,"5min":0.99199999999999999,"15min":0.99199999999999999},"last":0.439}]},{"osd":7,"last update":"Fri Jul 14 19:15:56 2023","interfaces":[{"interface":"back","average":{"1min":0.73599999999999999,"5min":0.71699999999999997,"15min":0.71399999999999997},"min":{"1min":0.47599999999999998,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":1.05,"5min":1.1080000000000001,"15min":1.1080000000000001},"last":0.57899999999999996},{"interface":"front","average":{"1min":0.70299999999999996,"5min":0.67600000000000005,"15min":0.67200000000000004},"min":{"1min":0.46500000000000002,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":0.89400000000000002,"5min":0.95599999999999996,"15min":0.95599999999999996},"last":0.61699999999999999}]},{"osd":8,"last update":"Fri Jul 14 19:15:15 2023","interfaces":[{"interface":"back","average":{"1min":0.747,"5min":0.747,"15min":0.747},"min":{"1min":0.55800000000000005,"5min":0.55800000000000005,"15min":0.55800000000000005},"max":{"1min":1.0640000000000001,"5min":1.0640000000000001,"15min":1.0640000000000001},"last":0.66700000000000004},{"interface":"front","average":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"min":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":0.98499999999999999,"5min":0.98499999999999999,"15min":0.98499999999999999},"last":0.68500000000000005}]},{"osd":9,"last update":"Fri Jul 14 19:15:36 2023","interfaces":[{"interface":"back","average":{"1min":0.69799999999999995,"5min":0.69799999999999995,"15min":0.69799999999999995},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":1.048,"5min":1.048,"15min":1.048},"last":0.41799999999999998},{"interface":"front","average":{"1min":0.77300000000000002,"5min":0.77300000000000002,"15min":0.77300000000000002},"min":{"1min":0.49099999999999999,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":1.0289999999999999,"5min":1.0289999999999999,"15min":1.0289999999999999},"last":0.60499999999999998}]},{"osd":10,"last update":"Fri Jul 14 19:15:56 2023","interfaces":[{"interface":"back","average":{"1min":0.81299999999999994,"5min":0.81299999999999994,"15min":0.81299999999999994},"min":{"1min":0.58199999999999996,"5min":0.58199999999999996,"15min":0.58199999999999996},"max":{"1min":1.0469999999999999,"5min":1.0469999999999999,"15min":1.0469999999999999},"last":0.64400000000000002},{"interface":"front","average":{"1min":0.77200000000000002,"5min":0.77200000000000002,"15min":0.77200000000000002},"min":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":1.0069999999999999,"5min":1.0069999999999999,"15min":1.0069999999999999},"last":0.56799999999999995}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65500000000000003}]}]},{"osd":4,"up_from":32,"seq":137438953514,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27872,"kb_used_data":852,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717280,"statfs":{"total":95995035648,"available":95966494720,"internally_reserved":0,"allocated":872448,"data_stored":415942,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jul 14 19:14:53 2023","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.503,"15min":0.495},"min":{"1min":0.27200000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.81999999999999995,"5min":0.81999999999999995,"15min":0.81999999999999995},"last":0.56299999999999994},{"interface":"front","average":{"1min":0.58199999999999996,"5min":0.53000000000000003,"15min":0.52200000000000002},"min":{"1min":0.378,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.84599999999999997,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.58299999999999996}]},{"osd":1,"last update":"Fri Jul 14 19:15:56 2023","interfaces":[{"interface":"back","average":{"1min":0.46600000000000003,"5min":0.39000000000000001,"15min":0.36399999999999999},"min":{"1min":0.30499999999999999,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.78400000000000003,"5min":1.0049999999999999,"15min":1.0049999999999999},"last":0.30399999999999999},{"interface":"front","average":{"1min":0.46600000000000003,"5min":0.39700000000000002,"15min":0.36899999999999999},"min":{"1min":0.26900000000000002,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":0.71699999999999997,"5min":0.91500000000000004,"15min":0.91500000000000004},"last":0.32400000000000001}]},{"osd":2,"last update":"Fri Jul 14 19:15:56 2023","interfaces":[{"interface":"back","average":{"1min":0.55100000000000005,"5min":0.54600000000000004,"15min":0.53400000000000003},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.69099999999999995,"5min":0.98499999999999999,"15min":0.98499999999999999},"last":0.495},{"interface":"front","average":{"1min":0.66000000000000003,"5min":0.59899999999999998,"15min":0.56899999999999995},"min":{"1min":0.5,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.88700000000000001,"5min":0.92900000000000005,"15min":0.92900000000000005},"last":0.441}]},{"osd":3,"last update":"Fri Jul 14 19:15:56 2023","interfaces":[{"interface":"back","average":{"1min":0.63700000000000001,"5min":0.55000000000000004,"15min":0.52500000000000002},"min":{"1min":0.41299999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.89300000000000002,"5min":0.89300000000000002,"15min":0.89300000000000002},"last":0.59199999999999997},{"interface":"front","average":{"1min":0.56100000000000005,"5min":0.55000000000000004,"15min":0.53800000000000003},"min":{"1min":0.36899999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.73499999999999999,"5min":0.94199999999999995,"15min":0.94199999999999995},"last":0.58299999999999996}]},{"osd":5,"last update":"Fri Jul 14 19:15:14 2023","interfaces":[{"interface":"back","average":{"1min":0.64700000000000002,"5min":0.57899999999999996,"15min":0.56699999999999995},"min":{"1min":0.433,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":0.87,"5min":0.87,"15min":0.87},"last":0.51400000000000001},{"interface":"front","average":{"1min":0.627,"5min":0.57899999999999996,"15min":0.57099999999999995},"min":{"1min":0.42299999999999999,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.86699999999999999,"5min":0.86699999999999999,"15min":0.86699999999999999},"last":0.52700000000000002}]},{"osd":6,"last update":"Fri Jul 14 19:15:30 2023","interfaces":[{"interface":"back","average":{"1min":0.64500000000000002,"5min":0.625,"15min":0.622},"min":{"1min":0.44400000000000001,"5min":0.38,"15min":0.38},"max":{"1min":0.94299999999999995,"5min":0.94299999999999995,"15min":0.94299999999999995},"last":0.57199999999999995},{"interface":"front","average":{"1min":0.66500000000000004,"5min":0.60799999999999998,"15min":0.59799999999999998},"min":{"1min":0.42699999999999999,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":1.0049999999999999,"5min":1.0049999999999999,"15min":1.0049999999999999},"last":0.47999999999999998}]},{"osd":7,"last update":"Fri Jul 14 19:15:56 2023","interfaces":[{"interface":"back","average":{"1min":0.68100000000000005,"5min":0.60499999999999998,"15min":0.59299999999999997},"min":{"1min":0.44500000000000001,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.97599999999999998,"5min":1.0529999999999999,"15min":1.0529999999999999},"last":0.53900000000000003},{"interface":"front","average":{"1min":0.70899999999999996,"5min":0.59599999999999997,"15min":0.57699999999999996},"min":{"1min":0.47599999999999998,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.93100000000000005,"5min":0.93100000000000005,"15min":0.93100000000000005},"last":0.46100000000000002}]},{"osd":8,"last update":"Fri Jul 14 19:15:14 2023","interfaces":[{"interface":"back","average":{"1min":0.63500000000000001,"5min":0.63500000000000001,"15min":0.63500000000000001},"min":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":1.0169999999999999,"5min":1.0169999999999999,"15min":1.0169999999999999},"last":0.55200000000000005},{"interface":"front","average":{"1min":0.76500000000000001,"5min":0.76500000000000001,"15min":0.76500000000000001},"min":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":0.98499999999999999,"5min":0.98499999999999999,"15min":0.98499999999999999},"last":0.56299999999999994}]},{"osd":9,"last update":"Fri Jul 14 19:15:35 2023","interfaces":[{"interface":"back","average":{"1min":0.77400000000000002,"5min":0.77400000000000002,"15min":0.77400000000000002},"min":{"1min":0.57399999999999995,"5min":0.57399999999999995,"15min":0.57399999999999995},"max":{"1min":0.99299999999999999,"5min":0.99299999999999999,"15min":0.99299999999999999},"last":0.64200000000000002},{"interface":"front","average":{"1min":0.76800000000000002,"5min":0.76800000000000002,"15min":0.76800000000000002},"min":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":1.05,"5min":1.05,"15min":1.05},"last":0.63600000000000001}]},{"osd":10,"last update":"Fri Jul 14 19:15:56 2023","interfaces":[{"interface":"back","average":{"1min":0.82199999999999995,"5min":0.82199999999999995,"15min":0.82199999999999995},"min":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.56000000000000005},"max":{"1min":1.0669999999999999,"5min":1.0669999999999999,"15min":1.0669999999999999},"last":0.62},{"interface":"front","average":{"1min":0.80900000000000005,"5min":0.80900000000000005,"15min":0.80900000000000005},"min":{"1min":0.54900000000000004,"5min":0.54900000000000004,"15min":0.54900000000000004},"max":{"1min":1.028,"5min":1.028,"15min":1.028},"last":0.61199999999999999}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60099999999999998}]}]},{"osd":5,"up_from":37,"seq":158913789990,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27940,"kb_used_data":924,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717212,"statfs":{"total":95995035648,"available":95966425088,"internally_reserved":0,"allocated":946176,"data_stored":1007716,"data_compressed":35714,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jul 14 19:15:12 2023","interfaces":[{"interface":"back","average":{"1min":0.628,"5min":0.53200000000000003,"15min":0.51600000000000001},"min":{"1min":0.41299999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":1.028,"5min":1.028,"15min":1.028},"last":0.28399999999999997},{"interface":"front","average":{"1min":0.63,"5min":0.56200000000000006,"15min":0.55000000000000004},"min":{"1min":0.40999999999999998,"5min":0.307,"15min":0.307},"max":{"1min":0.93300000000000005,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.33700000000000002}]},{"osd":1,"last update":"Fri Jul 14 19:15:12 2023","interfaces":[{"interface":"back","average":{"1min":0.59799999999999998,"5min":0.51100000000000001,"15min":0.497},"min":{"1min":0.36699999999999999,"5min":0.307,"15min":0.307},"max":{"1min":1.2410000000000001,"5min":1.2410000000000001,"15min":1.2410000000000001},"last":0.57599999999999996},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.48299999999999998,"15min":0.47199999999999998},"min":{"1min":0.39500000000000002,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.85799999999999998,"5min":0.85799999999999998,"15min":0.85799999999999998},"last":0.63}]},{"osd":2,"last update":"Fri Jul 14 19:15:12 2023","interfaces":[{"interface":"back","average":{"1min":0.42799999999999999,"5min":0.41199999999999998,"15min":0.40999999999999998},"min":{"1min":0.27200000000000002,"5min":0.246,"15min":0.246},"max":{"1min":0.66500000000000004,"5min":0.68400000000000005,"15min":0.68400000000000005},"last":0.38200000000000001},{"interface":"front","average":{"1min":0.45300000000000001,"5min":0.40899999999999997,"15min":0.40200000000000002},"min":{"1min":0.309,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.73999999999999999,"5min":0.73999999999999999,"15min":0.73999999999999999},"last":0.51400000000000001}]},{"osd":3,"last update":"Fri Jul 14 19:15:12 2023","interfaces":[{"interface":"back","average":{"1min":0.65000000000000002,"5min":0.58999999999999997,"15min":0.57999999999999996},"min":{"1min":0.41399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.98199999999999998,"5min":0.98199999999999998,"15min":0.98199999999999998},"last":0.441},{"interface":"front","average":{"1min":0.65200000000000002,"5min":0.60099999999999998,"15min":0.59299999999999997},"min":{"1min":0.44900000000000001,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":0.95899999999999996,"5min":0.95899999999999996,"15min":0.95899999999999996},"last":0.48299999999999998}]},{"osd":4,"last update":"Fri Jul 14 19:15:12 2023","interfaces":[{"interface":"back","average":{"1min":0.59799999999999998,"5min":0.54400000000000004,"15min":0.53500000000000003},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.79400000000000004,"5min":0.79400000000000004,"15min":0.79400000000000004},"last":0.59599999999999997},{"interface":"front","average":{"1min":0.626,"5min":0.57799999999999996,"15min":0.56999999999999995},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.875,"5min":0.875,"15min":0.875},"last":0.61099999999999999}]},{"osd":6,"last update":"Fri Jul 14 19:15:35 2023","interfaces":[{"interface":"back","average":{"1min":0.65700000000000003,"5min":0.61899999999999999,"15min":0.61299999999999999},"min":{"1min":0.437,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.91000000000000003,"5min":0.91000000000000003,"15min":0.91000000000000003},"last":0.55500000000000005},{"interface":"front","average":{"1min":0.70699999999999996,"5min":0.64000000000000001,"15min":0.629},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":1.0800000000000001,"5min":1.0800000000000001,"15min":1.0800000000000001},"last":0.41999999999999998}]},{"osd":7,"last update":"Fri Jul 14 19:15:53 2023","interfaces":[{"interface":"back","average":{"1min":0.64500000000000002,"5min":0.66000000000000003,"15min":0.66200000000000003},"min":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":0.84599999999999997,"5min":0.91400000000000003,"15min":0.91400000000000003},"last":0.72999999999999998},{"interface":"front","average":{"1min":0.66300000000000003,"5min":0.68000000000000005,"15min":0.68300000000000005},"min":{"1min":0.45700000000000002,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":0.83499999999999996,"5min":1.1240000000000001,"15min":1.1240000000000001},"last":0.71699999999999997}]},{"osd":8,"last update":"Fri Jul 14 19:15:14 2023","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.57999999999999996,"15min":0.57999999999999996},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.85199999999999998,"5min":0.85199999999999998,"15min":0.85199999999999998},"last":0.46000000000000002},{"interface":"front","average":{"1min":0.66300000000000003,"5min":0.66300000000000003,"15min":0.66300000000000003},"min":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":0.94199999999999995,"5min":0.94199999999999995,"15min":0.94199999999999995},"last":0.70199999999999996}]},{"osd":9,"last update":"Fri Jul 14 19:15:35 2023","interfaces":[{"interface":"back","average":{"1min":0.73599999999999999,"5min":0.73599999999999999,"15min":0.73599999999999999},"min":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"max":{"1min":1.014,"5min":1.014,"15min":1.014},"last":0.66500000000000004},{"interface":"front","average":{"1min":0.76000000000000001,"5min":0.76000000000000001,"15min":0.76000000000000001},"min":{"1min":0.51500000000000001,"5min":0.51500000000000001,"15min":0.51500000000000001},"max":{"1min":1.196,"5min":1.196,"15min":1.196},"last":0.64800000000000002}]},{"osd":10,"last update":"Fri Jul 14 19:15:59 2023","interfaces":[{"interface":"back","average":{"1min":0.70599999999999996,"5min":0.70599999999999996,"15min":0.70599999999999996},"min":{"1min":0.56200000000000006,"5min":0.56200000000000006,"15min":0.56200000000000006},"max":{"1min":0.89300000000000002,"5min":0.89300000000000002,"15min":0.89300000000000002},"last":0.753},{"interface":"front","average":{"1min":0.70999999999999996,"5min":0.70999999999999996,"15min":0.70999999999999996},"min":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":0.89000000000000001,"5min":0.89000000000000001,"15min":0.89000000000000001},"last":0.68300000000000005}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34079,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34079,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34079,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35714,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35714,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35714,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35714,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35714,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1564,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":2866,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1428,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1156,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":908,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1758,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-07-14T19:16:06.389 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-07-14T19:16:06.548 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:06.543+0000 7f2ed5674700 1 -- 172.21.15.40:0/3113678577 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2ed01006d0 msgr2=0x7f2ed0106f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:16:06.548 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:06.543+0000 7f2ed5674700 1 --2- 172.21.15.40:0/3113678577 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2ed01006d0 0x7f2ed0106f90 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f2ec800b0a0 tx=0x7f2ec800b3b0 comp rx=0 tx=0).stop 2023-07-14T19:16:06.549 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:06.543+0000 7f2ed5674700 1 -- 172.21.15.40:0/3113678577 shutdown_connections 2023-07-14T19:16:06.549 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:06.543+0000 7f2ed5674700 1 --2- 172.21.15.40:0/3113678577 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2ed00feb50 0x7f2ed00fef60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:06.549 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:06.543+0000 7f2ed5674700 1 --2- 172.21.15.40:0/3113678577 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2ed01006d0 0x7f2ed0106f90 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:06.550 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:06.543+0000 7f2ed5674700 1 --2- 172.21.15.40:0/3113678577 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2ed00ffd40 0x7f2ed0100190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:06.550 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:06.543+0000 7f2ed5674700 1 -- 172.21.15.40:0/3113678577 >> 172.21.15.40:0/3113678577 conn(0x7f2ed00fa190 msgr2=0x7f2ed00fc5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:16:06.550 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:06.543+0000 7f2ed5674700 1 -- 172.21.15.40:0/3113678577 shutdown_connections 2023-07-14T19:16:06.550 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:06.543+0000 7f2ed5674700 1 -- 172.21.15.40:0/3113678577 wait complete. 2023-07-14T19:16:06.550 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:06.543+0000 7f2ed5674700 1 Processor -- start 2023-07-14T19:16:06.551 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:06.544+0000 7f2ed5674700 1 -- start start 2023-07-14T19:16:06.551 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:06.544+0000 7f2ed5674700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2ed00feb50 0x7f2ed01948a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:16:06.551 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:06.544+0000 7f2ed5674700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2ed00ffd40 0x7f2ed0194de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:16:06.551 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:06.544+0000 7f2ed5674700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2ed01006d0 0x7f2ed019be50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:16:06.551 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:06.544+0000 7f2ed5674700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ed019c420 con 0x7f2ed00feb50 2023-07-14T19:16:06.551 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:06.544+0000 7f2ed5674700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ed019c560 con 0x7f2ed01006d0 2023-07-14T19:16:06.551 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:06.544+0000 7f2ed5674700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ed019c6f0 con 0x7f2ed00ffd40 2023-07-14T19:16:06.551 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:06.544+0000 7f2eceffd700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2ed00feb50 0x7f2ed01948a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:16:06.552 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:06.544+0000 7f2eceffd700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2ed00feb50 0x7f2ed01948a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:33076/0 (socket says 172.21.15.40:33076) 2023-07-14T19:16:06.552 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:06.544+0000 7f2eceffd700 1 -- 172.21.15.40:0/207490563 learned_addr learned my addr 172.21.15.40:0/207490563 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-07-14T19:16:06.552 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:06.544+0000 7f2ece7fc700 1 --2- 172.21.15.40:0/207490563 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2ed00ffd40 0x7f2ed0194de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:16:06.552 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:06.544+0000 7f2ecf7fe700 1 --2- 172.21.15.40:0/207490563 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2ed01006d0 0x7f2ed019be50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:16:06.552 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:06.544+0000 7f2eceffd700 1 -- 172.21.15.40:0/207490563 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2ed00ffd40 msgr2=0x7f2ed0194de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:16:06.552 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:06.544+0000 7f2eceffd700 1 --2- 172.21.15.40:0/207490563 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2ed00ffd40 0x7f2ed0194de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:06.552 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:06.544+0000 7f2eceffd700 1 -- 172.21.15.40:0/207490563 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2ed01006d0 msgr2=0x7f2ed019be50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:16:06.553 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:06.544+0000 7f2eceffd700 1 --2- 172.21.15.40:0/207490563 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2ed01006d0 0x7f2ed019be50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:06.553 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:06.544+0000 7f2eceffd700 1 -- 172.21.15.40:0/207490563 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ec8009d00 con 0x7f2ed00feb50 2023-07-14T19:16:06.553 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:06.544+0000 7f2eceffd700 1 --2- 172.21.15.40:0/207490563 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2ed00feb50 0x7f2ed01948a0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f2ec000d980 tx=0x7f2ec000dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:16:06.553 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:06.545+0000 7f2ec7fff700 1 -- 172.21.15.40:0/207490563 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ec0004020 con 0x7f2ed00feb50 2023-07-14T19:16:06.553 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:06.545+0000 7f2ec7fff700 1 -- 172.21.15.40:0/207490563 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f2ec0009730 con 0x7f2ed00feb50 2023-07-14T19:16:06.553 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:06.545+0000 7f2ed5674700 1 -- 172.21.15.40:0/207490563 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ed019c9d0 con 0x7f2ed00feb50 2023-07-14T19:16:06.553 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:06.545+0000 7f2ec7fff700 1 -- 172.21.15.40:0/207490563 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ec0003a80 con 0x7f2ed00feb50 2023-07-14T19:16:06.554 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:06.545+0000 7f2ed5674700 1 -- 172.21.15.40:0/207490563 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ed019cee0 con 0x7f2ed00feb50 2023-07-14T19:16:06.554 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:06.546+0000 7f2ed5674700 1 -- 172.21.15.40:0/207490563 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2ed010a330 con 0x7f2ed00feb50 2023-07-14T19:16:06.554 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:06.547+0000 7f2ec7fff700 1 -- 172.21.15.40:0/207490563 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7f2ec00098a0 con 0x7f2ed00feb50 2023-07-14T19:16:06.555 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:06.547+0000 7f2ec7fff700 1 --2- 172.21.15.40:0/207490563 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f2ebc0a1220 0x7f2ebc0a36d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:16:06.555 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:06.547+0000 7f2ec7fff700 1 -- 172.21.15.40:0/207490563 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f2ec00cc930 con 0x7f2ed00feb50 2023-07-14T19:16:06.555 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:06.547+0000 7f2ece7fc700 1 --2- 172.21.15.40:0/207490563 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f2ebc0a1220 0x7f2ebc0a36d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:16:06.555 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:06.548+0000 7f2ece7fc700 1 --2- 172.21.15.40:0/207490563 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f2ebc0a1220 0x7f2ebc0a36d0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f2eb8009c60 tx=0x7f2eb8009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:16:06.555 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:06.551+0000 7f2ec7fff700 1 -- 172.21.15.40:0/207490563 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f2ec00960d0 con 0x7f2ed00feb50 2023-07-14T19:16:06.721 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:16:06 smithi040 ceph-mon[96518]: pgmap v258: 97 pgs: 97 active+clean; 581 KiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s wr, 14 op/s 2023-07-14T19:16:06.721 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:16:06 smithi040 ceph-mon[96518]: from='client.14706 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T19:16:06.721 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:16:06 smithi040 ceph-mon[96518]: from='client.14712 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T19:16:06.721 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:16:06 smithi040 ceph-mon[96518]: mds.? [v2:172.21.15.130:6834/3088490808,v1:172.21.15.130:6835/3088490808] up:active 2023-07-14T19:16:06.722 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:16:06 smithi040 ceph-mon[96518]: fsmap cephfs:5 7 up:standby 5 up:active 2023-07-14T19:16:06.722 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:16:06 smithi040 ceph-mon[96518]: mgrmap e17: x(active, since 6m), standbys: z, y 2023-07-14T19:16:06.722 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:16:06 smithi040 ceph-mon[96518]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2023-07-14T19:16:06.722 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:16:06 smithi040 ceph-mon[96518]: from='client.? 172.21.15.40:0/208544988' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T19:16:06.722 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:16:06 smithi040 ceph-mon[96518]: from='client.? 172.21.15.40:0/128476494' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T19:16:06.750 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:06.748+0000 7f2ed5674700 1 -- 172.21.15.40:0/207490563 --> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f2ed0060a80 con 0x7f2ebc0a1220 2023-07-14T19:16:06.762 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:06.760+0000 7f2ec7fff700 1 -- 172.21.15.40:0/207490563 <== mgr.14152 v2:172.21.15.40:6800/1693027118 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+335894 (secure 0 0 0) 0x7f2ed0060a80 con 0x7f2ebc0a1220 2023-07-14T19:16:06.765 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-07-14T19:16:06.767 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:06.764+0000 7f2ed5674700 1 -- 172.21.15.40:0/207490563 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f2ebc0a1220 msgr2=0x7f2ebc0a36d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:16:06.767 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:06.764+0000 7f2ed5674700 1 --2- 172.21.15.40:0/207490563 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f2ebc0a1220 0x7f2ebc0a36d0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f2eb8009c60 tx=0x7f2eb8009260 comp rx=0 tx=0).stop 2023-07-14T19:16:06.767 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:06.764+0000 7f2ed5674700 1 -- 172.21.15.40:0/207490563 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2ed00feb50 msgr2=0x7f2ed01948a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:16:06.768 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:06.764+0000 7f2ed5674700 1 --2- 172.21.15.40:0/207490563 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2ed00feb50 0x7f2ed01948a0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f2ec000d980 tx=0x7f2ec000dc90 comp rx=0 tx=0).stop 2023-07-14T19:16:06.768 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:06.764+0000 7f2ed5674700 1 -- 172.21.15.40:0/207490563 shutdown_connections 2023-07-14T19:16:06.768 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:06.764+0000 7f2ed5674700 1 --2- 172.21.15.40:0/207490563 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f2ebc0a1220 0x7f2ebc0a36d0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:06.768 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:06.764+0000 7f2ed5674700 1 --2- 172.21.15.40:0/207490563 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2ed01006d0 0x7f2ed019be50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:06.768 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:06.764+0000 7f2ed5674700 1 --2- 172.21.15.40:0/207490563 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2ed00feb50 0x7f2ed01948a0 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:06.768 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:06.764+0000 7f2ed5674700 1 --2- 172.21.15.40:0/207490563 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2ed00ffd40 0x7f2ed0194de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:06.768 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:06.764+0000 7f2ed5674700 1 -- 172.21.15.40:0/207490563 >> 172.21.15.40:0/207490563 conn(0x7f2ed00fa190 msgr2=0x7f2ed00fc2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:16:06.769 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:06.764+0000 7f2ed5674700 1 -- 172.21.15.40:0/207490563 shutdown_connections 2023-07-14T19:16:06.769 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:06.764+0000 7f2ed5674700 1 -- 172.21.15.40:0/207490563 wait complete. 2023-07-14T19:16:06.769 INFO:teuthology.orchestra.run.smithi040.stderr:dumped all 2023-07-14T19:16:06.778 INFO:teuthology.orchestra.run.smithi040.stdout:{"pg_ready":true,"pg_map":{"version":258,"stamp":"2023-07-14T19:16:04.809979+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":594602,"num_objects":53,"num_object_clones":0,"num_object_copies":159,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":293,"num_write_kb":2971,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":36,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":233,"ondisk_log_size":233,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":251,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":334696,"kb_used_data":10468,"kb_used_omap":20,"kb_used_meta":324075,"kb_avail":1124607128,"statfs":{"total":1151940427776,"available":1151597699072,"internally_reserved":0,"allocated":10719232,"data_stored":6770996,"data_compressed":107142,"data_compressed_allocated":135168,"data_compressed_original":1769472,"omap_allocated":21372,"internal_metadata":331852932},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":4234,"num_objects":51,"num_object_clones":0,"num_object_copies":153,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":51,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":112,"num_write_kb":33,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":36,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.585197"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-07-14T19:15:59.938963+0000","last_change":"2023-07-14T19:15:58.250082+0000","last_active":"2023-07-14T19:15:59.938963+0000","last_peered":"2023-07-14T19:15:59.938963+0000","last_clean":"2023-07-14T19:15:59.938963+0000","last_became_active":"2023-07-14T19:15:58.249783+0000","last_became_peered":"2023-07-14T19:15:58.249783+0000","last_unstale":"2023-07-14T19:15:59.938963+0000","last_undegraded":"2023-07-14T19:15:59.938963+0000","last_fullsized":"2023-07-14T19:15:59.938963+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_clean_scrub_stamp":"2023-07-14T19:15:57.219223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T03:38:35.618742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-07-14T19:15:59.573145+0000","last_change":"2023-07-14T19:15:58.240082+0000","last_active":"2023-07-14T19:15:59.573145+0000","last_peered":"2023-07-14T19:15:59.573145+0000","last_clean":"2023-07-14T19:15:59.573145+0000","last_became_active":"2023-07-14T19:15:58.239788+0000","last_became_peered":"2023-07-14T19:15:58.239788+0000","last_unstale":"2023-07-14T19:15:59.573145+0000","last_undegraded":"2023-07-14T19:15:59.573145+0000","last_fullsized":"2023-07-14T19:15:59.573145+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_clean_scrub_stamp":"2023-07-14T19:15:57.219223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T05:20:02.889590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-07-14T19:15:59.573092+0000","last_change":"2023-07-14T19:15:58.248010+0000","last_active":"2023-07-14T19:15:59.573092+0000","last_peered":"2023-07-14T19:15:59.573092+0000","last_clean":"2023-07-14T19:15:59.573092+0000","last_became_active":"2023-07-14T19:15:58.247755+0000","last_became_peered":"2023-07-14T19:15:58.247755+0000","last_unstale":"2023-07-14T19:15:59.573092+0000","last_undegraded":"2023-07-14T19:15:59.573092+0000","last_fullsized":"2023-07-14T19:15:59.573092+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_clean_scrub_stamp":"2023-07-14T19:15:57.219223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T19:47:43.289654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-07-14T19:15:59.229952+0000","last_change":"2023-07-14T19:15:58.246946+0000","last_active":"2023-07-14T19:15:59.229952+0000","last_peered":"2023-07-14T19:15:59.229952+0000","last_clean":"2023-07-14T19:15:59.229952+0000","last_became_active":"2023-07-14T19:15:58.246598+0000","last_became_peered":"2023-07-14T19:15:58.246598+0000","last_unstale":"2023-07-14T19:15:59.229952+0000","last_undegraded":"2023-07-14T19:15:59.229952+0000","last_fullsized":"2023-07-14T19:15:59.229952+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_clean_scrub_stamp":"2023-07-14T19:15:57.219223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T06:05:42.069121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-07-14T19:15:59.232238+0000","last_change":"2023-07-14T19:15:58.252632+0000","last_active":"2023-07-14T19:15:59.232238+0000","last_peered":"2023-07-14T19:15:59.232238+0000","last_clean":"2023-07-14T19:15:59.232238+0000","last_became_active":"2023-07-14T19:15:58.252319+0000","last_became_peered":"2023-07-14T19:15:58.252319+0000","last_unstale":"2023-07-14T19:15:59.232238+0000","last_undegraded":"2023-07-14T19:15:59.232238+0000","last_fullsized":"2023-07-14T19:15:59.232238+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_clean_scrub_stamp":"2023-07-14T19:15:57.219223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T06:07:06.243539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-07-14T19:15:59.722095+0000","last_change":"2023-07-14T19:15:58.248305+0000","last_active":"2023-07-14T19:15:59.722095+0000","last_peered":"2023-07-14T19:15:59.722095+0000","last_clean":"2023-07-14T19:15:59.722095+0000","last_became_active":"2023-07-14T19:15:58.247347+0000","last_became_peered":"2023-07-14T19:15:58.247347+0000","last_unstale":"2023-07-14T19:15:59.722095+0000","last_undegraded":"2023-07-14T19:15:59.722095+0000","last_fullsized":"2023-07-14T19:15:59.722095+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_clean_scrub_stamp":"2023-07-14T19:15:57.219223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T05:16:26.859500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-07-14T19:15:59.721610+0000","last_change":"2023-07-14T19:15:58.240871+0000","last_active":"2023-07-14T19:15:59.721610+0000","last_peered":"2023-07-14T19:15:59.721610+0000","last_clean":"2023-07-14T19:15:59.721610+0000","last_became_active":"2023-07-14T19:15:58.240574+0000","last_became_peered":"2023-07-14T19:15:58.240574+0000","last_unstale":"2023-07-14T19:15:59.721610+0000","last_undegraded":"2023-07-14T19:15:59.721610+0000","last_fullsized":"2023-07-14T19:15:59.721610+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_clean_scrub_stamp":"2023-07-14T19:15:57.219223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T00:07:11.508517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-07-14T19:15:59.721987+0000","last_change":"2023-07-14T19:15:58.241979+0000","last_active":"2023-07-14T19:15:59.721987+0000","last_peered":"2023-07-14T19:15:59.721987+0000","last_clean":"2023-07-14T19:15:59.721987+0000","last_became_active":"2023-07-14T19:15:58.241111+0000","last_became_peered":"2023-07-14T19:15:58.241111+0000","last_unstale":"2023-07-14T19:15:59.721987+0000","last_undegraded":"2023-07-14T19:15:59.721987+0000","last_fullsized":"2023-07-14T19:15:59.721987+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_clean_scrub_stamp":"2023-07-14T19:15:57.219223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T07:04:12.177825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-07-14T19:15:59.721751+0000","last_change":"2023-07-14T19:15:58.248866+0000","last_active":"2023-07-14T19:15:59.721751+0000","last_peered":"2023-07-14T19:15:59.721751+0000","last_clean":"2023-07-14T19:15:59.721751+0000","last_became_active":"2023-07-14T19:15:58.248456+0000","last_became_peered":"2023-07-14T19:15:58.248456+0000","last_unstale":"2023-07-14T19:15:59.721751+0000","last_undegraded":"2023-07-14T19:15:59.721751+0000","last_fullsized":"2023-07-14T19:15:59.721751+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_clean_scrub_stamp":"2023-07-14T19:15:57.219223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:28:00.287289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-07-14T19:15:59.722575+0000","last_change":"2023-07-14T19:15:58.248296+0000","last_active":"2023-07-14T19:15:59.722575+0000","last_peered":"2023-07-14T19:15:59.722575+0000","last_clean":"2023-07-14T19:15:59.722575+0000","last_became_active":"2023-07-14T19:15:58.247810+0000","last_became_peered":"2023-07-14T19:15:58.247810+0000","last_unstale":"2023-07-14T19:15:59.722575+0000","last_undegraded":"2023-07-14T19:15:59.722575+0000","last_fullsized":"2023-07-14T19:15:59.722575+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_clean_scrub_stamp":"2023-07-14T19:15:57.219223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T03:06:56.929583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-07-14T19:15:59.722644+0000","last_change":"2023-07-14T19:15:58.252057+0000","last_active":"2023-07-14T19:15:59.722644+0000","last_peered":"2023-07-14T19:15:59.722644+0000","last_clean":"2023-07-14T19:15:59.722644+0000","last_became_active":"2023-07-14T19:15:58.251425+0000","last_became_peered":"2023-07-14T19:15:58.251425+0000","last_unstale":"2023-07-14T19:15:59.722644+0000","last_undegraded":"2023-07-14T19:15:59.722644+0000","last_fullsized":"2023-07-14T19:15:59.722644+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_clean_scrub_stamp":"2023-07-14T19:15:57.219223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T00:23:30.017734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-07-14T19:15:59.229817+0000","last_change":"2023-07-14T19:15:58.241009+0000","last_active":"2023-07-14T19:15:59.229817+0000","last_peered":"2023-07-14T19:15:59.229817+0000","last_clean":"2023-07-14T19:15:59.229817+0000","last_became_active":"2023-07-14T19:15:58.240622+0000","last_became_peered":"2023-07-14T19:15:58.240622+0000","last_unstale":"2023-07-14T19:15:59.229817+0000","last_undegraded":"2023-07-14T19:15:59.229817+0000","last_fullsized":"2023-07-14T19:15:59.229817+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_clean_scrub_stamp":"2023-07-14T19:15:57.219223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T05:28:57.780030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-07-14T19:15:59.722108+0000","last_change":"2023-07-14T19:15:58.243377+0000","last_active":"2023-07-14T19:15:59.722108+0000","last_peered":"2023-07-14T19:15:59.722108+0000","last_clean":"2023-07-14T19:15:59.722108+0000","last_became_active":"2023-07-14T19:15:58.242225+0000","last_became_peered":"2023-07-14T19:15:58.242225+0000","last_unstale":"2023-07-14T19:15:59.722108+0000","last_undegraded":"2023-07-14T19:15:59.722108+0000","last_fullsized":"2023-07-14T19:15:59.722108+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_clean_scrub_stamp":"2023-07-14T19:15:57.219223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:21:59.269854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-07-14T19:15:59.573307+0000","last_change":"2023-07-14T19:15:58.244556+0000","last_active":"2023-07-14T19:15:59.573307+0000","last_peered":"2023-07-14T19:15:59.573307+0000","last_clean":"2023-07-14T19:15:59.573307+0000","last_became_active":"2023-07-14T19:15:58.244228+0000","last_became_peered":"2023-07-14T19:15:58.244228+0000","last_unstale":"2023-07-14T19:15:59.573307+0000","last_undegraded":"2023-07-14T19:15:59.573307+0000","last_fullsized":"2023-07-14T19:15:59.573307+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_clean_scrub_stamp":"2023-07-14T19:15:57.219223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:11:22.589361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-07-14T19:15:59.572875+0000","last_change":"2023-07-14T19:15:58.250176+0000","last_active":"2023-07-14T19:15:59.572875+0000","last_peered":"2023-07-14T19:15:59.572875+0000","last_clean":"2023-07-14T19:15:59.572875+0000","last_became_active":"2023-07-14T19:15:58.249708+0000","last_became_peered":"2023-07-14T19:15:58.249708+0000","last_unstale":"2023-07-14T19:15:59.572875+0000","last_undegraded":"2023-07-14T19:15:59.572875+0000","last_fullsized":"2023-07-14T19:15:59.572875+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_clean_scrub_stamp":"2023-07-14T19:15:57.219223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T22:33:15.826269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-07-14T19:15:59.721419+0000","last_change":"2023-07-14T19:15:58.246859+0000","last_active":"2023-07-14T19:15:59.721419+0000","last_peered":"2023-07-14T19:15:59.721419+0000","last_clean":"2023-07-14T19:15:59.721419+0000","last_became_active":"2023-07-14T19:15:58.246541+0000","last_became_peered":"2023-07-14T19:15:58.246541+0000","last_unstale":"2023-07-14T19:15:59.721419+0000","last_undegraded":"2023-07-14T19:15:59.721419+0000","last_fullsized":"2023-07-14T19:15:59.721419+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_clean_scrub_stamp":"2023-07-14T19:15:57.219223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T03:07:22.056544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-07-14T19:15:59.939090+0000","last_change":"2023-07-14T19:15:58.250112+0000","last_active":"2023-07-14T19:15:59.939090+0000","last_peered":"2023-07-14T19:15:59.939090+0000","last_clean":"2023-07-14T19:15:59.939090+0000","last_became_active":"2023-07-14T19:15:58.249824+0000","last_became_peered":"2023-07-14T19:15:58.249824+0000","last_unstale":"2023-07-14T19:15:59.939090+0000","last_undegraded":"2023-07-14T19:15:59.939090+0000","last_fullsized":"2023-07-14T19:15:59.939090+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_clean_scrub_stamp":"2023-07-14T19:15:57.219223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T22:34:26.596011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-07-14T19:15:59.723229+0000","last_change":"2023-07-14T19:15:58.243011+0000","last_active":"2023-07-14T19:15:59.723229+0000","last_peered":"2023-07-14T19:15:59.723229+0000","last_clean":"2023-07-14T19:15:59.723229+0000","last_became_active":"2023-07-14T19:15:58.242693+0000","last_became_peered":"2023-07-14T19:15:58.242693+0000","last_unstale":"2023-07-14T19:15:59.723229+0000","last_undegraded":"2023-07-14T19:15:59.723229+0000","last_fullsized":"2023-07-14T19:15:59.723229+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_clean_scrub_stamp":"2023-07-14T19:15:57.219223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:04:37.187798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-07-14T19:15:59.721093+0000","last_change":"2023-07-14T19:15:58.251298+0000","last_active":"2023-07-14T19:15:59.721093+0000","last_peered":"2023-07-14T19:15:59.721093+0000","last_clean":"2023-07-14T19:15:59.721093+0000","last_became_active":"2023-07-14T19:15:58.250984+0000","last_became_peered":"2023-07-14T19:15:58.250984+0000","last_unstale":"2023-07-14T19:15:59.721093+0000","last_undegraded":"2023-07-14T19:15:59.721093+0000","last_fullsized":"2023-07-14T19:15:59.721093+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_clean_scrub_stamp":"2023-07-14T19:15:57.219223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T20:29:32.350678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"81'11","reported_seq":25,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-07-14T19:16:03.613085+0000","last_change":"2023-07-14T19:15:57.235142+0000","last_active":"2023-07-14T19:16:03.613085+0000","last_peered":"2023-07-14T19:16:03.613085+0000","last_clean":"2023-07-14T19:16:03.613085+0000","last_became_active":"2023-07-14T19:15:57.234803+0000","last_became_peered":"2023-07-14T19:15:57.234803+0000","last_unstale":"2023-07-14T19:16:03.613085+0000","last_undegraded":"2023-07-14T19:16:03.613085+0000","last_fullsized":"2023-07-14T19:16:03.613085+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:15:56.210166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:15:56.210166+0000","last_clean_scrub_stamp":"2023-07-14T19:15:56.210166+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:27:04.191229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-07-14T19:15:59.722681+0000","last_change":"2023-07-14T19:15:58.247688+0000","last_active":"2023-07-14T19:15:59.722681+0000","last_peered":"2023-07-14T19:15:59.722681+0000","last_clean":"2023-07-14T19:15:59.722681+0000","last_became_active":"2023-07-14T19:15:58.247400+0000","last_became_peered":"2023-07-14T19:15:58.247400+0000","last_unstale":"2023-07-14T19:15:59.722681+0000","last_undegraded":"2023-07-14T19:15:59.722681+0000","last_fullsized":"2023-07-14T19:15:59.722681+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_clean_scrub_stamp":"2023-07-14T19:15:57.219223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T21:28:57.315894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"80'1","reported_seq":15,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-07-14T19:15:59.722989+0000","last_change":"2023-07-14T19:15:57.240026+0000","last_active":"2023-07-14T19:15:59.722989+0000","last_peered":"2023-07-14T19:15:59.722989+0000","last_clean":"2023-07-14T19:15:59.722989+0000","last_became_active":"2023-07-14T19:15:57.239687+0000","last_became_peered":"2023-07-14T19:15:57.239687+0000","last_unstale":"2023-07-14T19:15:59.722989+0000","last_undegraded":"2023-07-14T19:15:59.722989+0000","last_fullsized":"2023-07-14T19:15:59.722989+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:15:56.210166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:15:56.210166+0000","last_clean_scrub_stamp":"2023-07-14T19:15:56.210166+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:18:22.780240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-07-14T19:15:59.573195+0000","last_change":"2023-07-14T19:15:58.247039+0000","last_active":"2023-07-14T19:15:59.573195+0000","last_peered":"2023-07-14T19:15:59.573195+0000","last_clean":"2023-07-14T19:15:59.573195+0000","last_became_active":"2023-07-14T19:15:58.246685+0000","last_became_peered":"2023-07-14T19:15:58.246685+0000","last_unstale":"2023-07-14T19:15:59.573195+0000","last_undegraded":"2023-07-14T19:15:59.573195+0000","last_fullsized":"2023-07-14T19:15:59.573195+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_clean_scrub_stamp":"2023-07-14T19:15:57.219223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T03:20:32.272596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"81'7","reported_seq":21,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-07-14T19:16:03.612329+0000","last_change":"2023-07-14T19:15:57.244612+0000","last_active":"2023-07-14T19:16:03.612329+0000","last_peered":"2023-07-14T19:16:03.612329+0000","last_clean":"2023-07-14T19:16:03.612329+0000","last_became_active":"2023-07-14T19:15:57.243761+0000","last_became_peered":"2023-07-14T19:15:57.243761+0000","last_unstale":"2023-07-14T19:16:03.612329+0000","last_undegraded":"2023-07-14T19:16:03.612329+0000","last_fullsized":"2023-07-14T19:16:03.612329+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:15:56.210166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:15:56.210166+0000","last_clean_scrub_stamp":"2023-07-14T19:15:56.210166+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T21:44:07.276569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-07-14T19:15:59.721874+0000","last_change":"2023-07-14T19:15:58.239903+0000","last_active":"2023-07-14T19:15:59.721874+0000","last_peered":"2023-07-14T19:15:59.721874+0000","last_clean":"2023-07-14T19:15:59.721874+0000","last_became_active":"2023-07-14T19:15:58.239627+0000","last_became_peered":"2023-07-14T19:15:58.239627+0000","last_unstale":"2023-07-14T19:15:59.721874+0000","last_undegraded":"2023-07-14T19:15:59.721874+0000","last_fullsized":"2023-07-14T19:15:59.721874+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_clean_scrub_stamp":"2023-07-14T19:15:57.219223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:56:30.086686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"80'2","reported_seq":16,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-07-14T19:15:59.232454+0000","last_change":"2023-07-14T19:15:57.241773+0000","last_active":"2023-07-14T19:15:59.232454+0000","last_peered":"2023-07-14T19:15:59.232454+0000","last_clean":"2023-07-14T19:15:59.232454+0000","last_became_active":"2023-07-14T19:15:57.241451+0000","last_became_peered":"2023-07-14T19:15:57.241451+0000","last_unstale":"2023-07-14T19:15:59.232454+0000","last_undegraded":"2023-07-14T19:15:59.232454+0000","last_fullsized":"2023-07-14T19:15:59.232454+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:15:56.210166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:15:56.210166+0000","last_clean_scrub_stamp":"2023-07-14T19:15:56.210166+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T00:53:37.977462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-07-14T19:15:59.231297+0000","last_change":"2023-07-14T19:15:58.250564+0000","last_active":"2023-07-14T19:15:59.231297+0000","last_peered":"2023-07-14T19:15:59.231297+0000","last_clean":"2023-07-14T19:15:59.231297+0000","last_became_active":"2023-07-14T19:15:58.250277+0000","last_became_peered":"2023-07-14T19:15:58.250277+0000","last_unstale":"2023-07-14T19:15:59.231297+0000","last_undegraded":"2023-07-14T19:15:59.231297+0000","last_fullsized":"2023-07-14T19:15:59.231297+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_clean_scrub_stamp":"2023-07-14T19:15:57.219223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:37:56.001097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"81'5","reported_seq":19,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-07-14T19:16:01.574166+0000","last_change":"2023-07-14T19:15:57.245586+0000","last_active":"2023-07-14T19:16:01.574166+0000","last_peered":"2023-07-14T19:16:01.574166+0000","last_clean":"2023-07-14T19:16:01.574166+0000","last_became_active":"2023-07-14T19:15:57.244767+0000","last_became_peered":"2023-07-14T19:15:57.244767+0000","last_unstale":"2023-07-14T19:16:01.574166+0000","last_undegraded":"2023-07-14T19:16:01.574166+0000","last_fullsized":"2023-07-14T19:16:01.574166+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:15:56.210166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:15:56.210166+0000","last_clean_scrub_stamp":"2023-07-14T19:15:56.210166+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T06:43:35.975084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":574,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-07-14T19:15:59.232307+0000","last_change":"2023-07-14T19:15:58.240509+0000","last_active":"2023-07-14T19:15:59.232307+0000","last_peered":"2023-07-14T19:15:59.232307+0000","last_clean":"2023-07-14T19:15:59.232307+0000","last_became_active":"2023-07-14T19:15:58.240198+0000","last_became_peered":"2023-07-14T19:15:58.240198+0000","last_unstale":"2023-07-14T19:15:59.232307+0000","last_undegraded":"2023-07-14T19:15:59.232307+0000","last_fullsized":"2023-07-14T19:15:59.232307+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_clean_scrub_stamp":"2023-07-14T19:15:57.219223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T19:29:25.962372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"80'1","reported_seq":15,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-07-14T19:15:59.721966+0000","last_change":"2023-07-14T19:15:57.239582+0000","last_active":"2023-07-14T19:15:59.721966+0000","last_peered":"2023-07-14T19:15:59.721966+0000","last_clean":"2023-07-14T19:15:59.721966+0000","last_became_active":"2023-07-14T19:15:57.239244+0000","last_became_peered":"2023-07-14T19:15:57.239244+0000","last_unstale":"2023-07-14T19:15:59.721966+0000","last_undegraded":"2023-07-14T19:15:59.721966+0000","last_fullsized":"2023-07-14T19:15:59.721966+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:15:56.210166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:15:56.210166+0000","last_clean_scrub_stamp":"2023-07-14T19:15:56.210166+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T03:16:58.719650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-07-14T19:15:59.231791+0000","last_change":"2023-07-14T19:15:58.249443+0000","last_active":"2023-07-14T19:15:59.231791+0000","last_peered":"2023-07-14T19:15:59.231791+0000","last_clean":"2023-07-14T19:15:59.231791+0000","last_became_active":"2023-07-14T19:15:58.249131+0000","last_became_peered":"2023-07-14T19:15:58.249131+0000","last_unstale":"2023-07-14T19:15:59.231791+0000","last_undegraded":"2023-07-14T19:15:59.231791+0000","last_fullsized":"2023-07-14T19:15:59.231791+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_clean_scrub_stamp":"2023-07-14T19:15:57.219223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T03:45:01.049663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":14,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-07-14T19:15:59.722732+0000","last_change":"2023-07-14T19:15:57.243725+0000","last_active":"2023-07-14T19:15:59.722732+0000","last_peered":"2023-07-14T19:15:59.722732+0000","last_clean":"2023-07-14T19:15:59.722732+0000","last_became_active":"2023-07-14T19:15:57.243431+0000","last_became_peered":"2023-07-14T19:15:57.243431+0000","last_unstale":"2023-07-14T19:15:59.722732+0000","last_undegraded":"2023-07-14T19:15:59.722732+0000","last_fullsized":"2023-07-14T19:15:59.722732+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:15:56.210166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:15:56.210166+0000","last_clean_scrub_stamp":"2023-07-14T19:15:56.210166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:41:49.174581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-07-14T19:15:59.721836+0000","last_change":"2023-07-14T19:15:58.239460+0000","last_active":"2023-07-14T19:15:59.721836+0000","last_peered":"2023-07-14T19:15:59.721836+0000","last_clean":"2023-07-14T19:15:59.721836+0000","last_became_active":"2023-07-14T19:15:58.239086+0000","last_became_peered":"2023-07-14T19:15:58.239086+0000","last_unstale":"2023-07-14T19:15:59.721836+0000","last_undegraded":"2023-07-14T19:15:59.721836+0000","last_fullsized":"2023-07-14T19:15:59.721836+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_clean_scrub_stamp":"2023-07-14T19:15:57.219223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T23:25:07.452583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"81'1","reported_seq":15,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-07-14T19:15:59.231044+0000","last_change":"2023-07-14T19:15:57.240173+0000","last_active":"2023-07-14T19:15:59.231044+0000","last_peered":"2023-07-14T19:15:59.231044+0000","last_clean":"2023-07-14T19:15:59.231044+0000","last_became_active":"2023-07-14T19:15:57.239822+0000","last_became_peered":"2023-07-14T19:15:57.239822+0000","last_unstale":"2023-07-14T19:15:59.231044+0000","last_undegraded":"2023-07-14T19:15:59.231044+0000","last_fullsized":"2023-07-14T19:15:59.231044+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:15:56.210166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:15:56.210166+0000","last_clean_scrub_stamp":"2023-07-14T19:15:56.210166+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T20:22:32.066346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-07-14T19:15:59.722607+0000","last_change":"2023-07-14T19:15:58.241533+0000","last_active":"2023-07-14T19:15:59.722607+0000","last_peered":"2023-07-14T19:15:59.722607+0000","last_clean":"2023-07-14T19:15:59.722607+0000","last_became_active":"2023-07-14T19:15:58.241216+0000","last_became_peered":"2023-07-14T19:15:58.241216+0000","last_unstale":"2023-07-14T19:15:59.722607+0000","last_undegraded":"2023-07-14T19:15:59.722607+0000","last_fullsized":"2023-07-14T19:15:59.722607+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_clean_scrub_stamp":"2023-07-14T19:15:57.219223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:40:27.802949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":14,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-07-14T19:15:59.722052+0000","last_change":"2023-07-14T19:15:57.236808+0000","last_active":"2023-07-14T19:15:59.722052+0000","last_peered":"2023-07-14T19:15:59.722052+0000","last_clean":"2023-07-14T19:15:59.722052+0000","last_became_active":"2023-07-14T19:15:57.236480+0000","last_became_peered":"2023-07-14T19:15:57.236480+0000","last_unstale":"2023-07-14T19:15:59.722052+0000","last_undegraded":"2023-07-14T19:15:59.722052+0000","last_fullsized":"2023-07-14T19:15:59.722052+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:15:56.210166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:15:56.210166+0000","last_clean_scrub_stamp":"2023-07-14T19:15:56.210166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T20:50:18.427213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-07-14T19:15:59.721741+0000","last_change":"2023-07-14T19:15:58.243236+0000","last_active":"2023-07-14T19:15:59.721741+0000","last_peered":"2023-07-14T19:15:59.721741+0000","last_clean":"2023-07-14T19:15:59.721741+0000","last_became_active":"2023-07-14T19:15:58.242951+0000","last_became_peered":"2023-07-14T19:15:58.242951+0000","last_unstale":"2023-07-14T19:15:59.721741+0000","last_undegraded":"2023-07-14T19:15:59.721741+0000","last_fullsized":"2023-07-14T19:15:59.721741+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_clean_scrub_stamp":"2023-07-14T19:15:57.219223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T05:20:59.047342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"81'2","reported_seq":16,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-07-14T19:16:01.572601+0000","last_change":"2023-07-14T19:15:57.242251+0000","last_active":"2023-07-14T19:16:01.572601+0000","last_peered":"2023-07-14T19:16:01.572601+0000","last_clean":"2023-07-14T19:16:01.572601+0000","last_became_active":"2023-07-14T19:15:57.241907+0000","last_became_peered":"2023-07-14T19:15:57.241907+0000","last_unstale":"2023-07-14T19:16:01.572601+0000","last_undegraded":"2023-07-14T19:16:01.572601+0000","last_fullsized":"2023-07-14T19:16:01.572601+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:15:56.210166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:15:56.210166+0000","last_clean_scrub_stamp":"2023-07-14T19:15:56.210166+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:52:53.425429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-07-14T19:15:59.573106+0000","last_change":"2023-07-14T19:15:58.248412+0000","last_active":"2023-07-14T19:15:59.573106+0000","last_peered":"2023-07-14T19:15:59.573106+0000","last_clean":"2023-07-14T19:15:59.573106+0000","last_became_active":"2023-07-14T19:15:58.247619+0000","last_became_peered":"2023-07-14T19:15:58.247619+0000","last_unstale":"2023-07-14T19:15:59.573106+0000","last_undegraded":"2023-07-14T19:15:59.573106+0000","last_fullsized":"2023-07-14T19:15:59.573106+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_clean_scrub_stamp":"2023-07-14T19:15:57.219223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T00:15:52.855579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"81'4","reported_seq":18,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-07-14T19:15:59.231404+0000","last_change":"2023-07-14T19:15:57.240436+0000","last_active":"2023-07-14T19:15:59.231404+0000","last_peered":"2023-07-14T19:15:59.231404+0000","last_clean":"2023-07-14T19:15:59.231404+0000","last_became_active":"2023-07-14T19:15:57.240148+0000","last_became_peered":"2023-07-14T19:15:57.240148+0000","last_unstale":"2023-07-14T19:15:59.231404+0000","last_undegraded":"2023-07-14T19:15:59.231404+0000","last_fullsized":"2023-07-14T19:15:59.231404+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:15:56.210166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:15:56.210166+0000","last_clean_scrub_stamp":"2023-07-14T19:15:56.210166+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T03:15:40.160793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-07-14T19:15:59.722062+0000","last_change":"2023-07-14T19:15:58.246127+0000","last_active":"2023-07-14T19:15:59.722062+0000","last_peered":"2023-07-14T19:15:59.722062+0000","last_clean":"2023-07-14T19:15:59.722062+0000","last_became_active":"2023-07-14T19:15:58.245778+0000","last_became_peered":"2023-07-14T19:15:58.245778+0000","last_unstale":"2023-07-14T19:15:59.722062+0000","last_undegraded":"2023-07-14T19:15:59.722062+0000","last_fullsized":"2023-07-14T19:15:59.722062+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_clean_scrub_stamp":"2023-07-14T19:15:57.219223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T00:21:57.683936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"80'6","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-07-14T19:15:59.722153+0000","last_change":"2023-07-14T19:15:57.243660+0000","last_active":"2023-07-14T19:15:59.722153+0000","last_peered":"2023-07-14T19:15:59.722153+0000","last_clean":"2023-07-14T19:15:59.722153+0000","last_became_active":"2023-07-14T19:15:57.243338+0000","last_became_peered":"2023-07-14T19:15:57.243338+0000","last_unstale":"2023-07-14T19:15:59.722153+0000","last_undegraded":"2023-07-14T19:15:59.722153+0000","last_fullsized":"2023-07-14T19:15:59.722153+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:15:56.210166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:15:56.210166+0000","last_clean_scrub_stamp":"2023-07-14T19:15:56.210166+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T06:12:23.930218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1564,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-07-14T19:15:59.232525+0000","last_change":"2023-07-14T19:15:58.244162+0000","last_active":"2023-07-14T19:15:59.232525+0000","last_peered":"2023-07-14T19:15:59.232525+0000","last_clean":"2023-07-14T19:15:59.232525+0000","last_became_active":"2023-07-14T19:15:58.243912+0000","last_became_peered":"2023-07-14T19:15:58.243912+0000","last_unstale":"2023-07-14T19:15:59.232525+0000","last_undegraded":"2023-07-14T19:15:59.232525+0000","last_fullsized":"2023-07-14T19:15:59.232525+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_clean_scrub_stamp":"2023-07-14T19:15:57.219223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T23:05:24.992671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"81'6","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-07-14T19:16:02.596564+0000","last_change":"2023-07-14T19:15:57.246975+0000","last_active":"2023-07-14T19:16:02.596564+0000","last_peered":"2023-07-14T19:16:02.596564+0000","last_clean":"2023-07-14T19:16:02.596564+0000","last_became_active":"2023-07-14T19:15:57.246481+0000","last_became_peered":"2023-07-14T19:15:57.246481+0000","last_unstale":"2023-07-14T19:16:02.596564+0000","last_undegraded":"2023-07-14T19:16:02.596564+0000","last_fullsized":"2023-07-14T19:16:02.596564+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:15:56.210166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:15:56.210166+0000","last_clean_scrub_stamp":"2023-07-14T19:15:56.210166+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T06:50:24.165830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-07-14T19:15:59.721632+0000","last_change":"2023-07-14T19:15:58.241870+0000","last_active":"2023-07-14T19:15:59.721632+0000","last_peered":"2023-07-14T19:15:59.721632+0000","last_clean":"2023-07-14T19:15:59.721632+0000","last_became_active":"2023-07-14T19:15:58.241503+0000","last_became_peered":"2023-07-14T19:15:58.241503+0000","last_unstale":"2023-07-14T19:15:59.721632+0000","last_undegraded":"2023-07-14T19:15:59.721632+0000","last_fullsized":"2023-07-14T19:15:59.721632+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_clean_scrub_stamp":"2023-07-14T19:15:57.219223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T23:05:14.779450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"81'3","reported_seq":17,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-07-14T19:16:01.571518+0000","last_change":"2023-07-14T19:15:57.236715+0000","last_active":"2023-07-14T19:16:01.571518+0000","last_peered":"2023-07-14T19:16:01.571518+0000","last_clean":"2023-07-14T19:16:01.571518+0000","last_became_active":"2023-07-14T19:15:57.236391+0000","last_became_peered":"2023-07-14T19:15:57.236391+0000","last_unstale":"2023-07-14T19:16:01.571518+0000","last_undegraded":"2023-07-14T19:16:01.571518+0000","last_fullsized":"2023-07-14T19:16:01.571518+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:15:56.210166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:15:56.210166+0000","last_clean_scrub_stamp":"2023-07-14T19:15:56.210166+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:06:23.938675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-07-14T19:15:59.721870+0000","last_change":"2023-07-14T19:15:58.245110+0000","last_active":"2023-07-14T19:15:59.721870+0000","last_peered":"2023-07-14T19:15:59.721870+0000","last_clean":"2023-07-14T19:15:59.721870+0000","last_became_active":"2023-07-14T19:15:58.244779+0000","last_became_peered":"2023-07-14T19:15:58.244779+0000","last_unstale":"2023-07-14T19:15:59.721870+0000","last_undegraded":"2023-07-14T19:15:59.721870+0000","last_fullsized":"2023-07-14T19:15:59.721870+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_clean_scrub_stamp":"2023-07-14T19:15:57.219223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T00:24:23.826897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":14,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-07-14T19:15:59.229648+0000","last_change":"2023-07-14T19:15:57.236795+0000","last_active":"2023-07-14T19:15:59.229648+0000","last_peered":"2023-07-14T19:15:59.229648+0000","last_clean":"2023-07-14T19:15:59.229648+0000","last_became_active":"2023-07-14T19:15:57.236442+0000","last_became_peered":"2023-07-14T19:15:57.236442+0000","last_unstale":"2023-07-14T19:15:59.229648+0000","last_undegraded":"2023-07-14T19:15:59.229648+0000","last_fullsized":"2023-07-14T19:15:59.229648+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:15:56.210166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:15:56.210166+0000","last_clean_scrub_stamp":"2023-07-14T19:15:56.210166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:05:41.764871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-07-14T19:15:59.721448+0000","last_change":"2023-07-14T19:15:58.247790+0000","last_active":"2023-07-14T19:15:59.721448+0000","last_peered":"2023-07-14T19:15:59.721448+0000","last_clean":"2023-07-14T19:15:59.721448+0000","last_became_active":"2023-07-14T19:15:58.247461+0000","last_became_peered":"2023-07-14T19:15:58.247461+0000","last_unstale":"2023-07-14T19:15:59.721448+0000","last_undegraded":"2023-07-14T19:15:59.721448+0000","last_fullsized":"2023-07-14T19:15:59.721448+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_clean_scrub_stamp":"2023-07-14T19:15:57.219223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:29:24.306834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"81'16","reported_seq":30,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-07-14T19:16:02.592714+0000","last_change":"2023-07-14T19:15:57.244534+0000","last_active":"2023-07-14T19:16:02.592714+0000","last_peered":"2023-07-14T19:16:02.592714+0000","last_clean":"2023-07-14T19:16:02.592714+0000","last_became_active":"2023-07-14T19:15:57.243107+0000","last_became_peered":"2023-07-14T19:15:57.243107+0000","last_unstale":"2023-07-14T19:16:02.592714+0000","last_undegraded":"2023-07-14T19:16:02.592714+0000","last_fullsized":"2023-07-14T19:16:02.592714+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:15:56.210166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:15:56.210166+0000","last_clean_scrub_stamp":"2023-07-14T19:15:56.210166+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T21:48:40.199873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-07-14T19:15:59.231092+0000","last_change":"2023-07-14T19:15:58.250511+0000","last_active":"2023-07-14T19:15:59.231092+0000","last_peered":"2023-07-14T19:15:59.231092+0000","last_clean":"2023-07-14T19:15:59.231092+0000","last_became_active":"2023-07-14T19:15:58.250159+0000","last_became_peered":"2023-07-14T19:15:58.250159+0000","last_unstale":"2023-07-14T19:15:59.231092+0000","last_undegraded":"2023-07-14T19:15:59.231092+0000","last_fullsized":"2023-07-14T19:15:59.231092+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_clean_scrub_stamp":"2023-07-14T19:15:57.219223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T21:32:01.440745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"80'1","reported_seq":15,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-07-14T19:15:59.232676+0000","last_change":"2023-07-14T19:15:57.238719+0000","last_active":"2023-07-14T19:15:59.232676+0000","last_peered":"2023-07-14T19:15:59.232676+0000","last_clean":"2023-07-14T19:15:59.232676+0000","last_became_active":"2023-07-14T19:15:57.238404+0000","last_became_peered":"2023-07-14T19:15:57.238404+0000","last_unstale":"2023-07-14T19:15:59.232676+0000","last_undegraded":"2023-07-14T19:15:59.232676+0000","last_fullsized":"2023-07-14T19:15:59.232676+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:15:56.210166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:15:56.210166+0000","last_clean_scrub_stamp":"2023-07-14T19:15:56.210166+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T03:14:38.948894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-07-14T19:15:59.938888+0000","last_change":"2023-07-14T19:15:58.245052+0000","last_active":"2023-07-14T19:15:59.938888+0000","last_peered":"2023-07-14T19:15:59.938888+0000","last_clean":"2023-07-14T19:15:59.938888+0000","last_became_active":"2023-07-14T19:15:58.244733+0000","last_became_peered":"2023-07-14T19:15:58.244733+0000","last_unstale":"2023-07-14T19:15:59.938888+0000","last_undegraded":"2023-07-14T19:15:59.938888+0000","last_fullsized":"2023-07-14T19:15:59.938888+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_clean_scrub_stamp":"2023-07-14T19:15:57.219223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T03:43:49.689439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"81'1","reported_seq":15,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-07-14T19:15:59.722595+0000","last_change":"2023-07-14T19:15:57.245402+0000","last_active":"2023-07-14T19:15:59.722595+0000","last_peered":"2023-07-14T19:15:59.722595+0000","last_clean":"2023-07-14T19:15:59.722595+0000","last_became_active":"2023-07-14T19:15:57.245092+0000","last_became_peered":"2023-07-14T19:15:57.245092+0000","last_unstale":"2023-07-14T19:15:59.722595+0000","last_undegraded":"2023-07-14T19:15:59.722595+0000","last_fullsized":"2023-07-14T19:15:59.722595+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:15:56.210166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:15:56.210166+0000","last_clean_scrub_stamp":"2023-07-14T19:15:56.210166+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:17:28.260009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-07-14T19:15:59.722257+0000","last_change":"2023-07-14T19:15:58.247611+0000","last_active":"2023-07-14T19:15:59.722257+0000","last_peered":"2023-07-14T19:15:59.722257+0000","last_clean":"2023-07-14T19:15:59.722257+0000","last_became_active":"2023-07-14T19:15:58.247313+0000","last_became_peered":"2023-07-14T19:15:58.247313+0000","last_unstale":"2023-07-14T19:15:59.722257+0000","last_undegraded":"2023-07-14T19:15:59.722257+0000","last_fullsized":"2023-07-14T19:15:59.722257+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_clean_scrub_stamp":"2023-07-14T19:15:57.219223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T06:40:19.317380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":14,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-07-14T19:15:59.722196+0000","last_change":"2023-07-14T19:15:57.242149+0000","last_active":"2023-07-14T19:15:59.722196+0000","last_peered":"2023-07-14T19:15:59.722196+0000","last_clean":"2023-07-14T19:15:59.722196+0000","last_became_active":"2023-07-14T19:15:57.241847+0000","last_became_peered":"2023-07-14T19:15:57.241847+0000","last_unstale":"2023-07-14T19:15:59.722196+0000","last_undegraded":"2023-07-14T19:15:59.722196+0000","last_fullsized":"2023-07-14T19:15:59.722196+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:15:56.210166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:15:56.210166+0000","last_clean_scrub_stamp":"2023-07-14T19:15:56.210166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:14:47.640280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-07-14T19:15:59.722352+0000","last_change":"2023-07-14T19:15:58.244573+0000","last_active":"2023-07-14T19:15:59.722352+0000","last_peered":"2023-07-14T19:15:59.722352+0000","last_clean":"2023-07-14T19:15:59.722352+0000","last_became_active":"2023-07-14T19:15:58.244165+0000","last_became_peered":"2023-07-14T19:15:58.244165+0000","last_unstale":"2023-07-14T19:15:59.722352+0000","last_undegraded":"2023-07-14T19:15:59.722352+0000","last_fullsized":"2023-07-14T19:15:59.722352+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_clean_scrub_stamp":"2023-07-14T19:15:57.219223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T05:36:19.851400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-07-14T19:15:59.231138+0000","last_change":"2023-07-14T19:15:58.246282+0000","last_active":"2023-07-14T19:15:59.231138+0000","last_peered":"2023-07-14T19:15:59.231138+0000","last_clean":"2023-07-14T19:15:59.231138+0000","last_became_active":"2023-07-14T19:15:58.245984+0000","last_became_peered":"2023-07-14T19:15:58.245984+0000","last_unstale":"2023-07-14T19:15:59.231138+0000","last_undegraded":"2023-07-14T19:15:59.231138+0000","last_fullsized":"2023-07-14T19:15:59.231138+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_clean_scrub_stamp":"2023-07-14T19:15:57.219223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T00:42:41.109802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":14,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-07-14T19:15:59.722605+0000","last_change":"2023-07-14T19:15:57.244731+0000","last_active":"2023-07-14T19:15:59.722605+0000","last_peered":"2023-07-14T19:15:59.722605+0000","last_clean":"2023-07-14T19:15:59.722605+0000","last_became_active":"2023-07-14T19:15:57.244384+0000","last_became_peered":"2023-07-14T19:15:57.244384+0000","last_unstale":"2023-07-14T19:15:59.722605+0000","last_undegraded":"2023-07-14T19:15:59.722605+0000","last_fullsized":"2023-07-14T19:15:59.722605+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:15:56.210166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:15:56.210166+0000","last_clean_scrub_stamp":"2023-07-14T19:15:56.210166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T22:20:50.336578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-07-14T19:15:59.722302+0000","last_change":"2023-07-14T19:15:58.241457+0000","last_active":"2023-07-14T19:15:59.722302+0000","last_peered":"2023-07-14T19:15:59.722302+0000","last_clean":"2023-07-14T19:15:59.722302+0000","last_became_active":"2023-07-14T19:15:58.241113+0000","last_became_peered":"2023-07-14T19:15:58.241113+0000","last_unstale":"2023-07-14T19:15:59.722302+0000","last_undegraded":"2023-07-14T19:15:59.722302+0000","last_fullsized":"2023-07-14T19:15:59.722302+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_clean_scrub_stamp":"2023-07-14T19:15:57.219223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T21:28:59.714151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"81'2","reported_seq":16,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-07-14T19:16:01.572074+0000","last_change":"2023-07-14T19:15:57.235057+0000","last_active":"2023-07-14T19:16:01.572074+0000","last_peered":"2023-07-14T19:16:01.572074+0000","last_clean":"2023-07-14T19:16:01.572074+0000","last_became_active":"2023-07-14T19:15:57.234751+0000","last_became_peered":"2023-07-14T19:15:57.234751+0000","last_unstale":"2023-07-14T19:16:01.572074+0000","last_undegraded":"2023-07-14T19:16:01.572074+0000","last_fullsized":"2023-07-14T19:16:01.572074+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:15:56.210166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:15:56.210166+0000","last_clean_scrub_stamp":"2023-07-14T19:15:56.210166+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:12:00.277001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":336,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-07-14T19:15:59.721531+0000","last_change":"2023-07-14T19:14:58.969281+0000","last_active":"2023-07-14T19:15:59.721531+0000","last_peered":"2023-07-14T19:15:59.721531+0000","last_clean":"2023-07-14T19:15:59.721531+0000","last_became_active":"2023-07-14T19:14:58.968767+0000","last_became_peered":"2023-07-14T19:14:58.968767+0000","last_unstale":"2023-07-14T19:15:59.721531+0000","last_undegraded":"2023-07-14T19:15:59.721531+0000","last_fullsized":"2023-07-14T19:15:59.721531+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:12:03.589019+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:12:03.589019+0000","last_clean_scrub_stamp":"2023-07-14T19:12:03.589019+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T05:39:04.087637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-07-14T19:15:59.722476+0000","last_change":"2023-07-14T19:15:58.250737+0000","last_active":"2023-07-14T19:15:59.722476+0000","last_peered":"2023-07-14T19:15:59.722476+0000","last_clean":"2023-07-14T19:15:59.722476+0000","last_became_active":"2023-07-14T19:15:58.250379+0000","last_became_peered":"2023-07-14T19:15:58.250379+0000","last_unstale":"2023-07-14T19:15:59.722476+0000","last_undegraded":"2023-07-14T19:15:59.722476+0000","last_fullsized":"2023-07-14T19:15:59.722476+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_clean_scrub_stamp":"2023-07-14T19:15:57.219223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T19:57:17.147937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"81'1","reported_seq":15,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-07-14T19:15:59.721738+0000","last_change":"2023-07-14T19:15:57.236799+0000","last_active":"2023-07-14T19:15:59.721738+0000","last_peered":"2023-07-14T19:15:59.721738+0000","last_clean":"2023-07-14T19:15:59.721738+0000","last_became_active":"2023-07-14T19:15:57.236503+0000","last_became_peered":"2023-07-14T19:15:57.236503+0000","last_unstale":"2023-07-14T19:15:59.721738+0000","last_undegraded":"2023-07-14T19:15:59.721738+0000","last_fullsized":"2023-07-14T19:15:59.721738+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:15:56.210166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:15:56.210166+0000","last_clean_scrub_stamp":"2023-07-14T19:15:56.210166+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T00:26:24.967329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"81'3","reported_seq":17,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-07-14T19:16:03.614535+0000","last_change":"2023-07-14T19:15:57.245788+0000","last_active":"2023-07-14T19:16:03.614535+0000","last_peered":"2023-07-14T19:16:03.614535+0000","last_clean":"2023-07-14T19:16:03.614535+0000","last_became_active":"2023-07-14T19:15:57.244749+0000","last_became_peered":"2023-07-14T19:15:57.244749+0000","last_unstale":"2023-07-14T19:16:03.614535+0000","last_undegraded":"2023-07-14T19:16:03.614535+0000","last_fullsized":"2023-07-14T19:16:03.614535+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:15:56.210166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:15:56.210166+0000","last_clean_scrub_stamp":"2023-07-14T19:15:56.210166+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T00:56:58.078952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-07-14T19:15:59.722962+0000","last_change":"2023-07-14T19:15:58.250973+0000","last_active":"2023-07-14T19:15:59.722962+0000","last_peered":"2023-07-14T19:15:59.722962+0000","last_clean":"2023-07-14T19:15:59.722962+0000","last_became_active":"2023-07-14T19:15:58.250634+0000","last_became_peered":"2023-07-14T19:15:58.250634+0000","last_unstale":"2023-07-14T19:15:59.722962+0000","last_undegraded":"2023-07-14T19:15:59.722962+0000","last_fullsized":"2023-07-14T19:15:59.722962+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_clean_scrub_stamp":"2023-07-14T19:15:57.219223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T06:14:32.307565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"81'7","reported_seq":21,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-07-14T19:16:03.613824+0000","last_change":"2023-07-14T19:15:57.234996+0000","last_active":"2023-07-14T19:16:03.613824+0000","last_peered":"2023-07-14T19:16:03.613824+0000","last_clean":"2023-07-14T19:16:03.613824+0000","last_became_active":"2023-07-14T19:15:57.234697+0000","last_became_peered":"2023-07-14T19:15:57.234697+0000","last_unstale":"2023-07-14T19:16:03.613824+0000","last_undegraded":"2023-07-14T19:16:03.613824+0000","last_fullsized":"2023-07-14T19:16:03.613824+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:15:56.210166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:15:56.210166+0000","last_clean_scrub_stamp":"2023-07-14T19:15:56.210166+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T21:46:18.232628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-07-14T19:15:59.722957+0000","last_change":"2023-07-14T19:15:58.241753+0000","last_active":"2023-07-14T19:15:59.722957+0000","last_peered":"2023-07-14T19:15:59.722957+0000","last_clean":"2023-07-14T19:15:59.722957+0000","last_became_active":"2023-07-14T19:15:58.241485+0000","last_became_peered":"2023-07-14T19:15:58.241485+0000","last_unstale":"2023-07-14T19:15:59.722957+0000","last_undegraded":"2023-07-14T19:15:59.722957+0000","last_fullsized":"2023-07-14T19:15:59.722957+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_clean_scrub_stamp":"2023-07-14T19:15:57.219223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T03:02:48.399750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"81'4","reported_seq":18,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-07-14T19:16:01.572337+0000","last_change":"2023-07-14T19:15:57.240019+0000","last_active":"2023-07-14T19:16:01.572337+0000","last_peered":"2023-07-14T19:16:01.572337+0000","last_clean":"2023-07-14T19:16:01.572337+0000","last_became_active":"2023-07-14T19:15:57.239521+0000","last_became_peered":"2023-07-14T19:15:57.239521+0000","last_unstale":"2023-07-14T19:16:01.572337+0000","last_undegraded":"2023-07-14T19:16:01.572337+0000","last_fullsized":"2023-07-14T19:16:01.572337+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:15:56.210166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:15:56.210166+0000","last_clean_scrub_stamp":"2023-07-14T19:15:56.210166+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T23:32:40.283168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-07-14T19:15:59.573249+0000","last_change":"2023-07-14T19:15:58.248187+0000","last_active":"2023-07-14T19:15:59.573249+0000","last_peered":"2023-07-14T19:15:59.573249+0000","last_clean":"2023-07-14T19:15:59.573249+0000","last_became_active":"2023-07-14T19:15:58.247854+0000","last_became_peered":"2023-07-14T19:15:58.247854+0000","last_unstale":"2023-07-14T19:15:59.573249+0000","last_undegraded":"2023-07-14T19:15:59.573249+0000","last_fullsized":"2023-07-14T19:15:59.573249+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_clean_scrub_stamp":"2023-07-14T19:15:57.219223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T21:01:16.786236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"81'2","reported_seq":16,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-07-14T19:16:01.572678+0000","last_change":"2023-07-14T19:15:57.245012+0000","last_active":"2023-07-14T19:16:01.572678+0000","last_peered":"2023-07-14T19:16:01.572678+0000","last_clean":"2023-07-14T19:16:01.572678+0000","last_became_active":"2023-07-14T19:15:57.244776+0000","last_became_peered":"2023-07-14T19:15:57.244776+0000","last_unstale":"2023-07-14T19:16:01.572678+0000","last_undegraded":"2023-07-14T19:16:01.572678+0000","last_fullsized":"2023-07-14T19:16:01.572678+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:15:56.210166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:15:56.210166+0000","last_clean_scrub_stamp":"2023-07-14T19:15:56.210166+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T20:16:41.721374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-07-14T19:15:59.232170+0000","last_change":"2023-07-14T19:15:58.247704+0000","last_active":"2023-07-14T19:15:59.232170+0000","last_peered":"2023-07-14T19:15:59.232170+0000","last_clean":"2023-07-14T19:15:59.232170+0000","last_became_active":"2023-07-14T19:15:58.247364+0000","last_became_peered":"2023-07-14T19:15:58.247364+0000","last_unstale":"2023-07-14T19:15:59.232170+0000","last_undegraded":"2023-07-14T19:15:59.232170+0000","last_fullsized":"2023-07-14T19:15:59.232170+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_clean_scrub_stamp":"2023-07-14T19:15:57.219223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T19:44:25.096100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":14,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-07-14T19:15:59.722984+0000","last_change":"2023-07-14T19:15:57.234991+0000","last_active":"2023-07-14T19:15:59.722984+0000","last_peered":"2023-07-14T19:15:59.722984+0000","last_clean":"2023-07-14T19:15:59.722984+0000","last_became_active":"2023-07-14T19:15:57.234680+0000","last_became_peered":"2023-07-14T19:15:57.234680+0000","last_unstale":"2023-07-14T19:15:59.722984+0000","last_undegraded":"2023-07-14T19:15:59.722984+0000","last_fullsized":"2023-07-14T19:15:59.722984+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:15:56.210166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:15:56.210166+0000","last_clean_scrub_stamp":"2023-07-14T19:15:56.210166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T03:09:07.479398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-07-14T19:15:59.938707+0000","last_change":"2023-07-14T19:15:58.247224+0000","last_active":"2023-07-14T19:15:59.938707+0000","last_peered":"2023-07-14T19:15:59.938707+0000","last_clean":"2023-07-14T19:15:59.938707+0000","last_became_active":"2023-07-14T19:15:58.246889+0000","last_became_peered":"2023-07-14T19:15:58.246889+0000","last_unstale":"2023-07-14T19:15:59.938707+0000","last_undegraded":"2023-07-14T19:15:59.938707+0000","last_fullsized":"2023-07-14T19:15:59.938707+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_clean_scrub_stamp":"2023-07-14T19:15:57.219223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:25:39.860559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":14,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-07-14T19:15:59.720943+0000","last_change":"2023-07-14T19:15:57.246491+0000","last_active":"2023-07-14T19:15:59.720943+0000","last_peered":"2023-07-14T19:15:59.720943+0000","last_clean":"2023-07-14T19:15:59.720943+0000","last_became_active":"2023-07-14T19:15:57.246178+0000","last_became_peered":"2023-07-14T19:15:57.246178+0000","last_unstale":"2023-07-14T19:15:59.720943+0000","last_undegraded":"2023-07-14T19:15:59.720943+0000","last_fullsized":"2023-07-14T19:15:59.720943+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:15:56.210166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:15:56.210166+0000","last_clean_scrub_stamp":"2023-07-14T19:15:56.210166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T03:15:13.275825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-07-14T19:15:59.723335+0000","last_change":"2023-07-14T19:15:58.247005+0000","last_active":"2023-07-14T19:15:59.723335+0000","last_peered":"2023-07-14T19:15:59.723335+0000","last_clean":"2023-07-14T19:15:59.723335+0000","last_became_active":"2023-07-14T19:15:58.246156+0000","last_became_peered":"2023-07-14T19:15:58.246156+0000","last_unstale":"2023-07-14T19:15:59.723335+0000","last_undegraded":"2023-07-14T19:15:59.723335+0000","last_fullsized":"2023-07-14T19:15:59.723335+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_clean_scrub_stamp":"2023-07-14T19:15:57.219223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T20:11:25.410164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"81'4","reported_seq":18,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-07-14T19:16:01.572528+0000","last_change":"2023-07-14T19:15:57.245551+0000","last_active":"2023-07-14T19:16:01.572528+0000","last_peered":"2023-07-14T19:16:01.572528+0000","last_clean":"2023-07-14T19:16:01.572528+0000","last_became_active":"2023-07-14T19:15:57.245179+0000","last_became_peered":"2023-07-14T19:15:57.245179+0000","last_unstale":"2023-07-14T19:16:01.572528+0000","last_undegraded":"2023-07-14T19:16:01.572528+0000","last_fullsized":"2023-07-14T19:16:01.572528+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:15:56.210166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:15:56.210166+0000","last_clean_scrub_stamp":"2023-07-14T19:15:56.210166+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T00:35:19.482494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-07-14T19:15:59.722416+0000","last_change":"2023-07-14T19:15:58.250454+0000","last_active":"2023-07-14T19:15:59.722416+0000","last_peered":"2023-07-14T19:15:59.722416+0000","last_clean":"2023-07-14T19:15:59.722416+0000","last_became_active":"2023-07-14T19:15:58.250145+0000","last_became_peered":"2023-07-14T19:15:58.250145+0000","last_unstale":"2023-07-14T19:15:59.722416+0000","last_undegraded":"2023-07-14T19:15:59.722416+0000","last_fullsized":"2023-07-14T19:15:59.722416+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_clean_scrub_stamp":"2023-07-14T19:15:57.219223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T23:46:27.565214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"81'3","reported_seq":17,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-07-14T19:16:02.594567+0000","last_change":"2023-07-14T19:15:57.247545+0000","last_active":"2023-07-14T19:16:02.594567+0000","last_peered":"2023-07-14T19:16:02.594567+0000","last_clean":"2023-07-14T19:16:02.594567+0000","last_became_active":"2023-07-14T19:15:57.247201+0000","last_became_peered":"2023-07-14T19:15:57.247201+0000","last_unstale":"2023-07-14T19:16:02.594567+0000","last_undegraded":"2023-07-14T19:16:02.594567+0000","last_fullsized":"2023-07-14T19:16:02.594567+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:15:56.210166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:15:56.210166+0000","last_clean_scrub_stamp":"2023-07-14T19:15:56.210166+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T23:52:58.892773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-07-14T19:15:59.722674+0000","last_change":"2023-07-14T19:15:58.252144+0000","last_active":"2023-07-14T19:15:59.722674+0000","last_peered":"2023-07-14T19:15:59.722674+0000","last_clean":"2023-07-14T19:15:59.722674+0000","last_became_active":"2023-07-14T19:15:58.251844+0000","last_became_peered":"2023-07-14T19:15:58.251844+0000","last_unstale":"2023-07-14T19:15:59.722674+0000","last_undegraded":"2023-07-14T19:15:59.722674+0000","last_fullsized":"2023-07-14T19:15:59.722674+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_clean_scrub_stamp":"2023-07-14T19:15:57.219223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:19:09.690589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"80'1","reported_seq":15,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-07-14T19:15:59.231229+0000","last_change":"2023-07-14T19:15:57.239560+0000","last_active":"2023-07-14T19:15:59.231229+0000","last_peered":"2023-07-14T19:15:59.231229+0000","last_clean":"2023-07-14T19:15:59.231229+0000","last_became_active":"2023-07-14T19:15:57.239250+0000","last_became_peered":"2023-07-14T19:15:57.239250+0000","last_unstale":"2023-07-14T19:15:59.231229+0000","last_undegraded":"2023-07-14T19:15:59.231229+0000","last_fullsized":"2023-07-14T19:15:59.231229+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:15:56.210166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:15:56.210166+0000","last_clean_scrub_stamp":"2023-07-14T19:15:56.210166+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T21:04:10.055051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-07-14T19:15:59.722104+0000","last_change":"2023-07-14T19:15:58.242144+0000","last_active":"2023-07-14T19:15:59.722104+0000","last_peered":"2023-07-14T19:15:59.722104+0000","last_clean":"2023-07-14T19:15:59.722104+0000","last_became_active":"2023-07-14T19:15:58.241597+0000","last_became_peered":"2023-07-14T19:15:58.241597+0000","last_unstale":"2023-07-14T19:15:59.722104+0000","last_undegraded":"2023-07-14T19:15:59.722104+0000","last_fullsized":"2023-07-14T19:15:59.722104+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_clean_scrub_stamp":"2023-07-14T19:15:57.219223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T22:43:21.380478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"80'2","reported_seq":16,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-07-14T19:15:59.721650+0000","last_change":"2023-07-14T19:15:57.247059+0000","last_active":"2023-07-14T19:15:59.721650+0000","last_peered":"2023-07-14T19:15:59.721650+0000","last_clean":"2023-07-14T19:15:59.721650+0000","last_became_active":"2023-07-14T19:15:57.246736+0000","last_became_peered":"2023-07-14T19:15:57.246736+0000","last_unstale":"2023-07-14T19:15:59.721650+0000","last_undegraded":"2023-07-14T19:15:59.721650+0000","last_fullsized":"2023-07-14T19:15:59.721650+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:15:56.210166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:15:56.210166+0000","last_clean_scrub_stamp":"2023-07-14T19:15:56.210166+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T20:57:38.014630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-07-14T19:15:59.722005+0000","last_change":"2023-07-14T19:15:58.244180+0000","last_active":"2023-07-14T19:15:59.722005+0000","last_peered":"2023-07-14T19:15:59.722005+0000","last_clean":"2023-07-14T19:15:59.722005+0000","last_became_active":"2023-07-14T19:15:58.243885+0000","last_became_peered":"2023-07-14T19:15:58.243885+0000","last_unstale":"2023-07-14T19:15:59.722005+0000","last_undegraded":"2023-07-14T19:15:59.722005+0000","last_fullsized":"2023-07-14T19:15:59.722005+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_clean_scrub_stamp":"2023-07-14T19:15:57.219223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:56:30.077276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-07-14T19:15:59.938781+0000","last_change":"2023-07-14T19:15:58.246944+0000","last_active":"2023-07-14T19:15:59.938781+0000","last_peered":"2023-07-14T19:15:59.938781+0000","last_clean":"2023-07-14T19:15:59.938781+0000","last_became_active":"2023-07-14T19:15:58.246672+0000","last_became_peered":"2023-07-14T19:15:58.246672+0000","last_unstale":"2023-07-14T19:15:59.938781+0000","last_undegraded":"2023-07-14T19:15:59.938781+0000","last_fullsized":"2023-07-14T19:15:59.938781+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_clean_scrub_stamp":"2023-07-14T19:15:57.219223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:59:58.787272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-07-14T19:15:59.720933+0000","last_change":"2023-07-14T19:15:58.246761+0000","last_active":"2023-07-14T19:15:59.720933+0000","last_peered":"2023-07-14T19:15:59.720933+0000","last_clean":"2023-07-14T19:15:59.720933+0000","last_became_active":"2023-07-14T19:15:58.246318+0000","last_became_peered":"2023-07-14T19:15:58.246318+0000","last_unstale":"2023-07-14T19:15:59.720933+0000","last_undegraded":"2023-07-14T19:15:59.720933+0000","last_fullsized":"2023-07-14T19:15:59.720933+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_clean_scrub_stamp":"2023-07-14T19:15:57.219223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T22:22:00.801887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-07-14T19:15:59.721079+0000","last_change":"2023-07-14T19:15:58.251465+0000","last_active":"2023-07-14T19:15:59.721079+0000","last_peered":"2023-07-14T19:15:59.721079+0000","last_clean":"2023-07-14T19:15:59.721079+0000","last_became_active":"2023-07-14T19:15:58.251138+0000","last_became_peered":"2023-07-14T19:15:58.251138+0000","last_unstale":"2023-07-14T19:15:59.721079+0000","last_undegraded":"2023-07-14T19:15:59.721079+0000","last_fullsized":"2023-07-14T19:15:59.721079+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_clean_scrub_stamp":"2023-07-14T19:15:57.219223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:53:28.483753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-07-14T19:15:59.573084+0000","last_change":"2023-07-14T19:15:58.244335+0000","last_active":"2023-07-14T19:15:59.573084+0000","last_peered":"2023-07-14T19:15:59.573084+0000","last_clean":"2023-07-14T19:15:59.573084+0000","last_became_active":"2023-07-14T19:15:58.243955+0000","last_became_peered":"2023-07-14T19:15:58.243955+0000","last_unstale":"2023-07-14T19:15:59.573084+0000","last_undegraded":"2023-07-14T19:15:59.573084+0000","last_fullsized":"2023-07-14T19:15:59.573084+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_clean_scrub_stamp":"2023-07-14T19:15:57.219223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T06:54:43.644336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-07-14T19:15:59.723074+0000","last_change":"2023-07-14T19:15:58.244373+0000","last_active":"2023-07-14T19:15:59.723074+0000","last_peered":"2023-07-14T19:15:59.723074+0000","last_clean":"2023-07-14T19:15:59.723074+0000","last_became_active":"2023-07-14T19:15:58.244042+0000","last_became_peered":"2023-07-14T19:15:58.244042+0000","last_unstale":"2023-07-14T19:15:59.723074+0000","last_undegraded":"2023-07-14T19:15:59.723074+0000","last_fullsized":"2023-07-14T19:15:59.723074+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_clean_scrub_stamp":"2023-07-14T19:15:57.219223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T03:40:02.041528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-07-14T19:15:59.938908+0000","last_change":"2023-07-14T19:15:58.240770+0000","last_active":"2023-07-14T19:15:59.938908+0000","last_peered":"2023-07-14T19:15:59.938908+0000","last_clean":"2023-07-14T19:15:59.938908+0000","last_became_active":"2023-07-14T19:15:58.240408+0000","last_became_peered":"2023-07-14T19:15:58.240408+0000","last_unstale":"2023-07-14T19:15:59.938908+0000","last_undegraded":"2023-07-14T19:15:59.938908+0000","last_fullsized":"2023-07-14T19:15:59.938908+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_clean_scrub_stamp":"2023-07-14T19:15:57.219223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T19:45:10.482496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-07-14T19:15:59.721267+0000","last_change":"2023-07-14T19:15:58.247557+0000","last_active":"2023-07-14T19:15:59.721267+0000","last_peered":"2023-07-14T19:15:59.721267+0000","last_clean":"2023-07-14T19:15:59.721267+0000","last_became_active":"2023-07-14T19:15:58.247228+0000","last_became_peered":"2023-07-14T19:15:58.247228+0000","last_unstale":"2023-07-14T19:15:59.721267+0000","last_undegraded":"2023-07-14T19:15:59.721267+0000","last_fullsized":"2023-07-14T19:15:59.721267+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_clean_scrub_stamp":"2023-07-14T19:15:57.219223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T05:18:52.409847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-07-14T19:15:59.722818+0000","last_change":"2023-07-14T19:15:58.247013+0000","last_active":"2023-07-14T19:15:59.722818+0000","last_peered":"2023-07-14T19:15:59.722818+0000","last_clean":"2023-07-14T19:15:59.722818+0000","last_became_active":"2023-07-14T19:15:58.246729+0000","last_became_peered":"2023-07-14T19:15:58.246729+0000","last_unstale":"2023-07-14T19:15:59.722818+0000","last_undegraded":"2023-07-14T19:15:59.722818+0000","last_fullsized":"2023-07-14T19:15:59.722818+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_clean_scrub_stamp":"2023-07-14T19:15:57.219223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T22:17:27.678424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-07-14T19:15:59.231044+0000","last_change":"2023-07-14T19:15:58.243023+0000","last_active":"2023-07-14T19:15:59.231044+0000","last_peered":"2023-07-14T19:15:59.231044+0000","last_clean":"2023-07-14T19:15:59.231044+0000","last_became_active":"2023-07-14T19:15:58.242630+0000","last_became_peered":"2023-07-14T19:15:58.242630+0000","last_unstale":"2023-07-14T19:15:59.231044+0000","last_undegraded":"2023-07-14T19:15:59.231044+0000","last_fullsized":"2023-07-14T19:15:59.231044+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_clean_scrub_stamp":"2023-07-14T19:15:57.219223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T23:27:40.017638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-07-14T19:15:59.723119+0000","last_change":"2023-07-14T19:15:58.246205+0000","last_active":"2023-07-14T19:15:59.723119+0000","last_peered":"2023-07-14T19:15:59.723119+0000","last_clean":"2023-07-14T19:15:59.723119+0000","last_became_active":"2023-07-14T19:15:58.245720+0000","last_became_peered":"2023-07-14T19:15:58.245720+0000","last_unstale":"2023-07-14T19:15:59.723119+0000","last_undegraded":"2023-07-14T19:15:59.723119+0000","last_fullsized":"2023-07-14T19:15:59.723119+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_clean_scrub_stamp":"2023-07-14T19:15:57.219223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T05:22:46.730021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-07-14T19:15:59.722477+0000","last_change":"2023-07-14T19:15:58.248611+0000","last_active":"2023-07-14T19:15:59.722477+0000","last_peered":"2023-07-14T19:15:59.722477+0000","last_clean":"2023-07-14T19:15:59.722477+0000","last_became_active":"2023-07-14T19:15:58.248352+0000","last_became_peered":"2023-07-14T19:15:58.248352+0000","last_unstale":"2023-07-14T19:15:59.722477+0000","last_undegraded":"2023-07-14T19:15:59.722477+0000","last_fullsized":"2023-07-14T19:15:59.722477+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_clean_scrub_stamp":"2023-07-14T19:15:57.219223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T06:43:46.682942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-07-14T19:15:59.722246+0000","last_change":"2023-07-14T19:15:58.243167+0000","last_active":"2023-07-14T19:15:59.722246+0000","last_peered":"2023-07-14T19:15:59.722246+0000","last_clean":"2023-07-14T19:15:59.722246+0000","last_became_active":"2023-07-14T19:15:58.241770+0000","last_became_peered":"2023-07-14T19:15:58.241770+0000","last_unstale":"2023-07-14T19:15:59.722246+0000","last_undegraded":"2023-07-14T19:15:59.722246+0000","last_fullsized":"2023-07-14T19:15:59.722246+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_clean_scrub_stamp":"2023-07-14T19:15:57.219223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T05:56:34.485106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-07-14T19:15:59.938811+0000","last_change":"2023-07-14T19:15:58.243972+0000","last_active":"2023-07-14T19:15:59.938811+0000","last_peered":"2023-07-14T19:15:59.938811+0000","last_clean":"2023-07-14T19:15:59.938811+0000","last_became_active":"2023-07-14T19:15:58.243685+0000","last_became_peered":"2023-07-14T19:15:58.243685+0000","last_unstale":"2023-07-14T19:15:59.938811+0000","last_undegraded":"2023-07-14T19:15:59.938811+0000","last_fullsized":"2023-07-14T19:15:59.938811+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_clean_scrub_stamp":"2023-07-14T19:15:57.219223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:44:54.955614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":442368,"data_stored":4722944,"data_compressed":280807,"data_compressed_allocated":360448,"data_compressed_original":4669440,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":4234,"num_objects":51,"num_object_clones":0,"num_object_copies":153,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":51,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":112,"num_write_kb":33,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":36,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":180224,"data_stored":11216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":96,"ondisk_log_size":96,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":76,"seq":326417514508,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27868,"kb_used_data":848,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717284,"statfs":{"total":95995035648,"available":95966498816,"internally_reserved":0,"allocated":868352,"data_stored":415920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47699999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59899999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.44800000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.436}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61699999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52700000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.626}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65400000000000003}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60799999999999998}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70599999999999996}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69499999999999995}]}]},{"osd":10,"up_from":69,"seq":296352743439,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27940,"kb_used_data":920,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717212,"statfs":{"total":95995035648,"available":95966425088,"internally_reserved":0,"allocated":942080,"data_stored":1008046,"data_compressed":35714,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41999999999999998}]},{"osd":1,"last update":"Fri Jul 14 19:16:00 2023","interfaces":[{"interface":"back","average":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.66500000000000004,"5min":0.66500000000000004,"15min":0.66500000000000004},"last":0.47999999999999998},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.76900000000000002,"5min":0.76900000000000002,"15min":0.76900000000000002},"last":0.442}]},{"osd":2,"last update":"Fri Jul 14 19:16:00 2023","interfaces":[{"interface":"back","average":{"1min":0.67100000000000004,"5min":0.67100000000000004,"15min":0.67100000000000004},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":1.0009999999999999,"5min":1.0009999999999999,"15min":1.0009999999999999},"last":0.60999999999999999},{"interface":"front","average":{"1min":0.751,"5min":0.751,"15min":0.751},"min":{"1min":0.51500000000000001,"5min":0.51500000000000001,"15min":0.51500000000000001},"max":{"1min":1.1379999999999999,"5min":1.1379999999999999,"15min":1.1379999999999999},"last":0.70199999999999996}]},{"osd":3,"last update":"Fri Jul 14 19:16:00 2023","interfaces":[{"interface":"back","average":{"1min":0.70099999999999996,"5min":0.70099999999999996,"15min":0.70099999999999996},"min":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":0.92200000000000004,"5min":0.92200000000000004,"15min":0.92200000000000004},"last":0.53800000000000003},{"interface":"front","average":{"1min":0.71599999999999997,"5min":0.71599999999999997,"15min":0.71599999999999997},"min":{"1min":0.45300000000000001,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":0.90900000000000003,"5min":0.90900000000000003,"15min":0.90900000000000003},"last":0.55100000000000005}]},{"osd":4,"last update":"Fri Jul 14 19:16:00 2023","interfaces":[{"interface":"back","average":{"1min":0.64700000000000002,"5min":0.64700000000000002,"15min":0.64700000000000002},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.83199999999999996,"5min":0.83199999999999996,"15min":0.83199999999999996},"last":0.50800000000000001},{"interface":"front","average":{"1min":0.59999999999999998,"5min":0.59999999999999998,"15min":0.59999999999999998},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.878,"5min":0.878,"15min":0.878},"last":0.46100000000000002}]},{"osd":5,"last update":"Fri Jul 14 19:16:00 2023","interfaces":[{"interface":"back","average":{"1min":0.76000000000000001,"5min":0.76000000000000001,"15min":0.76000000000000001},"min":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"max":{"1min":1.1259999999999999,"5min":1.1259999999999999,"15min":1.1259999999999999},"last":0.77300000000000002},{"interface":"front","average":{"1min":0.77900000000000003,"5min":0.77900000000000003,"15min":0.77900000000000003},"min":{"1min":0.56299999999999994,"5min":0.56299999999999994,"15min":0.56299999999999994},"max":{"1min":1.018,"5min":1.018,"15min":1.018},"last":0.66300000000000003}]},{"osd":6,"last update":"Fri Jul 14 19:16:00 2023","interfaces":[{"interface":"back","average":{"1min":0.82799999999999996,"5min":0.82799999999999996,"15min":0.82799999999999996},"min":{"1min":0.56899999999999995,"5min":0.56899999999999995,"15min":0.56899999999999995},"max":{"1min":1.294,"5min":1.294,"15min":1.294},"last":0.56899999999999995},{"interface":"front","average":{"1min":0.80400000000000005,"5min":0.80400000000000005,"15min":0.80400000000000005},"min":{"1min":0.57999999999999996,"5min":0.57999999999999996,"15min":0.57999999999999996},"max":{"1min":1.0960000000000001,"5min":1.0960000000000001,"15min":1.0960000000000001},"last":0.57999999999999996}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.49099999999999999}]},{"osd":8,"last update":"Fri Jul 14 19:16:00 2023","interfaces":[{"interface":"back","average":{"1min":0.84599999999999997,"5min":0.84599999999999997,"15min":0.84599999999999997},"min":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"max":{"1min":1.0349999999999999,"5min":1.0349999999999999,"15min":1.0349999999999999},"last":0.74099999999999999},{"interface":"front","average":{"1min":0.88900000000000001,"5min":0.88900000000000001,"15min":0.88900000000000001},"min":{"1min":0.60499999999999998,"5min":0.60499999999999998,"15min":0.60499999999999998},"max":{"1min":1.369,"5min":1.369,"15min":1.369},"last":0.67400000000000004}]},{"osd":9,"last update":"Fri Jul 14 19:16:00 2023","interfaces":[{"interface":"back","average":{"1min":0.88600000000000001,"5min":0.88600000000000001,"15min":0.88600000000000001},"min":{"1min":0.65000000000000002,"5min":0.65000000000000002,"15min":0.65000000000000002},"max":{"1min":1.1759999999999999,"5min":1.1759999999999999,"15min":1.1759999999999999},"last":0.65000000000000002},{"interface":"front","average":{"1min":0.88200000000000001,"5min":0.88200000000000001,"15min":0.88200000000000001},"min":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":1.468,"5min":1.468,"15min":1.468},"last":0.52800000000000002}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.80900000000000005}]}]},{"osd":9,"up_from":64,"seq":274877906964,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27888,"kb_used_data":872,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717264,"statfs":{"total":95995035648,"available":95966478336,"internally_reserved":0,"allocated":892928,"data_stored":416828,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jul 14 19:15:35 2023","interfaces":[{"interface":"back","average":{"1min":0.48699999999999999,"5min":0.48699999999999999,"15min":0.48699999999999999},"min":{"1min":0.29999999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.82099999999999995,"5min":0.82099999999999995,"15min":0.82099999999999995},"last":0.30099999999999999},{"interface":"front","average":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.46100000000000002},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.65300000000000002,"5min":0.65300000000000002,"15min":0.65300000000000002},"last":0.32000000000000001}]},{"osd":1,"last update":"Fri Jul 14 19:15:35 2023","interfaces":[{"interface":"back","average":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"min":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":1.0149999999999999,"5min":1.0149999999999999,"15min":1.0149999999999999},"last":0.53800000000000003},{"interface":"front","average":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":0.65700000000000003},"min":{"1min":0.53700000000000003,"5min":0.53700000000000003,"15min":0.53700000000000003},"max":{"1min":0.85199999999999998,"5min":0.85199999999999998,"15min":0.85199999999999998},"last":0.47899999999999998}]},{"osd":2,"last update":"Fri Jul 14 19:15:35 2023","interfaces":[{"interface":"back","average":{"1min":0.65800000000000003,"5min":0.65800000000000003,"15min":0.65800000000000003},"min":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":0.92200000000000004,"5min":0.92200000000000004,"15min":0.92200000000000004},"last":0.58399999999999996},{"interface":"front","average":{"1min":0.65800000000000003,"5min":0.65800000000000003,"15min":0.65800000000000003},"min":{"1min":0.46500000000000002,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":0.91000000000000003,"5min":0.91000000000000003,"15min":0.91000000000000003},"last":0.55100000000000005}]},{"osd":3,"last update":"Fri Jul 14 19:15:35 2023","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":0.876,"5min":0.876,"15min":0.876},"last":0.41899999999999998},{"interface":"front","average":{"1min":0.61699999999999999,"5min":0.61699999999999999,"15min":0.61699999999999999},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.96499999999999997,"5min":0.96499999999999997,"15min":0.96499999999999997},"last":0.34999999999999998}]},{"osd":4,"last update":"Fri Jul 14 19:15:35 2023","interfaces":[{"interface":"back","average":{"1min":0.68500000000000005,"5min":0.68500000000000005,"15min":0.68500000000000005},"min":{"1min":0.48999999999999999,"5min":0.48999999999999999,"15min":0.48999999999999999},"max":{"1min":1.0700000000000001,"5min":1.0700000000000001,"15min":1.0700000000000001},"last":0.50600000000000001},{"interface":"front","average":{"1min":0.67000000000000004,"5min":0.67000000000000004,"15min":0.67000000000000004},"min":{"1min":0.47199999999999998,"5min":0.47199999999999998,"15min":0.47199999999999998},"max":{"1min":0.94099999999999995,"5min":0.94099999999999995,"15min":0.94099999999999995},"last":0.496}]},{"osd":5,"last update":"Fri Jul 14 19:15:35 2023","interfaces":[{"interface":"back","average":{"1min":0.76500000000000001,"5min":0.76500000000000001,"15min":0.76500000000000001},"min":{"1min":0.43099999999999999,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":1.1140000000000001,"5min":1.1140000000000001,"15min":1.1140000000000001},"last":0.59699999999999998},{"interface":"front","average":{"1min":0.72399999999999998,"5min":0.72399999999999998,"15min":0.72399999999999998},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":1.018,"5min":1.018,"15min":1.018},"last":0.60999999999999999}]},{"osd":6,"last update":"Fri Jul 14 19:15:35 2023","interfaces":[{"interface":"back","average":{"1min":0.79800000000000004,"5min":0.79800000000000004,"15min":0.79800000000000004},"min":{"1min":0.44800000000000001,"5min":0.44800000000000001,"15min":0.44800000000000001},"max":{"1min":1.6839999999999999,"5min":1.6839999999999999,"15min":1.6839999999999999},"last":0.52500000000000002},{"interface":"front","average":{"1min":0.76700000000000002,"5min":0.76700000000000002,"15min":0.76700000000000002},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":1.145,"5min":1.145,"15min":1.145},"last":0.439}]},{"osd":7,"last update":"Fri Jul 14 19:15:35 2023","interfaces":[{"interface":"back","average":{"1min":0.78300000000000003,"5min":0.78300000000000003,"15min":0.78300000000000003},"min":{"1min":0.48699999999999999,"5min":0.48699999999999999,"15min":0.48699999999999999},"max":{"1min":1.0800000000000001,"5min":1.0800000000000001,"15min":1.0800000000000001},"last":0.56399999999999995},{"interface":"front","average":{"1min":0.83099999999999996,"5min":0.83099999999999996,"15min":0.83099999999999996},"min":{"1min":0.58499999999999996,"5min":0.58499999999999996,"15min":0.58499999999999996},"max":{"1min":1.0469999999999999,"5min":1.0469999999999999,"15min":1.0469999999999999},"last":0.621}]},{"osd":8,"last update":"Fri Jul 14 19:15:35 2023","interfaces":[{"interface":"back","average":{"1min":0.871,"5min":0.871,"15min":0.871},"min":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"max":{"1min":1.242,"5min":1.242,"15min":1.242},"last":0.629},{"interface":"front","average":{"1min":0.82699999999999996,"5min":0.82699999999999996,"15min":0.82699999999999996},"min":{"1min":0.58499999999999996,"5min":0.58499999999999996,"15min":0.58499999999999996},"max":{"1min":1.173,"5min":1.173,"15min":1.173},"last":0.65100000000000002}]},{"osd":10,"last update":"Fri Jul 14 19:15:58 2023","interfaces":[{"interface":"back","average":{"1min":0.79200000000000004,"5min":0.79200000000000004,"15min":0.79200000000000004},"min":{"1min":0.54900000000000004,"5min":0.54900000000000004,"15min":0.54900000000000004},"max":{"1min":1.0820000000000001,"5min":1.0820000000000001,"15min":1.0820000000000001},"last":0.64200000000000002},{"interface":"front","average":{"1min":0.79100000000000004,"5min":0.79100000000000004,"15min":0.79100000000000004},"min":{"1min":0.59599999999999997,"5min":0.59599999999999997,"15min":0.59599999999999997},"max":{"1min":1.044,"5min":1.044,"15min":1.044},"last":0.66100000000000003}]}]},{"osd":8,"up_from":57,"seq":244813135897,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27880,"kb_used_data":864,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717272,"statfs":{"total":95995035648,"available":95966486528,"internally_reserved":0,"allocated":884736,"data_stored":416327,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jul 14 19:15:13 2023","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.52100000000000002},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.78600000000000003,"5min":0.78600000000000003,"15min":0.78600000000000003},"last":0.505},{"interface":"front","average":{"1min":0.58599999999999997,"5min":0.58599999999999997,"15min":0.58599999999999997},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.80500000000000005,"5min":0.80500000000000005,"15min":0.80500000000000005},"last":0.54800000000000004}]},{"osd":1,"last update":"Fri Jul 14 19:15:13 2023","interfaces":[{"interface":"back","average":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"min":{"1min":0.375,"5min":0.375,"15min":0.375},"max":{"1min":1.0069999999999999,"5min":1.0069999999999999,"15min":1.0069999999999999},"last":0.69099999999999995},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.59299999999999997,"15min":0.59299999999999997},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.91500000000000004,"5min":0.91500000000000004,"15min":0.91500000000000004},"last":0.78400000000000003}]},{"osd":2,"last update":"Fri Jul 14 19:15:13 2023","interfaces":[{"interface":"back","average":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.54300000000000004},"last":0.46899999999999997},{"interface":"front","average":{"1min":0.48699999999999999,"5min":0.48699999999999999,"15min":0.48699999999999999},"min":{"1min":0.31,"5min":0.31,"15min":0.31},"max":{"1min":0.69399999999999995,"5min":0.69399999999999995,"15min":0.69399999999999995},"last":0.66700000000000004}]},{"osd":3,"last update":"Fri Jul 14 19:15:13 2023","interfaces":[{"interface":"back","average":{"1min":0.60999999999999999,"5min":0.60999999999999999,"15min":0.60999999999999999},"min":{"1min":0.44,"5min":0.44,"15min":0.44},"max":{"1min":0.84799999999999998,"5min":0.84799999999999998,"15min":0.84799999999999998},"last":0.64700000000000002},{"interface":"front","average":{"1min":0.64800000000000002,"5min":0.64800000000000002,"15min":0.64800000000000002},"min":{"1min":0.49099999999999999,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":0.80700000000000005,"5min":0.80700000000000005,"15min":0.80700000000000005},"last":0.61699999999999999}]},{"osd":4,"last update":"Fri Jul 14 19:15:13 2023","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.58099999999999996,"15min":0.58099999999999996},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":1.069,"5min":1.069,"15min":1.069},"last":0.74299999999999999},{"interface":"front","average":{"1min":0.68500000000000005,"5min":0.68500000000000005,"15min":0.68500000000000005},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":1.474,"5min":1.474,"15min":1.474},"last":0.75800000000000001}]},{"osd":5,"last update":"Fri Jul 14 19:15:13 2023","interfaces":[{"interface":"back","average":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":0.75700000000000001,"5min":0.75700000000000001,"15min":0.75700000000000001},"last":0.72299999999999998},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.56899999999999995,"15min":0.56899999999999995},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":1.0349999999999999,"5min":1.0349999999999999,"15min":1.0349999999999999},"last":0.71099999999999997}]},{"osd":6,"last update":"Fri Jul 14 19:15:13 2023","interfaces":[{"interface":"back","average":{"1min":0.66400000000000003,"5min":0.66400000000000003,"15min":0.66400000000000003},"min":{"1min":0.44800000000000001,"5min":0.44800000000000001,"15min":0.44800000000000001},"max":{"1min":0.86299999999999999,"5min":0.86299999999999999,"15min":0.86299999999999999},"last":0.84299999999999997},{"interface":"front","average":{"1min":0.73199999999999998,"5min":0.73199999999999998,"15min":0.73199999999999998},"min":{"1min":0.43099999999999999,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":1.1399999999999999,"5min":1.1399999999999999,"15min":1.1399999999999999},"last":0.82999999999999996}]},{"osd":7,"last update":"Fri Jul 14 19:15:13 2023","interfaces":[{"interface":"back","average":{"1min":0.72599999999999998,"5min":0.72599999999999998,"15min":0.72599999999999998},"min":{"1min":0.48399999999999999,"5min":0.48399999999999999,"15min":0.48399999999999999},"max":{"1min":1.1879999999999999,"5min":1.1879999999999999,"15min":1.1879999999999999},"last":0.93500000000000005},{"interface":"front","average":{"1min":0.72799999999999998,"5min":0.72799999999999998,"15min":0.72799999999999998},"min":{"1min":0.5,"5min":0.5,"15min":0.5},"max":{"1min":1.093,"5min":1.093,"15min":1.093},"last":0.86399999999999999}]},{"osd":9,"last update":"Fri Jul 14 19:15:33 2023","interfaces":[{"interface":"back","average":{"1min":0.754,"5min":0.754,"15min":0.754},"min":{"1min":0.55600000000000005,"5min":0.55600000000000005,"15min":0.55600000000000005},"max":{"1min":0.97399999999999998,"5min":0.97399999999999998,"15min":0.97399999999999998},"last":0.89800000000000002},{"interface":"front","average":{"1min":0.75600000000000001,"5min":0.75600000000000001,"15min":0.75600000000000001},"min":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"max":{"1min":0.97099999999999997,"5min":0.97099999999999997,"15min":0.97099999999999997},"last":0.88700000000000001}]},{"osd":10,"last update":"Fri Jul 14 19:15:55 2023","interfaces":[{"interface":"back","average":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"min":{"1min":0.52700000000000002,"5min":0.52700000000000002,"15min":0.52700000000000002},"max":{"1min":1.006,"5min":1.006,"15min":1.006},"last":0.877},{"interface":"front","average":{"1min":0.73599999999999999,"5min":0.73599999999999999,"15min":0.73599999999999999},"min":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"max":{"1min":0.95999999999999996,"5min":0.95999999999999996,"15min":0.95999999999999996},"last":0.95399999999999996}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.91700000000000004}]}]},{"osd":7,"up_from":50,"seq":214748364828,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27916,"kb_used_data":896,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717236,"statfs":{"total":95995035648,"available":95966449664,"internally_reserved":0,"allocated":917504,"data_stored":1006597,"data_compressed":35714,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jul 14 19:15:51 2023","interfaces":[{"interface":"back","average":{"1min":0.57199999999999995,"5min":0.53300000000000003,"15min":0.52700000000000002},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.82999999999999996,"5min":0.82999999999999996,"15min":0.82999999999999996},"last":0.5},{"interface":"front","average":{"1min":0.55200000000000005,"5min":0.55600000000000005,"15min":0.55600000000000005},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.78100000000000003,"5min":0.89300000000000002,"15min":0.89300000000000002},"last":0.51800000000000002}]},{"osd":1,"last update":"Fri Jul 14 19:15:51 2023","interfaces":[{"interface":"back","average":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"min":{"1min":0.25700000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.67100000000000004,"5min":0.67100000000000004,"15min":0.67100000000000004},"last":0.44},{"interface":"front","average":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"min":{"1min":0.254,"5min":0.254,"15min":0.254},"max":{"1min":0.69099999999999995,"5min":0.91300000000000003,"15min":0.91300000000000003},"last":0.36299999999999999}]},{"osd":2,"last update":"Fri Jul 14 19:15:51 2023","interfaces":[{"interface":"back","average":{"1min":0.60199999999999998,"5min":0.59299999999999997,"15min":0.59099999999999997},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.78600000000000003,"5min":0.83999999999999997,"15min":0.83999999999999997},"last":0.58899999999999997},{"interface":"front","average":{"1min":0.65600000000000003,"5min":0.628,"15min":0.623},"min":{"1min":0.43099999999999999,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":0.96999999999999997,"5min":0.96999999999999997,"15min":0.96999999999999997},"last":0.79400000000000004}]},{"osd":3,"last update":"Fri Jul 14 19:15:51 2023","interfaces":[{"interface":"back","average":{"1min":0.61699999999999999,"5min":0.56599999999999995,"15min":0.55800000000000005},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":0.89200000000000002,"5min":0.89200000000000002,"15min":0.89200000000000002},"last":0.72299999999999998},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.80300000000000005,"5min":0.91700000000000004,"15min":0.91700000000000004},"last":0.56299999999999994}]},{"osd":4,"last update":"Fri Jul 14 19:15:51 2023","interfaces":[{"interface":"back","average":{"1min":0.53800000000000003,"5min":0.54000000000000004,"15min":0.54000000000000004},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.81499999999999995,"5min":0.86399999999999999,"15min":0.86399999999999999},"last":0.40600000000000003},{"interface":"front","average":{"1min":0.497,"5min":0.48099999999999998,"15min":0.47899999999999998},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":0.80600000000000005,"5min":0.80600000000000005,"15min":0.80600000000000005},"last":0.33100000000000002}]},{"osd":5,"last update":"Fri Jul 14 19:15:51 2023","interfaces":[{"interface":"back","average":{"1min":0.65100000000000002,"5min":0.67800000000000005,"15min":0.68200000000000005},"min":{"1min":0.43099999999999999,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":0.90600000000000003,"5min":1.355,"15min":1.355},"last":0.80600000000000005},{"interface":"front","average":{"1min":0.73599999999999999,"5min":0.71499999999999997,"15min":0.71099999999999997},"min":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":1.048,"5min":1.3009999999999999,"15min":1.3009999999999999},"last":0.625}]},{"osd":6,"last update":"Fri Jul 14 19:15:51 2023","interfaces":[{"interface":"back","average":{"1min":0.69599999999999995,"5min":0.69399999999999995,"15min":0.69399999999999995},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":0.93200000000000005,"5min":1.2310000000000001,"15min":1.2310000000000001},"last":0.61099999999999999},{"interface":"front","average":{"1min":0.70399999999999996,"5min":0.66800000000000004,"15min":0.66200000000000003},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":1.069,"5min":1.1140000000000001,"15min":1.1140000000000001},"last":0.68899999999999995}]},{"osd":8,"last update":"Fri Jul 14 19:15:13 2023","interfaces":[{"interface":"back","average":{"1min":0.74399999999999999,"5min":0.74399999999999999,"15min":0.74399999999999999},"min":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"max":{"1min":1.0620000000000001,"5min":1.0620000000000001,"15min":1.0620000000000001},"last":0.81999999999999995},{"interface":"front","average":{"1min":0.72999999999999998,"5min":0.72999999999999998,"15min":0.72999999999999998},"min":{"1min":0.495,"5min":0.495,"15min":0.495},"max":{"1min":0.93500000000000005,"5min":0.93500000000000005,"15min":0.93500000000000005},"last":0.73799999999999999}]},{"osd":9,"last update":"Fri Jul 14 19:15:35 2023","interfaces":[{"interface":"back","average":{"1min":0.754,"5min":0.754,"15min":0.754},"min":{"1min":0.48499999999999999,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":1.1599999999999999,"5min":1.1599999999999999,"15min":1.1599999999999999},"last":0.65400000000000003},{"interface":"front","average":{"1min":0.77800000000000002,"5min":0.77800000000000002,"15min":0.77800000000000002},"min":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":1.214,"5min":1.214,"15min":1.214},"last":0.75800000000000001}]},{"osd":10,"last update":"Fri Jul 14 19:15:55 2023","interfaces":[{"interface":"back","average":{"1min":0.753,"5min":0.753,"15min":0.753},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":1.0960000000000001,"5min":1.0960000000000001,"15min":1.0960000000000001},"last":0.496},{"interface":"front","average":{"1min":0.748,"5min":0.748,"15min":0.748},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":1.0980000000000001,"5min":1.0980000000000001,"15min":1.0980000000000001},"last":0.52100000000000002}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67500000000000004}]}]},{"osd":6,"up_from":45,"seq":193273528353,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27872,"kb_used_data":848,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717280,"statfs":{"total":95995035648,"available":95966494720,"internally_reserved":0,"allocated":868352,"data_stored":415920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jul 14 19:15:32 2023","interfaces":[{"interface":"back","average":{"1min":0.47699999999999998,"5min":0.42099999999999999,"15min":0.41099999999999998},"min":{"1min":0.34100000000000003,"5min":0.222,"15min":0.222},"max":{"1min":0.77100000000000002,"5min":0.77100000000000002,"15min":0.77100000000000002},"last":0.29499999999999998},{"interface":"front","average":{"1min":0.53200000000000003,"5min":0.45800000000000002,"15min":0.44600000000000001},"min":{"1min":0.32600000000000001,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":1.1459999999999999,"5min":1.1459999999999999,"15min":1.1459999999999999},"last":0.32700000000000001}]},{"osd":1,"last update":"Fri Jul 14 19:15:32 2023","interfaces":[{"interface":"back","average":{"1min":0.66300000000000003,"5min":0.57799999999999996,"15min":0.56399999999999995},"min":{"1min":0.51800000000000002,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.89600000000000002,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.371},{"interface":"front","average":{"1min":0.65700000000000003,"5min":0.69599999999999995,"15min":0.70199999999999996},"min":{"1min":0.51500000000000001,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":0.85299999999999998,"5min":3.1930000000000001,"15min":3.1930000000000001},"last":0.47099999999999997}]},{"osd":2,"last update":"Fri Jul 14 19:15:32 2023","interfaces":[{"interface":"back","average":{"1min":0.67100000000000004,"5min":0.59999999999999998,"15min":0.58799999999999997},"min":{"1min":0.436,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.93300000000000005,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.56999999999999995},{"interface":"front","average":{"1min":0.625,"5min":0.58399999999999996,"15min":0.57699999999999996},"min":{"1min":0.435,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.84699999999999998,"5min":0.85299999999999998,"15min":0.85299999999999998},"last":0.59899999999999998}]},{"osd":3,"last update":"Fri Jul 14 19:15:32 2023","interfaces":[{"interface":"back","average":{"1min":0.54200000000000004,"5min":0.504,"15min":0.498},"min":{"1min":0.35199999999999998,"5min":0.314,"15min":0.314},"max":{"1min":0.82799999999999996,"5min":0.82799999999999996,"15min":0.82799999999999996},"last":0.438},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.54700000000000004,"15min":0.54100000000000004},"min":{"1min":0.36799999999999999,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.78700000000000003,"5min":0.96199999999999997,"15min":0.96199999999999997},"last":0.42299999999999999}]},{"osd":4,"last update":"Fri Jul 14 19:15:32 2023","interfaces":[{"interface":"back","average":{"1min":0.73299999999999998,"5min":0.64200000000000002,"15min":0.627},"min":{"1min":0.51200000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.91500000000000004,"5min":0.91500000000000004,"15min":0.91500000000000004},"last":0.45900000000000002},{"interface":"front","average":{"1min":0.749,"5min":0.67000000000000004,"15min":0.65700000000000003},"min":{"1min":0.52600000000000002,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":0.97499999999999998,"5min":1.0309999999999999,"15min":1.0309999999999999},"last":0.55100000000000005}]},{"osd":5,"last update":"Fri Jul 14 19:15:32 2023","interfaces":[{"interface":"back","average":{"1min":0.73799999999999999,"5min":0.75800000000000001,"15min":0.76100000000000001},"min":{"1min":0.48799999999999999,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":1.085,"5min":3.1299999999999999,"15min":3.1299999999999999},"last":0.58199999999999996},{"interface":"front","average":{"1min":0.78200000000000003,"5min":0.68600000000000005,"15min":0.67000000000000004},"min":{"1min":0.53900000000000003,"5min":0.38,"15min":0.38},"max":{"1min":1.069,"5min":1.105,"15min":1.105},"last":0.67600000000000005}]},{"osd":7,"last update":"Fri Jul 14 19:15:56 2023","interfaces":[{"interface":"back","average":{"1min":0.79000000000000004,"5min":0.84199999999999997,"15min":0.85099999999999998},"min":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"max":{"1min":1.0489999999999999,"5min":3.25,"15min":3.25},"last":0.53500000000000003},{"interface":"front","average":{"1min":0.79600000000000004,"5min":0.749,"15min":0.74099999999999999},"min":{"1min":0.58299999999999996,"5min":0.54000000000000004,"15min":0.54000000000000004},"max":{"1min":1.0589999999999999,"5min":1.0589999999999999,"15min":1.0589999999999999},"last":0.61499999999999999}]},{"osd":8,"last update":"Fri Jul 14 19:15:14 2023","interfaces":[{"interface":"back","average":{"1min":0.74099999999999999,"5min":0.74099999999999999,"15min":0.74099999999999999},"min":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":1.044,"5min":1.044,"15min":1.044},"last":0.64700000000000002},{"interface":"front","average":{"1min":0.94999999999999996,"5min":0.94999999999999996,"15min":0.94999999999999996},"min":{"1min":0.57999999999999996,"5min":0.57999999999999996,"15min":0.57999999999999996},"max":{"1min":3.226,"5min":3.226,"15min":3.226},"last":0.66400000000000003}]},{"osd":9,"last update":"Fri Jul 14 19:15:38 2023","interfaces":[{"interface":"back","average":{"1min":0.85099999999999998,"5min":0.85099999999999998,"15min":0.85099999999999998},"min":{"1min":0.502,"5min":0.502,"15min":0.502},"max":{"1min":1.123,"5min":1.123,"15min":1.123},"last":0.51800000000000002},{"interface":"front","average":{"1min":0.83499999999999996,"5min":0.83499999999999996,"15min":0.83499999999999996},"min":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":1.135,"5min":1.135,"15min":1.135},"last":0.5}]},{"osd":10,"last update":"Fri Jul 14 19:15:56 2023","interfaces":[{"interface":"back","average":{"1min":0.85899999999999999,"5min":0.85899999999999999,"15min":0.85899999999999999},"min":{"1min":0.63,"5min":0.63,"15min":0.63},"max":{"1min":1.0840000000000001,"5min":1.0840000000000001,"15min":1.0840000000000001},"last":0.63},{"interface":"front","average":{"1min":0.85699999999999998,"5min":0.85699999999999998,"15min":0.85699999999999998},"min":{"1min":0.64100000000000001,"5min":0.64100000000000001,"15min":0.64100000000000001},"max":{"1min":1.0820000000000001,"5min":1.0820000000000001,"15min":1.0820000000000001},"last":0.70599999999999996}]}]},{"osd":1,"up_from":13,"seq":55834574902,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27872,"kb_used_data":856,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717280,"statfs":{"total":95995035648,"available":95966494720,"internally_reserved":0,"allocated":876544,"data_stored":417211,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jul 14 19:15:49 2023","interfaces":[{"interface":"back","average":{"1min":0.60199999999999998,"5min":0.51100000000000001,"15min":0.45400000000000001},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.81999999999999995,"5min":1.149,"15min":1.149},"last":0.42099999999999999},{"interface":"front","average":{"1min":0.63500000000000001,"5min":0.52500000000000002,"15min":0.46999999999999997},"min":{"1min":0.36099999999999999,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.97099999999999997,"5min":1.0740000000000001,"15min":1.0740000000000001},"last":0.51200000000000001}]},{"osd":2,"last update":"Fri Jul 14 19:15:09 2023","interfaces":[{"interface":"back","average":{"1min":0.51400000000000001,"5min":0.51300000000000001,"15min":0.51200000000000001},"min":{"1min":0.39300000000000002,"5min":0.372,"15min":0.372},"max":{"1min":0.70699999999999996,"5min":0.875,"15min":0.875},"last":0.35799999999999998},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.51700000000000002,"15min":0.505},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.79500000000000004,"5min":0.79500000000000004,"15min":0.79500000000000004},"last":0.45500000000000002}]},{"osd":3,"last update":"Fri Jul 14 19:15:28 2023","interfaces":[{"interface":"back","average":{"1min":0.65400000000000003,"5min":0.58199999999999996,"15min":0.56499999999999995},"min":{"1min":0.45500000000000002,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":1.1659999999999999,"5min":1.1659999999999999,"15min":1.1659999999999999},"last":0.436},{"interface":"front","average":{"1min":0.629,"5min":0.57799999999999996,"15min":0.56399999999999995},"min":{"1min":0.41599999999999998,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":1.177,"5min":1.177,"15min":1.177},"last":0.46899999999999997}]},{"osd":4,"last update":"Fri Jul 14 19:15:49 2023","interfaces":[{"interface":"back","average":{"1min":0.52800000000000002,"5min":0.435,"15min":0.42299999999999999},"min":{"1min":0.32100000000000001,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.77300000000000002,"5min":0.77300000000000002,"15min":0.77300000000000002},"last":0.38300000000000001},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.47099999999999997,"15min":0.45500000000000002},"min":{"1min":0.38500000000000001,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.80300000000000005,"5min":0.80300000000000005,"15min":0.80300000000000005},"last":0.48899999999999999}]},{"osd":5,"last update":"Fri Jul 14 19:15:09 2023","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.52600000000000002,"15min":0.52300000000000002},"min":{"1min":0.41799999999999998,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.71199999999999997,"5min":0.71199999999999997,"15min":0.71199999999999997},"last":0.40500000000000003},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.58499999999999996,"15min":0.58599999999999997},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.82699999999999996,"5min":0.82699999999999996,"15min":0.82699999999999996},"last":0.52300000000000002}]},{"osd":6,"last update":"Fri Jul 14 19:15:36 2023","interfaces":[{"interface":"back","average":{"1min":0.65400000000000003,"5min":0.63900000000000001,"15min":0.63700000000000001},"min":{"1min":0.42699999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.84699999999999998,"5min":1.266,"15min":1.266},"last":0.54000000000000004},{"interface":"front","average":{"1min":0.64800000000000002,"5min":0.624,"15min":0.62},"min":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":0.93500000000000005,"5min":1.083,"15min":1.083},"last":0.56299999999999994}]},{"osd":7,"last update":"Fri Jul 14 19:15:56 2023","interfaces":[{"interface":"back","average":{"1min":0.65600000000000003,"5min":0.55000000000000004,"15min":0.53200000000000003},"min":{"1min":0.378,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.86799999999999999,"5min":0.86799999999999999,"15min":0.86799999999999999},"last":0.55200000000000005},{"interface":"front","average":{"1min":0.68799999999999994,"5min":0.61199999999999999,"15min":0.59899999999999998},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.93300000000000005,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.58099999999999996}]},{"osd":8,"last update":"Fri Jul 14 19:15:15 2023","interfaces":[{"interface":"back","average":{"1min":0.65800000000000003,"5min":0.65800000000000003,"15min":0.65800000000000003},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.89100000000000001,"5min":0.89100000000000001,"15min":0.89100000000000001},"last":0.61899999999999999},{"interface":"front","average":{"1min":0.63500000000000001,"5min":0.63500000000000001,"15min":0.63500000000000001},"min":{"1min":0.47999999999999998,"5min":0.47999999999999998,"15min":0.47999999999999998},"max":{"1min":0.84099999999999997,"5min":0.84099999999999997,"15min":0.84099999999999997},"last":0.60799999999999998}]},{"osd":9,"last update":"Fri Jul 14 19:15:36 2023","interfaces":[{"interface":"back","average":{"1min":0.72799999999999998,"5min":0.72799999999999998,"15min":0.72799999999999998},"min":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":0.94899999999999995,"5min":0.94899999999999995,"15min":0.94899999999999995},"last":0.63200000000000001},{"interface":"front","average":{"1min":0.747,"5min":0.747,"15min":0.747},"min":{"1min":0.53900000000000003,"5min":0.53900000000000003,"15min":0.53900000000000003},"max":{"1min":0.93400000000000005,"5min":0.93400000000000005,"15min":0.93400000000000005},"last":0.66000000000000003}]},{"osd":10,"last update":"Fri Jul 14 19:15:56 2023","interfaces":[{"interface":"back","average":{"1min":0.65800000000000003,"5min":0.65800000000000003,"15min":0.65800000000000003},"min":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":0.86799999999999999,"5min":0.86799999999999999,"15min":0.86799999999999999},"last":0.64000000000000001},{"interface":"front","average":{"1min":0.72599999999999998,"5min":0.72599999999999998,"15min":0.72599999999999998},"min":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"max":{"1min":0.88300000000000001,"5min":0.88300000000000001,"15min":0.88300000000000001},"last":0.59799999999999998}]}]},{"osd":0,"up_from":8,"seq":34359738427,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27876,"kb_used_data":852,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717276,"statfs":{"total":95995035648,"available":95966490624,"internally_reserved":0,"allocated":872448,"data_stored":416010,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Jul 14 19:15:50 2023","interfaces":[{"interface":"back","average":{"1min":0.627,"5min":0.55900000000000005,"15min":0.52900000000000003},"min":{"1min":0.49099999999999999,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.88,"5min":0.95799999999999996,"15min":0.95799999999999996},"last":0.44600000000000001},{"interface":"front","average":{"1min":0.64800000000000002,"5min":0.56399999999999995,"15min":0.54200000000000004},"min":{"1min":0.40600000000000003,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.91700000000000004,"5min":0.99099999999999999,"15min":0.99099999999999999},"last":0.495}]},{"osd":2,"last update":"Fri Jul 14 19:15:04 2023","interfaces":[{"interface":"back","average":{"1min":0.58699999999999997,"5min":0.52200000000000002,"15min":0.50600000000000001},"min":{"1min":0.35699999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.90500000000000003,"5min":0.90500000000000003,"15min":0.90500000000000003},"last":0.41099999999999998},{"interface":"front","average":{"1min":0.63300000000000001,"5min":0.53900000000000003,"15min":0.51700000000000002},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":1.044,"5min":1.044,"15min":1.044},"last":0.51800000000000002}]},{"osd":3,"last update":"Fri Jul 14 19:14:30 2023","interfaces":[{"interface":"back","average":{"1min":0.437,"5min":0.40100000000000002,"15min":0.39500000000000002},"min":{"1min":0.315,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.623,"5min":0.623,"15min":0.623},"last":0.35799999999999998},{"interface":"front","average":{"1min":0.497,"5min":0.47299999999999998,"15min":0.46899999999999997},"min":{"1min":0.34100000000000003,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.59999999999999998,"5min":0.63200000000000001,"15min":0.63200000000000001},"last":0.375}]},{"osd":4,"last update":"Fri Jul 14 19:15:50 2023","interfaces":[{"interface":"back","average":{"1min":0.65900000000000003,"5min":0.57899999999999996,"15min":0.55100000000000005},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":0.92900000000000005,"5min":0.95399999999999996,"15min":0.95399999999999996},"last":0.59499999999999997},{"interface":"front","average":{"1min":0.63200000000000001,"5min":0.59199999999999997,"15min":0.56999999999999995},"min":{"1min":0.44,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.86099999999999999,"5min":0.93400000000000005,"15min":0.93400000000000005},"last":0.623}]},{"osd":5,"last update":"Fri Jul 14 19:15:14 2023","interfaces":[{"interface":"back","average":{"1min":0.71099999999999997,"5min":0.63100000000000001,"15min":0.61799999999999999},"min":{"1min":0.48099999999999998,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":1.0089999999999999,"5min":1.0169999999999999,"15min":1.0169999999999999},"last":0.66400000000000003},{"interface":"front","average":{"1min":0.72099999999999997,"5min":0.60699999999999998,"15min":0.58799999999999997},"min":{"1min":0.55100000000000005,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":1.073,"5min":1.073,"15min":1.073},"last":0.67700000000000005}]},{"osd":6,"last update":"Fri Jul 14 19:15:33 2023","interfaces":[{"interface":"back","average":{"1min":0.53800000000000003,"5min":0.53200000000000003,"15min":0.53100000000000003},"min":{"1min":0.30499999999999999,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.79600000000000004,"5min":0.79600000000000004,"15min":0.79600000000000004},"last":0.53600000000000003},{"interface":"front","average":{"1min":0.70599999999999996,"5min":0.61599999999999999,"15min":0.60099999999999998},"min":{"1min":0.432,"5min":0.312,"15min":0.312},"max":{"1min":1.0169999999999999,"5min":1.105,"15min":1.105},"last":0.55200000000000005}]},{"osd":7,"last update":"Fri Jul 14 19:15:55 2023","interfaces":[{"interface":"back","average":{"1min":0.77400000000000002,"5min":0.72499999999999998,"15min":0.71699999999999997},"min":{"1min":0.55700000000000005,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":1.0629999999999999,"5min":1.0629999999999999,"15min":1.0629999999999999},"last":0.81299999999999994},{"interface":"front","average":{"1min":0.68700000000000006,"5min":0.69999999999999996,"15min":0.70199999999999996},"min":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":0.99299999999999999,"5min":1.0229999999999999,"15min":1.0229999999999999},"last":0.77200000000000002}]},{"osd":8,"last update":"Fri Jul 14 19:15:14 2023","interfaces":[{"interface":"back","average":{"1min":0.69299999999999995,"5min":0.69299999999999995,"15min":0.69299999999999995},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.91100000000000003,"5min":0.91100000000000003,"15min":0.91100000000000003},"last":0.58299999999999996},{"interface":"front","average":{"1min":0.84599999999999997,"5min":0.84599999999999997,"15min":0.84599999999999997},"min":{"1min":0.623,"5min":0.623,"15min":0.623},"max":{"1min":1.097,"5min":1.097,"15min":1.097},"last":0.70299999999999996}]},{"osd":9,"last update":"Fri Jul 14 19:15:37 2023","interfaces":[{"interface":"back","average":{"1min":0.73099999999999998,"5min":0.73099999999999998,"15min":0.73099999999999998},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":1.111,"5min":1.111,"15min":1.111},"last":0.56699999999999995},{"interface":"front","average":{"1min":0.70399999999999996,"5min":0.70399999999999996,"15min":0.70399999999999996},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.98099999999999998,"5min":0.98099999999999998,"15min":0.98099999999999998},"last":0.64900000000000002}]},{"osd":10,"last update":"Fri Jul 14 19:15:59 2023","interfaces":[{"interface":"back","average":{"1min":0.82799999999999996,"5min":0.82799999999999996,"15min":0.82799999999999996},"min":{"1min":0.66200000000000003,"5min":0.66200000000000003,"15min":0.66200000000000003},"max":{"1min":1.1379999999999999,"5min":1.1379999999999999,"15min":1.1379999999999999},"last":0.745},{"interface":"front","average":{"1min":0.77800000000000002,"5min":0.77800000000000002,"15min":0.77800000000000002},"min":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"max":{"1min":0.99199999999999999,"5min":0.99199999999999999,"15min":0.99199999999999999},"last":0.72699999999999998}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69399999999999995}]}]},{"osd":2,"up_from":18,"seq":77309411379,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27900,"kb_used_data":884,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717252,"statfs":{"total":95995035648,"available":95966466048,"internally_reserved":0,"allocated":905216,"data_stored":418786,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jul 14 19:15:08 2023","interfaces":[{"interface":"back","average":{"1min":0.52800000000000002,"5min":0.48799999999999999,"15min":0.47899999999999998},"min":{"1min":0.32600000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.76600000000000001,"5min":0.76600000000000001,"15min":0.76600000000000001},"last":0.36399999999999999},{"interface":"front","average":{"1min":0.58099999999999996,"5min":0.505,"15min":0.49299999999999999},"min":{"1min":0.373,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.79500000000000004,"5min":0.79500000000000004,"15min":0.79500000000000004},"last":0.38600000000000001}]},{"osd":1,"last update":"Fri Jul 14 19:15:08 2023","interfaces":[{"interface":"back","average":{"1min":0.58699999999999997,"5min":0.505,"15min":0.48499999999999999},"min":{"1min":0.38400000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.84599999999999997,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.56699999999999995},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.51500000000000001,"15min":0.497},"min":{"1min":0.29199999999999998,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":0.77700000000000002,"5min":0.90700000000000003,"15min":0.90700000000000003},"last":0.71299999999999997}]},{"osd":3,"last update":"Fri Jul 14 19:15:27 2023","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.53200000000000003,"15min":0.51300000000000001},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.74399999999999999,"5min":0.77700000000000002,"15min":0.77700000000000002},"last":0.44},{"interface":"front","average":{"1min":0.64800000000000002,"5min":0.56599999999999995,"15min":0.54900000000000004},"min":{"1min":0.432,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.80800000000000005,"5min":0.80800000000000005,"15min":0.80800000000000005},"last":0.501}]},{"osd":4,"last update":"Fri Jul 14 19:15:53 2023","interfaces":[{"interface":"back","average":{"1min":0.57899999999999996,"5min":0.53400000000000003,"15min":0.51500000000000001},"min":{"1min":0.45200000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.78700000000000003,"5min":0.78700000000000003,"15min":0.78700000000000003},"last":0.55100000000000005},{"interface":"front","average":{"1min":0.60599999999999998,"5min":0.54700000000000004,"15min":0.52700000000000002},"min":{"1min":0.436,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.81299999999999994,"5min":0.81299999999999994,"15min":0.81299999999999994},"last":0.69499999999999995}]},{"osd":5,"last update":"Fri Jul 14 19:15:13 2023","interfaces":[{"interface":"back","average":{"1min":0.432,"5min":0.40600000000000003,"15min":0.40200000000000002},"min":{"1min":0.32600000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.59499999999999997,"5min":0.59499999999999997,"15min":0.59499999999999997},"last":0.46600000000000003},{"interface":"front","average":{"1min":0.48799999999999999,"5min":0.432,"15min":0.42299999999999999},"min":{"1min":0.313,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.82599999999999996,"5min":0.82599999999999996,"15min":0.82599999999999996},"last":0.32100000000000001}]},{"osd":6,"last update":"Fri Jul 14 19:15:33 2023","interfaces":[{"interface":"back","average":{"1min":0.68799999999999994,"5min":0.627,"15min":0.61699999999999999},"min":{"1min":0.501,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.97799999999999998,"5min":0.97799999999999998,"15min":0.97799999999999998},"last":0.41699999999999998},{"interface":"front","average":{"1min":0.68799999999999994,"5min":0.62,"15min":0.60799999999999998},"min":{"1min":0.45800000000000002,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.97299999999999998,"5min":0.97299999999999998,"15min":0.97299999999999998},"last":0.67000000000000004}]},{"osd":7,"last update":"Fri Jul 14 19:15:53 2023","interfaces":[{"interface":"back","average":{"1min":0.64700000000000002,"5min":0.65100000000000002,"15min":0.65200000000000002},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.96299999999999997,"5min":0.96299999999999997,"15min":0.96299999999999997},"last":0.56999999999999995},{"interface":"front","average":{"1min":0.67100000000000004,"5min":0.66700000000000004,"15min":0.66700000000000004},"min":{"1min":0.48799999999999999,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.94099999999999995,"5min":0.94099999999999995,"15min":0.94099999999999995},"last":0.628}]},{"osd":8,"last update":"Fri Jul 14 19:15:13 2023","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.58299999999999996,"15min":0.58299999999999996},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.90600000000000003,"5min":0.90600000000000003,"15min":0.90600000000000003},"last":0.57899999999999996},{"interface":"front","average":{"1min":0.59399999999999997,"5min":0.59399999999999997,"15min":0.59399999999999997},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.90800000000000003,"5min":0.90800000000000003,"15min":0.90800000000000003},"last":0.60999999999999999}]},{"osd":9,"last update":"Fri Jul 14 19:15:39 2023","interfaces":[{"interface":"back","average":{"1min":0.73299999999999998,"5min":0.73299999999999998,"15min":0.73299999999999998},"min":{"1min":0.56299999999999994,"5min":0.56299999999999994,"15min":0.56299999999999994},"max":{"1min":1.046,"5min":1.046,"15min":1.046},"last":0.52900000000000003},{"interface":"front","average":{"1min":0.72799999999999998,"5min":0.72799999999999998,"15min":0.72799999999999998},"min":{"1min":0.49199999999999999,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":1.0289999999999999,"5min":1.0289999999999999,"15min":1.0289999999999999},"last":0.51600000000000001}]},{"osd":10,"last update":"Fri Jul 14 19:15:56 2023","interfaces":[{"interface":"back","average":{"1min":0.63900000000000001,"5min":0.63900000000000001,"15min":0.63900000000000001},"min":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":0.88500000000000001,"5min":0.88500000000000001,"15min":0.88500000000000001},"last":0.625},{"interface":"front","average":{"1min":0.71799999999999997,"5min":0.71799999999999997,"15min":0.71799999999999997},"min":{"1min":0.56200000000000006,"5min":0.56200000000000006,"15min":0.56200000000000006},"max":{"1min":1.073,"5min":1.073,"15min":1.073},"last":0.65100000000000002}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59499999999999997}]}]},{"osd":3,"up_from":25,"seq":107374182446,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27872,"kb_used_data":852,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717280,"statfs":{"total":95995035648,"available":95966494720,"internally_reserved":0,"allocated":872448,"data_stored":415693,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jul 14 19:15:29 2023","interfaces":[{"interface":"back","average":{"1min":0.40600000000000003,"5min":0.36199999999999999,"15min":0.35099999999999998},"min":{"1min":0.27400000000000002,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":0.59599999999999997,"5min":0.59599999999999997,"15min":0.59599999999999997},"last":0.35299999999999998},{"interface":"front","average":{"1min":0.44500000000000001,"5min":0.39200000000000002,"15min":0.378},"min":{"1min":0.28699999999999998,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.85899999999999999,"5min":0.85899999999999999,"15min":0.85899999999999999},"last":0.375}]},{"osd":1,"last update":"Fri Jul 14 19:15:29 2023","interfaces":[{"interface":"back","average":{"1min":0.64200000000000002,"5min":0.55700000000000005,"15min":0.53000000000000003},"min":{"1min":0.42399999999999999,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.94699999999999995,"5min":0.94699999999999995,"15min":0.94699999999999995},"last":0.59299999999999997},{"interface":"front","average":{"1min":0.63500000000000001,"5min":0.56999999999999995,"15min":0.55200000000000005},"min":{"1min":0.41799999999999998,"5min":0.372,"15min":0.372},"max":{"1min":0.79100000000000004,"5min":0.79100000000000004,"15min":0.79100000000000004},"last":0.54400000000000004}]},{"osd":2,"last update":"Fri Jul 14 19:15:29 2023","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.56999999999999995,"15min":0.55800000000000005},"min":{"1min":0.373,"5min":0.372,"15min":0.372},"max":{"1min":0.77500000000000002,"5min":0.871,"15min":0.871},"last":0.48599999999999999},{"interface":"front","average":{"1min":0.63300000000000001,"5min":0.57399999999999995,"15min":0.55600000000000005},"min":{"1min":0.434,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.93700000000000006,"5min":0.93700000000000006,"15min":0.93700000000000006},"last":0.502}]},{"osd":4,"last update":"Fri Jul 14 19:15:56 2023","interfaces":[{"interface":"back","average":{"1min":0.64500000000000002,"5min":0.58399999999999996,"15min":0.56200000000000006},"min":{"1min":0.47499999999999998,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.98599999999999999,"5min":0.98599999999999999,"15min":0.98599999999999999},"last":0.46100000000000002},{"interface":"front","average":{"1min":0.66800000000000004,"5min":0.60199999999999998,"15min":0.58999999999999997},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":0.86799999999999999,"5min":0.86799999999999999,"15min":0.86799999999999999},"last":0.55400000000000005}]},{"osd":5,"last update":"Fri Jul 14 19:15:15 2023","interfaces":[{"interface":"back","average":{"1min":0.64700000000000002,"5min":0.58899999999999997,"15min":0.57899999999999996},"min":{"1min":0.38600000000000001,"5min":0.374,"15min":0.374},"max":{"1min":0.84499999999999997,"5min":0.84499999999999997,"15min":0.84499999999999997},"last":0.51900000000000002},{"interface":"front","average":{"1min":0.63500000000000001,"5min":0.61299999999999999,"15min":0.60899999999999999},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.91700000000000004,"5min":0.91700000000000004,"15min":0.91700000000000004},"last":0.52900000000000003}]},{"osd":6,"last update":"Fri Jul 14 19:15:36 2023","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.51200000000000001,"15min":0.50600000000000001},"min":{"1min":0.35499999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.86399999999999999,"5min":0.86399999999999999,"15min":0.86399999999999999},"last":0.67600000000000005},{"interface":"front","average":{"1min":0.66300000000000003,"5min":0.56699999999999995,"15min":0.55100000000000005},"min":{"1min":0.378,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.99199999999999999,"5min":0.99199999999999999,"15min":0.99199999999999999},"last":0.439}]},{"osd":7,"last update":"Fri Jul 14 19:15:56 2023","interfaces":[{"interface":"back","average":{"1min":0.73599999999999999,"5min":0.71699999999999997,"15min":0.71399999999999997},"min":{"1min":0.47599999999999998,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":1.05,"5min":1.1080000000000001,"15min":1.1080000000000001},"last":0.57899999999999996},{"interface":"front","average":{"1min":0.70299999999999996,"5min":0.67600000000000005,"15min":0.67200000000000004},"min":{"1min":0.46500000000000002,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":0.89400000000000002,"5min":0.95599999999999996,"15min":0.95599999999999996},"last":0.61699999999999999}]},{"osd":8,"last update":"Fri Jul 14 19:15:15 2023","interfaces":[{"interface":"back","average":{"1min":0.747,"5min":0.747,"15min":0.747},"min":{"1min":0.55800000000000005,"5min":0.55800000000000005,"15min":0.55800000000000005},"max":{"1min":1.0640000000000001,"5min":1.0640000000000001,"15min":1.0640000000000001},"last":0.66700000000000004},{"interface":"front","average":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"min":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":0.98499999999999999,"5min":0.98499999999999999,"15min":0.98499999999999999},"last":0.68500000000000005}]},{"osd":9,"last update":"Fri Jul 14 19:15:36 2023","interfaces":[{"interface":"back","average":{"1min":0.69799999999999995,"5min":0.69799999999999995,"15min":0.69799999999999995},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":1.048,"5min":1.048,"15min":1.048},"last":0.41799999999999998},{"interface":"front","average":{"1min":0.77300000000000002,"5min":0.77300000000000002,"15min":0.77300000000000002},"min":{"1min":0.49099999999999999,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":1.0289999999999999,"5min":1.0289999999999999,"15min":1.0289999999999999},"last":0.60499999999999998}]},{"osd":10,"last update":"Fri Jul 14 19:15:56 2023","interfaces":[{"interface":"back","average":{"1min":0.81299999999999994,"5min":0.81299999999999994,"15min":0.81299999999999994},"min":{"1min":0.58199999999999996,"5min":0.58199999999999996,"15min":0.58199999999999996},"max":{"1min":1.0469999999999999,"5min":1.0469999999999999,"15min":1.0469999999999999},"last":0.64400000000000002},{"interface":"front","average":{"1min":0.77200000000000002,"5min":0.77200000000000002,"15min":0.77200000000000002},"min":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":1.0069999999999999,"5min":1.0069999999999999,"15min":1.0069999999999999},"last":0.56799999999999995}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65500000000000003}]}]},{"osd":4,"up_from":32,"seq":137438953514,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27872,"kb_used_data":852,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717280,"statfs":{"total":95995035648,"available":95966494720,"internally_reserved":0,"allocated":872448,"data_stored":415942,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jul 14 19:14:53 2023","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.503,"15min":0.495},"min":{"1min":0.27200000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.81999999999999995,"5min":0.81999999999999995,"15min":0.81999999999999995},"last":0.56299999999999994},{"interface":"front","average":{"1min":0.58199999999999996,"5min":0.53000000000000003,"15min":0.52200000000000002},"min":{"1min":0.378,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.84599999999999997,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.58299999999999996}]},{"osd":1,"last update":"Fri Jul 14 19:15:56 2023","interfaces":[{"interface":"back","average":{"1min":0.46600000000000003,"5min":0.39000000000000001,"15min":0.36399999999999999},"min":{"1min":0.30499999999999999,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.78400000000000003,"5min":1.0049999999999999,"15min":1.0049999999999999},"last":0.30399999999999999},{"interface":"front","average":{"1min":0.46600000000000003,"5min":0.39700000000000002,"15min":0.36899999999999999},"min":{"1min":0.26900000000000002,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":0.71699999999999997,"5min":0.91500000000000004,"15min":0.91500000000000004},"last":0.32400000000000001}]},{"osd":2,"last update":"Fri Jul 14 19:15:56 2023","interfaces":[{"interface":"back","average":{"1min":0.55100000000000005,"5min":0.54600000000000004,"15min":0.53400000000000003},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.69099999999999995,"5min":0.98499999999999999,"15min":0.98499999999999999},"last":0.495},{"interface":"front","average":{"1min":0.66000000000000003,"5min":0.59899999999999998,"15min":0.56899999999999995},"min":{"1min":0.5,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.88700000000000001,"5min":0.92900000000000005,"15min":0.92900000000000005},"last":0.441}]},{"osd":3,"last update":"Fri Jul 14 19:15:56 2023","interfaces":[{"interface":"back","average":{"1min":0.63700000000000001,"5min":0.55000000000000004,"15min":0.52500000000000002},"min":{"1min":0.41299999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.89300000000000002,"5min":0.89300000000000002,"15min":0.89300000000000002},"last":0.59199999999999997},{"interface":"front","average":{"1min":0.56100000000000005,"5min":0.55000000000000004,"15min":0.53800000000000003},"min":{"1min":0.36899999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.73499999999999999,"5min":0.94199999999999995,"15min":0.94199999999999995},"last":0.58299999999999996}]},{"osd":5,"last update":"Fri Jul 14 19:15:14 2023","interfaces":[{"interface":"back","average":{"1min":0.64700000000000002,"5min":0.57899999999999996,"15min":0.56699999999999995},"min":{"1min":0.433,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":0.87,"5min":0.87,"15min":0.87},"last":0.51400000000000001},{"interface":"front","average":{"1min":0.627,"5min":0.57899999999999996,"15min":0.57099999999999995},"min":{"1min":0.42299999999999999,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.86699999999999999,"5min":0.86699999999999999,"15min":0.86699999999999999},"last":0.52700000000000002}]},{"osd":6,"last update":"Fri Jul 14 19:15:30 2023","interfaces":[{"interface":"back","average":{"1min":0.64500000000000002,"5min":0.625,"15min":0.622},"min":{"1min":0.44400000000000001,"5min":0.38,"15min":0.38},"max":{"1min":0.94299999999999995,"5min":0.94299999999999995,"15min":0.94299999999999995},"last":0.57199999999999995},{"interface":"front","average":{"1min":0.66500000000000004,"5min":0.60799999999999998,"15min":0.59799999999999998},"min":{"1min":0.42699999999999999,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":1.0049999999999999,"5min":1.0049999999999999,"15min":1.0049999999999999},"last":0.47999999999999998}]},{"osd":7,"last update":"Fri Jul 14 19:15:56 2023","interfaces":[{"interface":"back","average":{"1min":0.68100000000000005,"5min":0.60499999999999998,"15min":0.59299999999999997},"min":{"1min":0.44500000000000001,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.97599999999999998,"5min":1.0529999999999999,"15min":1.0529999999999999},"last":0.53900000000000003},{"interface":"front","average":{"1min":0.70899999999999996,"5min":0.59599999999999997,"15min":0.57699999999999996},"min":{"1min":0.47599999999999998,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.93100000000000005,"5min":0.93100000000000005,"15min":0.93100000000000005},"last":0.46100000000000002}]},{"osd":8,"last update":"Fri Jul 14 19:15:14 2023","interfaces":[{"interface":"back","average":{"1min":0.63500000000000001,"5min":0.63500000000000001,"15min":0.63500000000000001},"min":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":1.0169999999999999,"5min":1.0169999999999999,"15min":1.0169999999999999},"last":0.55200000000000005},{"interface":"front","average":{"1min":0.76500000000000001,"5min":0.76500000000000001,"15min":0.76500000000000001},"min":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":0.98499999999999999,"5min":0.98499999999999999,"15min":0.98499999999999999},"last":0.56299999999999994}]},{"osd":9,"last update":"Fri Jul 14 19:15:35 2023","interfaces":[{"interface":"back","average":{"1min":0.77400000000000002,"5min":0.77400000000000002,"15min":0.77400000000000002},"min":{"1min":0.57399999999999995,"5min":0.57399999999999995,"15min":0.57399999999999995},"max":{"1min":0.99299999999999999,"5min":0.99299999999999999,"15min":0.99299999999999999},"last":0.64200000000000002},{"interface":"front","average":{"1min":0.76800000000000002,"5min":0.76800000000000002,"15min":0.76800000000000002},"min":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":1.05,"5min":1.05,"15min":1.05},"last":0.63600000000000001}]},{"osd":10,"last update":"Fri Jul 14 19:15:56 2023","interfaces":[{"interface":"back","average":{"1min":0.82199999999999995,"5min":0.82199999999999995,"15min":0.82199999999999995},"min":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.56000000000000005},"max":{"1min":1.0669999999999999,"5min":1.0669999999999999,"15min":1.0669999999999999},"last":0.62},{"interface":"front","average":{"1min":0.80900000000000005,"5min":0.80900000000000005,"15min":0.80900000000000005},"min":{"1min":0.54900000000000004,"5min":0.54900000000000004,"15min":0.54900000000000004},"max":{"1min":1.028,"5min":1.028,"15min":1.028},"last":0.61199999999999999}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60099999999999998}]}]},{"osd":5,"up_from":37,"seq":158913789990,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27940,"kb_used_data":924,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717212,"statfs":{"total":95995035648,"available":95966425088,"internally_reserved":0,"allocated":946176,"data_stored":1007716,"data_compressed":35714,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jul 14 19:15:12 2023","interfaces":[{"interface":"back","average":{"1min":0.628,"5min":0.53200000000000003,"15min":0.51600000000000001},"min":{"1min":0.41299999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":1.028,"5min":1.028,"15min":1.028},"last":0.28399999999999997},{"interface":"front","average":{"1min":0.63,"5min":0.56200000000000006,"15min":0.55000000000000004},"min":{"1min":0.40999999999999998,"5min":0.307,"15min":0.307},"max":{"1min":0.93300000000000005,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.33700000000000002}]},{"osd":1,"last update":"Fri Jul 14 19:15:12 2023","interfaces":[{"interface":"back","average":{"1min":0.59799999999999998,"5min":0.51100000000000001,"15min":0.497},"min":{"1min":0.36699999999999999,"5min":0.307,"15min":0.307},"max":{"1min":1.2410000000000001,"5min":1.2410000000000001,"15min":1.2410000000000001},"last":0.57599999999999996},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.48299999999999998,"15min":0.47199999999999998},"min":{"1min":0.39500000000000002,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.85799999999999998,"5min":0.85799999999999998,"15min":0.85799999999999998},"last":0.63}]},{"osd":2,"last update":"Fri Jul 14 19:15:12 2023","interfaces":[{"interface":"back","average":{"1min":0.42799999999999999,"5min":0.41199999999999998,"15min":0.40999999999999998},"min":{"1min":0.27200000000000002,"5min":0.246,"15min":0.246},"max":{"1min":0.66500000000000004,"5min":0.68400000000000005,"15min":0.68400000000000005},"last":0.38200000000000001},{"interface":"front","average":{"1min":0.45300000000000001,"5min":0.40899999999999997,"15min":0.40200000000000002},"min":{"1min":0.309,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.73999999999999999,"5min":0.73999999999999999,"15min":0.73999999999999999},"last":0.51400000000000001}]},{"osd":3,"last update":"Fri Jul 14 19:15:12 2023","interfaces":[{"interface":"back","average":{"1min":0.65000000000000002,"5min":0.58999999999999997,"15min":0.57999999999999996},"min":{"1min":0.41399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.98199999999999998,"5min":0.98199999999999998,"15min":0.98199999999999998},"last":0.441},{"interface":"front","average":{"1min":0.65200000000000002,"5min":0.60099999999999998,"15min":0.59299999999999997},"min":{"1min":0.44900000000000001,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":0.95899999999999996,"5min":0.95899999999999996,"15min":0.95899999999999996},"last":0.48299999999999998}]},{"osd":4,"last update":"Fri Jul 14 19:15:12 2023","interfaces":[{"interface":"back","average":{"1min":0.59799999999999998,"5min":0.54400000000000004,"15min":0.53500000000000003},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.79400000000000004,"5min":0.79400000000000004,"15min":0.79400000000000004},"last":0.59599999999999997},{"interface":"front","average":{"1min":0.626,"5min":0.57799999999999996,"15min":0.56999999999999995},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.875,"5min":0.875,"15min":0.875},"last":0.61099999999999999}]},{"osd":6,"last update":"Fri Jul 14 19:15:35 2023","interfaces":[{"interface":"back","average":{"1min":0.65700000000000003,"5min":0.61899999999999999,"15min":0.61299999999999999},"min":{"1min":0.437,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.91000000000000003,"5min":0.91000000000000003,"15min":0.91000000000000003},"last":0.55500000000000005},{"interface":"front","average":{"1min":0.70699999999999996,"5min":0.64000000000000001,"15min":0.629},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":1.0800000000000001,"5min":1.0800000000000001,"15min":1.0800000000000001},"last":0.41999999999999998}]},{"osd":7,"last update":"Fri Jul 14 19:15:53 2023","interfaces":[{"interface":"back","average":{"1min":0.64500000000000002,"5min":0.66000000000000003,"15min":0.66200000000000003},"min":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":0.84599999999999997,"5min":0.91400000000000003,"15min":0.91400000000000003},"last":0.72999999999999998},{"interface":"front","average":{"1min":0.66300000000000003,"5min":0.68000000000000005,"15min":0.68300000000000005},"min":{"1min":0.45700000000000002,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":0.83499999999999996,"5min":1.1240000000000001,"15min":1.1240000000000001},"last":0.71699999999999997}]},{"osd":8,"last update":"Fri Jul 14 19:15:14 2023","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.57999999999999996,"15min":0.57999999999999996},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.85199999999999998,"5min":0.85199999999999998,"15min":0.85199999999999998},"last":0.46000000000000002},{"interface":"front","average":{"1min":0.66300000000000003,"5min":0.66300000000000003,"15min":0.66300000000000003},"min":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":0.94199999999999995,"5min":0.94199999999999995,"15min":0.94199999999999995},"last":0.70199999999999996}]},{"osd":9,"last update":"Fri Jul 14 19:15:35 2023","interfaces":[{"interface":"back","average":{"1min":0.73599999999999999,"5min":0.73599999999999999,"15min":0.73599999999999999},"min":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"max":{"1min":1.014,"5min":1.014,"15min":1.014},"last":0.66500000000000004},{"interface":"front","average":{"1min":0.76000000000000001,"5min":0.76000000000000001,"15min":0.76000000000000001},"min":{"1min":0.51500000000000001,"5min":0.51500000000000001,"15min":0.51500000000000001},"max":{"1min":1.196,"5min":1.196,"15min":1.196},"last":0.64800000000000002}]},{"osd":10,"last update":"Fri Jul 14 19:15:59 2023","interfaces":[{"interface":"back","average":{"1min":0.70599999999999996,"5min":0.70599999999999996,"15min":0.70599999999999996},"min":{"1min":0.56200000000000006,"5min":0.56200000000000006,"15min":0.56200000000000006},"max":{"1min":0.89300000000000002,"5min":0.89300000000000002,"15min":0.89300000000000002},"last":0.753},{"interface":"front","average":{"1min":0.70999999999999996,"5min":0.70999999999999996,"15min":0.70999999999999996},"min":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":0.89000000000000001,"5min":0.89000000000000001,"15min":0.89000000000000001},"last":0.68300000000000005}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34079,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34079,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34079,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35714,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35714,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35714,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35714,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35714,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1564,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":2866,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1428,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1156,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":908,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1758,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-07-14T19:16:06.786 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2023-07-14T19:16:06.787 INFO:tasks.cephadm:Setting up client nodes... 2023-07-14T19:16:06.787 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid adc52f04-2279-11ee-9b33-001a4aab830c -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2023-07-14T19:16:06.972 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:16:06 smithi191 ceph-mon[98311]: pgmap v258: 97 pgs: 97 active+clean; 581 KiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s wr, 14 op/s 2023-07-14T19:16:06.972 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:16:06 smithi191 ceph-mon[98311]: from='client.14706 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T19:16:06.972 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:16:06 smithi191 ceph-mon[98311]: from='client.14712 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T19:16:06.972 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:16:06 smithi191 ceph-mon[98311]: mds.? [v2:172.21.15.130:6834/3088490808,v1:172.21.15.130:6835/3088490808] up:active 2023-07-14T19:16:06.973 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:16:06 smithi191 ceph-mon[98311]: fsmap cephfs:5 7 up:standby 5 up:active 2023-07-14T19:16:06.973 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:16:06 smithi191 ceph-mon[98311]: mgrmap e17: x(active, since 6m), standbys: z, y 2023-07-14T19:16:06.973 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:16:06 smithi191 ceph-mon[98311]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2023-07-14T19:16:06.973 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:16:06 smithi191 ceph-mon[98311]: from='client.? 172.21.15.40:0/208544988' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T19:16:06.973 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:16:06 smithi191 ceph-mon[98311]: from='client.? 172.21.15.40:0/128476494' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T19:16:06.992 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:16:06 smithi130 ceph-mon[97832]: pgmap v258: 97 pgs: 97 active+clean; 581 KiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s wr, 14 op/s 2023-07-14T19:16:06.992 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:16:06 smithi130 ceph-mon[97832]: from='client.14706 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T19:16:06.992 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:16:06 smithi130 ceph-mon[97832]: from='client.14712 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T19:16:06.992 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:16:06 smithi130 ceph-mon[97832]: mds.? [v2:172.21.15.130:6834/3088490808,v1:172.21.15.130:6835/3088490808] up:active 2023-07-14T19:16:06.993 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:16:06 smithi130 ceph-mon[97832]: fsmap cephfs:5 7 up:standby 5 up:active 2023-07-14T19:16:06.993 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:16:06 smithi130 ceph-mon[97832]: mgrmap e17: x(active, since 6m), standbys: z, y 2023-07-14T19:16:06.993 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:16:06 smithi130 ceph-mon[97832]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2023-07-14T19:16:06.993 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:16:06 smithi130 ceph-mon[97832]: from='client.? 172.21.15.40:0/208544988' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T19:16:06.993 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:16:06 smithi130 ceph-mon[97832]: from='client.? 172.21.15.40:0/128476494' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T19:16:07.371 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/adc52f04-2279-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T19:16:07.904 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:16:07 smithi040 ceph-mon[96518]: from='client.14730 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-07-14T19:16:07.904 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:16:07 smithi040 ceph-mon[96518]: from='client.14736 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-07-14T19:16:07.972 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:16:07 smithi191 ceph-mon[98311]: from='client.14730 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-07-14T19:16:07.972 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:16:07 smithi191 ceph-mon[98311]: from='client.14736 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-07-14T19:16:07.992 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:16:07 smithi130 ceph-mon[97832]: from='client.14730 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-07-14T19:16:07.992 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:16:07 smithi130 ceph-mon[97832]: from='client.14736 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-07-14T19:16:08.611 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:08.609+0000 7f047595c700 1 -- 172.21.15.40:0/2727706760 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0470105ab0 msgr2=0x7f0470107eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:16:08.611 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:08.609+0000 7f047595c700 1 --2- 172.21.15.40:0/2727706760 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0470105ab0 0x7f0470107eb0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f046800b0a0 tx=0x7f046800b3b0 comp rx=0 tx=0).stop 2023-07-14T19:16:08.612 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:08.609+0000 7f047595c700 1 -- 172.21.15.40:0/2727706760 shutdown_connections 2023-07-14T19:16:08.612 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:08.609+0000 7f047595c700 1 --2- 172.21.15.40:0/2727706760 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0470105ab0 0x7f0470107eb0 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:08.612 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:08.609+0000 7f047595c700 1 --2- 172.21.15.40:0/2727706760 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0470100870 0x7f0470102c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:08.612 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:08.609+0000 7f047595c700 1 --2- 172.21.15.40:0/2727706760 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0470103190 0x7f0470105570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:08.612 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:08.609+0000 7f047595c700 1 -- 172.21.15.40:0/2727706760 >> 172.21.15.40:0/2727706760 conn(0x7f04700fa2d0 msgr2=0x7f04700fc700 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:16:08.613 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:08.609+0000 7f047595c700 1 -- 172.21.15.40:0/2727706760 shutdown_connections 2023-07-14T19:16:08.613 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:08.609+0000 7f047595c700 1 -- 172.21.15.40:0/2727706760 wait complete. 2023-07-14T19:16:08.613 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:08.610+0000 7f047595c700 1 Processor -- start 2023-07-14T19:16:08.614 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:08.610+0000 7f047595c700 1 -- start start 2023-07-14T19:16:08.614 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:08.610+0000 7f047595c700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0470100870 0x7f0470195b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:16:08.614 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:08.610+0000 7f047595c700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0470103190 0x7f0470196060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:16:08.614 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:08.610+0000 7f047595c700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0470105ab0 0x7f047019d0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:16:08.615 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:08.610+0000 7f047595c700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f047019d6b0 con 0x7f0470100870 2023-07-14T19:16:08.615 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:08.610+0000 7f047595c700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f047019d7f0 con 0x7f0470105ab0 2023-07-14T19:16:08.615 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:08.610+0000 7f047595c700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f047019d980 con 0x7f0470103190 2023-07-14T19:16:08.615 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:08.610+0000 7f046effd700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0470100870 0x7f0470195b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:16:08.615 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:08.610+0000 7f046effd700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0470100870 0x7f0470195b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:33088/0 (socket says 172.21.15.40:33088) 2023-07-14T19:16:08.615 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:08.610+0000 7f046effd700 1 -- 172.21.15.40:0/4196468494 learned_addr learned my addr 172.21.15.40:0/4196468494 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-07-14T19:16:08.615 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:08.611+0000 7f046effd700 1 -- 172.21.15.40:0/4196468494 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0470103190 msgr2=0x7f0470196060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:16:08.615 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:08.611+0000 7f046e7fc700 1 --2- 172.21.15.40:0/4196468494 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0470103190 0x7f0470196060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:16:08.616 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:08.611+0000 7f046f7fe700 1 --2- 172.21.15.40:0/4196468494 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0470105ab0 0x7f047019d0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:16:08.616 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:08.611+0000 7f046effd700 1 --2- 172.21.15.40:0/4196468494 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0470103190 0x7f0470196060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:08.616 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:08.611+0000 7f046effd700 1 -- 172.21.15.40:0/4196468494 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0470105ab0 msgr2=0x7f047019d0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:16:08.616 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:08.611+0000 7f046effd700 1 --2- 172.21.15.40:0/4196468494 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0470105ab0 0x7f047019d0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:08.616 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:08.611+0000 7f046effd700 1 -- 172.21.15.40:0/4196468494 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0468009d00 con 0x7f0470100870 2023-07-14T19:16:08.616 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:08.611+0000 7f046effd700 1 --2- 172.21.15.40:0/4196468494 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0470100870 0x7f0470195b20 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f046400d9b0 tx=0x7f046400dd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:16:08.617 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:08.611+0000 7f047495a700 1 -- 172.21.15.40:0/4196468494 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f046400bb60 con 0x7f0470100870 2023-07-14T19:16:08.617 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:08.611+0000 7f047495a700 1 -- 172.21.15.40:0/4196468494 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f0464007e80 con 0x7f0470100870 2023-07-14T19:16:08.617 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:08.611+0000 7f047595c700 1 -- 172.21.15.40:0/4196468494 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f047019dc60 con 0x7f0470100870 2023-07-14T19:16:08.617 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:08.611+0000 7f047495a700 1 -- 172.21.15.40:0/4196468494 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0464004dd0 con 0x7f0470100870 2023-07-14T19:16:08.617 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:08.611+0000 7f047595c700 1 -- 172.21.15.40:0/4196468494 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f047019e160 con 0x7f0470100870 2023-07-14T19:16:08.617 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:08.612+0000 7f047595c700 1 -- 172.21.15.40:0/4196468494 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f04700fbe40 con 0x7f0470100870 2023-07-14T19:16:08.618 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:08.613+0000 7f047495a700 1 -- 172.21.15.40:0/4196468494 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7f0464004020 con 0x7f0470100870 2023-07-14T19:16:08.618 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:08.614+0000 7f047495a700 1 --2- 172.21.15.40:0/4196468494 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f04580a1330 0x7f04580a37e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:16:08.618 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:08.614+0000 7f046e7fc700 1 --2- 172.21.15.40:0/4196468494 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f04580a1330 0x7f04580a37e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:16:08.618 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:08.614+0000 7f047495a700 1 -- 172.21.15.40:0/4196468494 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f04640cc8a0 con 0x7f0470100870 2023-07-14T19:16:08.618 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:08.614+0000 7f046e7fc700 1 --2- 172.21.15.40:0/4196468494 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f04580a1330 0x7f04580a37e0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f0460009b10 tx=0x7f0460009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:16:08.620 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:08.618+0000 7f047495a700 1 -- 172.21.15.40:0/4196468494 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f0464095f40 con 0x7f0470100870 2023-07-14T19:16:08.721 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:16:08 smithi040 ceph-mon[96518]: pgmap v259: 97 pgs: 97 active+clean; 616 KiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail; 119 B/s rd, 9.6 KiB/s wr, 20 op/s 2023-07-14T19:16:08.900 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:08.899+0000 7f047595c700 1 -- 172.21.15.40:0/4196468494 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} v 0) v1 -- 0x7f047002c980 con 0x7f0470100870 2023-07-14T19:16:08.905 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:08.903+0000 7f047495a700 1 -- 172.21.15.40:0/4196468494 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]=0 v31) v1 ==== 170+0+59 (secure 0 0 0) 0x7f0464016070 con 0x7f0470100870 2023-07-14T19:16:08.905 INFO:teuthology.orchestra.run.smithi040.stdout:[client.0] 2023-07-14T19:16:08.906 INFO:teuthology.orchestra.run.smithi040.stdout: key = AQD4nrFk5emyNRAAC4IpxLBmo1rYkOpnN9hS1Q== 2023-07-14T19:16:08.907 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:08.906+0000 7f047595c700 1 -- 172.21.15.40:0/4196468494 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f04580a1330 msgr2=0x7f04580a37e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:16:08.908 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:08.906+0000 7f047595c700 1 --2- 172.21.15.40:0/4196468494 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f04580a1330 0x7f04580a37e0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f0460009b10 tx=0x7f0460009260 comp rx=0 tx=0).stop 2023-07-14T19:16:08.908 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:08.906+0000 7f047595c700 1 -- 172.21.15.40:0/4196468494 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0470100870 msgr2=0x7f0470195b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:16:08.908 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:08.906+0000 7f047595c700 1 --2- 172.21.15.40:0/4196468494 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0470100870 0x7f0470195b20 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f046400d9b0 tx=0x7f046400dd70 comp rx=0 tx=0).stop 2023-07-14T19:16:08.908 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:08.906+0000 7f047595c700 1 -- 172.21.15.40:0/4196468494 shutdown_connections 2023-07-14T19:16:08.908 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:08.906+0000 7f047595c700 1 --2- 172.21.15.40:0/4196468494 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f04580a1330 0x7f04580a37e0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:08.908 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:08.906+0000 7f047595c700 1 --2- 172.21.15.40:0/4196468494 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0470105ab0 0x7f047019d0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:08.909 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:08.906+0000 7f047595c700 1 --2- 172.21.15.40:0/4196468494 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0470100870 0x7f0470195b20 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:08.909 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:08.906+0000 7f047595c700 1 --2- 172.21.15.40:0/4196468494 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0470103190 0x7f0470196060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:08.909 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:08.906+0000 7f047595c700 1 -- 172.21.15.40:0/4196468494 >> 172.21.15.40:0/4196468494 conn(0x7f04700fa2d0 msgr2=0x7f04701014d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:16:08.909 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:08.906+0000 7f047595c700 1 -- 172.21.15.40:0/4196468494 shutdown_connections 2023-07-14T19:16:08.909 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:08.906+0000 7f047595c700 1 -- 172.21.15.40:0/4196468494 wait complete. 2023-07-14T19:16:08.972 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:16:08 smithi191 ceph-mon[98311]: pgmap v259: 97 pgs: 97 active+clean; 616 KiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail; 119 B/s rd, 9.6 KiB/s wr, 20 op/s 2023-07-14T19:16:08.992 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:16:08 smithi130 ceph-mon[97832]: pgmap v259: 97 pgs: 97 active+clean; 616 KiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail; 119 B/s rd, 9.6 KiB/s wr, 20 op/s 2023-07-14T19:16:08.993 INFO:journalctl@ceph.mds.b.smithi130.stdout:Jul 14 19:16:08 smithi130 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-mds-b[122843]: 2023-07-14T19:16:08.854+0000 7fc0c3b89700 -1 mds.pinger is_rank_lagging: rank=3 was never sent ping request. 2023-07-14T19:16:08.993 INFO:journalctl@ceph.mds.b.smithi130.stdout:Jul 14 19:16:08 smithi130 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-mds-b[122843]: 2023-07-14T19:16:08.854+0000 7fc0c3b89700 -1 mds.pinger is_rank_lagging: rank=4 was never sent ping request. 2023-07-14T19:16:09.417 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2023-07-14T19:16:09.417 DEBUG:teuthology.orchestra.run.smithi040:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2023-07-14T19:16:09.418 DEBUG:teuthology.orchestra.run.smithi040:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2023-07-14T19:16:09.460 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2023-07-14T19:16:09.461 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2023-07-14T19:16:09.461 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell --fsid adc52f04-2279-11ee-9b33-001a4aab830c -- ceph mgr dump --format=json 2023-07-14T19:16:09.971 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:16:09 smithi040 ceph-mon[96518]: from='client.? 172.21.15.40:0/4196468494' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2023-07-14T19:16:09.971 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:16:09 smithi040 ceph-mon[96518]: from='client.? 172.21.15.40:0/4196468494' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2023-07-14T19:16:09.972 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:16:09 smithi191 ceph-mon[98311]: from='client.? 172.21.15.40:0/4196468494' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2023-07-14T19:16:09.972 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:16:09 smithi191 ceph-mon[98311]: from='client.? 172.21.15.40:0/4196468494' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2023-07-14T19:16:09.992 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:16:09 smithi130 ceph-mon[97832]: from='client.? 172.21.15.40:0/4196468494' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2023-07-14T19:16:09.992 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:16:09 smithi130 ceph-mon[97832]: from='client.? 172.21.15.40:0/4196468494' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2023-07-14T19:16:10.060 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/adc52f04-2279-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T19:16:10.883 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:16:10 smithi040 ceph-mon[96518]: pgmap v260: 97 pgs: 97 active+clean; 618 KiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 8.8 KiB/s wr, 22 op/s 2023-07-14T19:16:10.972 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:16:10 smithi191 ceph-mon[98311]: pgmap v260: 97 pgs: 97 active+clean; 618 KiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 8.8 KiB/s wr, 22 op/s 2023-07-14T19:16:10.992 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:16:10 smithi130 ceph-mon[97832]: pgmap v260: 97 pgs: 97 active+clean; 618 KiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 8.8 KiB/s wr, 22 op/s 2023-07-14T19:16:11.395 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:11.393+0000 7f4b5ed9c700 1 -- 172.21.15.40:0/137320763 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4b58105000 msgr2=0x7f4b5810cb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:16:11.395 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:11.393+0000 7f4b5ed9c700 1 --2- 172.21.15.40:0/137320763 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4b58105000 0x7f4b5810cb80 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f4b5000b0d0 tx=0x7f4b5000b3e0 comp rx=0 tx=0).stop 2023-07-14T19:16:11.395 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:11.393+0000 7f4b5ed9c700 1 -- 172.21.15.40:0/137320763 shutdown_connections 2023-07-14T19:16:11.396 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:11.393+0000 7f4b5ed9c700 1 --2- 172.21.15.40:0/137320763 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4b58105000 0x7f4b5810cb80 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:11.396 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:11.393+0000 7f4b5ed9c700 1 --2- 172.21.15.40:0/137320763 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4b580fffb0 0x7f4b58100380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:11.396 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:11.393+0000 7f4b5ed9c700 1 --2- 172.21.15.40:0/137320763 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4b58100950 0x7f4b581048d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:11.396 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:11.393+0000 7f4b5ed9c700 1 -- 172.21.15.40:0/137320763 >> 172.21.15.40:0/137320763 conn(0x7f4b58077cd0 msgr2=0x7f4b580780d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:16:11.396 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:11.393+0000 7f4b5ed9c700 1 -- 172.21.15.40:0/137320763 shutdown_connections 2023-07-14T19:16:11.397 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:11.393+0000 7f4b5ed9c700 1 -- 172.21.15.40:0/137320763 wait complete. 2023-07-14T19:16:11.398 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:11.394+0000 7f4b5ed9c700 1 Processor -- start 2023-07-14T19:16:11.398 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:11.394+0000 7f4b5ed9c700 1 -- start start 2023-07-14T19:16:11.399 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:11.394+0000 7f4b5ed9c700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4b580fffb0 0x7f4b58195ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:16:11.399 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:11.394+0000 7f4b5ed9c700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4b58100950 0x7f4b58196430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:16:11.399 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:11.394+0000 7f4b5ed9c700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4b58105000 0x7f4b5819a8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:16:11.399 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:11.394+0000 7f4b5ed9c700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b5819aeb0 con 0x7f4b580fffb0 2023-07-14T19:16:11.399 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:11.394+0000 7f4b5ed9c700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b5819b020 con 0x7f4b58105000 2023-07-14T19:16:11.399 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:11.394+0000 7f4b5ed9c700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b5819b1e0 con 0x7f4b58100950 2023-07-14T19:16:11.399 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:11.394+0000 7f4b5cb38700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4b580fffb0 0x7f4b58195ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:16:11.400 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:11.394+0000 7f4b5cb38700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4b580fffb0 0x7f4b58195ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:33102/0 (socket says 172.21.15.40:33102) 2023-07-14T19:16:11.400 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:11.394+0000 7f4b5cb38700 1 -- 172.21.15.40:0/1176420770 learned_addr learned my addr 172.21.15.40:0/1176420770 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-07-14T19:16:11.400 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:11.394+0000 7f4b5d339700 1 --2- 172.21.15.40:0/1176420770 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4b58105000 0x7f4b5819a8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:16:11.400 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:11.394+0000 7f4b57fff700 1 --2- 172.21.15.40:0/1176420770 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4b58100950 0x7f4b58196430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:16:11.400 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:11.394+0000 7f4b5cb38700 1 -- 172.21.15.40:0/1176420770 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4b58100950 msgr2=0x7f4b58196430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:16:11.400 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:11.394+0000 7f4b5cb38700 1 --2- 172.21.15.40:0/1176420770 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4b58100950 0x7f4b58196430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:11.401 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:11.394+0000 7f4b5cb38700 1 -- 172.21.15.40:0/1176420770 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4b58105000 msgr2=0x7f4b5819a8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:16:11.401 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:11.394+0000 7f4b5cb38700 1 --2- 172.21.15.40:0/1176420770 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4b58105000 0x7f4b5819a8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:11.401 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:11.394+0000 7f4b5cb38700 1 -- 172.21.15.40:0/1176420770 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b50009d00 con 0x7f4b580fffb0 2023-07-14T19:16:11.401 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:11.394+0000 7f4b5d339700 1 --2- 172.21.15.40:0/1176420770 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4b58105000 0x7f4b5819a8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T19:16:11.401 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:11.394+0000 7f4b5cb38700 1 --2- 172.21.15.40:0/1176420770 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4b580fffb0 0x7f4b58195ef0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f4b4c00d950 tx=0x7f4b4c00dd10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:16:11.401 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:11.395+0000 7f4b55ffb700 1 -- 172.21.15.40:0/1176420770 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b4c009b40 con 0x7f4b580fffb0 2023-07-14T19:16:11.401 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:11.395+0000 7f4b5ed9c700 1 -- 172.21.15.40:0/1176420770 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4b5819b4c0 con 0x7f4b580fffb0 2023-07-14T19:16:11.402 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:11.395+0000 7f4b55ffb700 1 -- 172.21.15.40:0/1176420770 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f4b4c009ca0 con 0x7f4b580fffb0 2023-07-14T19:16:11.402 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:11.395+0000 7f4b55ffb700 1 -- 172.21.15.40:0/1176420770 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b4c00f620 con 0x7f4b580fffb0 2023-07-14T19:16:11.402 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:11.395+0000 7f4b5ed9c700 1 -- 172.21.15.40:0/1176420770 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4b581a2d20 con 0x7f4b580fffb0 2023-07-14T19:16:11.402 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:11.396+0000 7f4b5ed9c700 1 -- 172.21.15.40:0/1176420770 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4b5804e340 con 0x7f4b580fffb0 2023-07-14T19:16:11.402 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:11.397+0000 7f4b55ffb700 1 -- 172.21.15.40:0/1176420770 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7f4b4c010460 con 0x7f4b580fffb0 2023-07-14T19:16:11.404 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:11.397+0000 7f4b55ffb700 1 --2- 172.21.15.40:0/1176420770 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f4b480a11e0 0x7f4b480a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:16:11.404 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:11.397+0000 7f4b55ffb700 1 -- 172.21.15.40:0/1176420770 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f4b4c0ccd70 con 0x7f4b580fffb0 2023-07-14T19:16:11.404 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:11.397+0000 7f4b57fff700 1 --2- 172.21.15.40:0/1176420770 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f4b480a11e0 0x7f4b480a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:16:11.404 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:11.398+0000 7f4b57fff700 1 --2- 172.21.15.40:0/1176420770 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f4b480a11e0 0x7f4b480a3690 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f4b44006fd0 tx=0x7f4b44008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:16:11.404 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:11.402+0000 7f4b55ffb700 1 -- 172.21.15.40:0/1176420770 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f4b4c096410 con 0x7f4b580fffb0 2023-07-14T19:16:11.660 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:11.659+0000 7f4b5ed9c700 1 -- 172.21.15.40:0/1176420770 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) v1 -- 0x7f4b581a2fb0 con 0x7f4b580fffb0 2023-07-14T19:16:11.666 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:11.664+0000 7f4b55ffb700 1 -- 172.21.15.40:0/1176420770 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v17) v1 ==== 74+0+260124 (secure 0 0 0) 0x7f4b4c020090 con 0x7f4b580fffb0 2023-07-14T19:16:11.671 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-07-14T19:16:11.673 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:11.667+0000 7f4b5ed9c700 1 -- 172.21.15.40:0/1176420770 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f4b480a11e0 msgr2=0x7f4b480a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:16:11.673 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:11.667+0000 7f4b5ed9c700 1 --2- 172.21.15.40:0/1176420770 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f4b480a11e0 0x7f4b480a3690 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f4b44006fd0 tx=0x7f4b44008040 comp rx=0 tx=0).stop 2023-07-14T19:16:11.673 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:11.667+0000 7f4b5ed9c700 1 -- 172.21.15.40:0/1176420770 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4b580fffb0 msgr2=0x7f4b58195ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:16:11.674 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:11.667+0000 7f4b5ed9c700 1 --2- 172.21.15.40:0/1176420770 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4b580fffb0 0x7f4b58195ef0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f4b4c00d950 tx=0x7f4b4c00dd10 comp rx=0 tx=0).stop 2023-07-14T19:16:11.674 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:11.667+0000 7f4b5ed9c700 1 -- 172.21.15.40:0/1176420770 shutdown_connections 2023-07-14T19:16:11.674 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:11.667+0000 7f4b5ed9c700 1 --2- 172.21.15.40:0/1176420770 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f4b480a11e0 0x7f4b480a3690 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:11.674 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:11.667+0000 7f4b5ed9c700 1 --2- 172.21.15.40:0/1176420770 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4b58105000 0x7f4b5819a8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:11.675 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:11.667+0000 7f4b5ed9c700 1 --2- 172.21.15.40:0/1176420770 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4b580fffb0 0x7f4b58195ef0 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:11.675 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:11.667+0000 7f4b5ed9c700 1 --2- 172.21.15.40:0/1176420770 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f4b58100950 0x7f4b58196430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:11.675 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:11.667+0000 7f4b5ed9c700 1 -- 172.21.15.40:0/1176420770 >> 172.21.15.40:0/1176420770 conn(0x7f4b58077cd0 msgr2=0x7f4b580fd8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:16:11.675 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:11.667+0000 7f4b5ed9c700 1 -- 172.21.15.40:0/1176420770 shutdown_connections 2023-07-14T19:16:11.676 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:11.667+0000 7f4b5ed9c700 1 -- 172.21.15.40:0/1176420770 wait complete. 2023-07-14T19:16:12.108 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":17,"active_gid":14152,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":1693027118},{"type":"v1","addr":"172.21.15.40:6801","nonce":1693027118}]},"active_addr":"172.21.15.40:6801/1693027118","active_change":"2023-07-14T19:09:50.732577+0000","active_mgr_features":4540138322906710015,"available":true,"standbys":[{"gid":24106,"name":"z","mgr_features":4540138322906710015,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"alertmanager_web_password":{"name":"alertmanager_web_password","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"Alertmanager web password","long_desc":"","tags":[],"see_also":[]},"alertmanager_web_user":{"name":"alertmanager_web_user","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"Alertmanager web user","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"prometheus_web_password":{"name":"prometheus_web_password","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"Prometheus web password","long_desc":"","tags":[],"see_also":[]},"prometheus_web_user":{"name":"prometheus_web_user","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"Prometheus web user","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"noautoscale":{"name":"noautoscale","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"global autoscale flag","long_desc":"Option to turn on/off the autoscaler for all pools","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]},{"gid":24116,"name":"y","mgr_features":4540138322906710015,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"alertmanager_web_password":{"name":"alertmanager_web_password","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"Alertmanager web password","long_desc":"","tags":[],"see_also":[]},"alertmanager_web_user":{"name":"alertmanager_web_user","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"Alertmanager web user","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"prometheus_web_password":{"name":"prometheus_web_password","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"Prometheus web password","long_desc":"","tags":[],"see_also":[]},"prometheus_web_user":{"name":"prometheus_web_user","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"Prometheus web user","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"noautoscale":{"name":"noautoscale","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"global autoscale flag","long_desc":"Option to turn on/off the autoscaler for all pools","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"alertmanager_web_password":{"name":"alertmanager_web_password","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"Alertmanager web password","long_desc":"","tags":[],"see_also":[]},"alertmanager_web_user":{"name":"alertmanager_web_user","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"Alertmanager web user","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"prometheus_web_password":{"name":"prometheus_web_password","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"Prometheus web password","long_desc":"","tags":[],"see_also":[]},"prometheus_web_user":{"name":"prometheus_web_user","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"Prometheus web user","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"noautoscale":{"name":"noautoscale","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"global autoscale flag","long_desc":"Option to turn on/off the autoscaler for all pools","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.40:8443/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":3,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.40:0","nonce":773718621}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.40:0","nonce":2145507844}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.40:0","nonce":537002115}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.40:0","nonce":4161618047}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.40:0","nonce":2319412837}]}]} 2023-07-14T19:16:12.116 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2023-07-14T19:16:12.117 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2023-07-14T19:16:12.117 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell --fsid adc52f04-2279-11ee-9b33-001a4aab830c -- ceph osd dump --format=json 2023-07-14T19:16:12.676 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/adc52f04-2279-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T19:16:12.901 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:16:12 smithi040 ceph-mon[96518]: pgmap v261: 97 pgs: 97 active+clean; 623 KiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail; 8.2 KiB/s rd, 8.2 KiB/s wr, 22 op/s 2023-07-14T19:16:12.901 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:16:12 smithi040 ceph-mon[96518]: from='client.? 172.21.15.40:0/1176420770' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2023-07-14T19:16:12.972 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:16:12 smithi191 ceph-mon[98311]: pgmap v261: 97 pgs: 97 active+clean; 623 KiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail; 8.2 KiB/s rd, 8.2 KiB/s wr, 22 op/s 2023-07-14T19:16:12.973 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:16:12 smithi191 ceph-mon[98311]: from='client.? 172.21.15.40:0/1176420770' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2023-07-14T19:16:12.992 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:16:12 smithi130 ceph-mon[97832]: pgmap v261: 97 pgs: 97 active+clean; 623 KiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail; 8.2 KiB/s rd, 8.2 KiB/s wr, 22 op/s 2023-07-14T19:16:12.992 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:16:12 smithi130 ceph-mon[97832]: from='client.? 172.21.15.40:0/1176420770' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2023-07-14T19:16:14.130 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:14.128+0000 7faf714f6700 1 -- 172.21.15.40:0/283750670 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7faf6c0fff40 msgr2=0x7faf6c100310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:16:14.130 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:14.128+0000 7faf714f6700 1 --2- 172.21.15.40:0/283750670 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7faf6c0fff40 0x7faf6c100310 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7faf600099c0 tx=0x7faf60009cd0 comp rx=0 tx=0).stop 2023-07-14T19:16:14.130 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:14.128+0000 7faf714f6700 1 -- 172.21.15.40:0/283750670 shutdown_connections 2023-07-14T19:16:14.130 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:14.128+0000 7faf714f6700 1 --2- 172.21.15.40:0/283750670 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7faf6c104fe0 0x7faf6c10cb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:14.131 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:14.128+0000 7faf714f6700 1 --2- 172.21.15.40:0/283750670 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7faf6c0fff40 0x7faf6c100310 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:14.131 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:14.128+0000 7faf714f6700 1 --2- 172.21.15.40:0/283750670 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7faf6c1008e0 0x7faf6c1048b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:14.131 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:14.128+0000 7faf714f6700 1 -- 172.21.15.40:0/283750670 >> 172.21.15.40:0/283750670 conn(0x7faf6c0fb7c0 msgr2=0x7faf6c0fdbd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:16:14.131 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:14.128+0000 7faf714f6700 1 -- 172.21.15.40:0/283750670 shutdown_connections 2023-07-14T19:16:14.131 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:14.128+0000 7faf714f6700 1 -- 172.21.15.40:0/283750670 wait complete. 2023-07-14T19:16:14.131 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:14.128+0000 7faf714f6700 1 Processor -- start 2023-07-14T19:16:14.133 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:14.129+0000 7faf714f6700 1 -- start start 2023-07-14T19:16:14.133 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:14.129+0000 7faf714f6700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7faf6c0fff40 0x7faf6c072940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:16:14.133 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:14.129+0000 7faf714f6700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7faf6c1008e0 0x7faf6c072e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:16:14.133 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:14.129+0000 7faf714f6700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7faf6c104fe0 0x7faf6c073400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:16:14.133 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:14.129+0000 7faf714f6700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf6c1a41c0 con 0x7faf6c0fff40 2023-07-14T19:16:14.133 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:14.129+0000 7faf714f6700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf6c1a4330 con 0x7faf6c1008e0 2023-07-14T19:16:14.134 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:14.129+0000 7faf714f6700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf6c1a44f0 con 0x7faf6c104fe0 2023-07-14T19:16:14.134 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:14.129+0000 7faf6affd700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7faf6c0fff40 0x7faf6c072940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:16:14.134 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:14.129+0000 7faf6affd700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7faf6c0fff40 0x7faf6c072940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:33114/0 (socket says 172.21.15.40:33114) 2023-07-14T19:16:14.134 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:14.129+0000 7faf6affd700 1 -- 172.21.15.40:0/4020848162 learned_addr learned my addr 172.21.15.40:0/4020848162 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-07-14T19:16:14.134 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:14.129+0000 7faf6a7fc700 1 --2- 172.21.15.40:0/4020848162 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7faf6c1008e0 0x7faf6c072e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:16:14.134 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:14.129+0000 7faf6b7fe700 1 --2- 172.21.15.40:0/4020848162 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7faf6c104fe0 0x7faf6c073400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:16:14.134 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:14.129+0000 7faf6affd700 1 -- 172.21.15.40:0/4020848162 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7faf6c104fe0 msgr2=0x7faf6c073400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:16:14.135 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:14.129+0000 7faf6affd700 1 --2- 172.21.15.40:0/4020848162 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7faf6c104fe0 0x7faf6c073400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:14.135 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:14.129+0000 7faf6affd700 1 -- 172.21.15.40:0/4020848162 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7faf6c1008e0 msgr2=0x7faf6c072e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:16:14.135 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:14.129+0000 7faf6affd700 1 --2- 172.21.15.40:0/4020848162 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7faf6c1008e0 0x7faf6c072e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:14.135 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:14.129+0000 7faf6affd700 1 -- 172.21.15.40:0/4020848162 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf600095f0 con 0x7faf6c0fff40 2023-07-14T19:16:14.135 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:14.129+0000 7faf6affd700 1 --2- 172.21.15.40:0/4020848162 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7faf6c0fff40 0x7faf6c072940 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7faf60009990 tx=0x7faf60009580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:16:14.135 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:14.130+0000 7faf53fff700 1 -- 172.21.15.40:0/4020848162 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf6001d070 con 0x7faf6c0fff40 2023-07-14T19:16:14.135 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:14.130+0000 7faf53fff700 1 -- 172.21.15.40:0/4020848162 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7faf60004cb0 con 0x7faf6c0fff40 2023-07-14T19:16:14.135 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:14.130+0000 7faf714f6700 1 -- 172.21.15.40:0/4020848162 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faf6c1a4770 con 0x7faf6c0fff40 2023-07-14T19:16:14.136 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:14.130+0000 7faf53fff700 1 -- 172.21.15.40:0/4020848162 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf60003b30 con 0x7faf6c0fff40 2023-07-14T19:16:14.136 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:14.130+0000 7faf714f6700 1 -- 172.21.15.40:0/4020848162 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faf6c1a4b80 con 0x7faf6c0fff40 2023-07-14T19:16:14.136 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:14.131+0000 7faf714f6700 1 -- 172.21.15.40:0/4020848162 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faf6c04e340 con 0x7faf6c0fff40 2023-07-14T19:16:14.136 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:14.132+0000 7faf53fff700 1 -- 172.21.15.40:0/4020848162 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7faf600047f0 con 0x7faf6c0fff40 2023-07-14T19:16:14.137 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:14.132+0000 7faf53fff700 1 --2- 172.21.15.40:0/4020848162 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7faf540a11e0 0x7faf540a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:16:14.137 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:14.133+0000 7faf6a7fc700 1 --2- 172.21.15.40:0/4020848162 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7faf540a11e0 0x7faf540a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:16:14.137 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:14.133+0000 7faf53fff700 1 -- 172.21.15.40:0/4020848162 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7979+0+0 (secure 0 0 0) 0x7faf600cd7e0 con 0x7faf6c0fff40 2023-07-14T19:16:14.137 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:14.133+0000 7faf6a7fc700 1 --2- 172.21.15.40:0/4020848162 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7faf540a11e0 0x7faf540a3690 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7faf6c074130 tx=0x7faf5c00b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:16:14.138 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:14.137+0000 7faf53fff700 1 -- 172.21.15.40:0/4020848162 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7faf60096f10 con 0x7faf6c0fff40 2023-07-14T19:16:14.336 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:14.334+0000 7faf714f6700 1 -- 172.21.15.40:0/4020848162 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7faf6c073e70 con 0x7faf6c0fff40 2023-07-14T19:16:14.338 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:14.336+0000 7faf53fff700 1 -- 172.21.15.40:0/4020848162 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v81) v1 ==== 74+0+22121 (secure 0 0 0) 0x7faf6c073e70 con 0x7faf6c0fff40 2023-07-14T19:16:14.339 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-07-14T19:16:14.339 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":81,"fsid":"adc52f04-2279-11ee-9b33-001a4aab830c","created":"2023-07-14T19:08:56.893730+0000","modified":"2023-07-14T19:15:59.225912+0000","last_up_change":"2023-07-14T19:15:16.632979+0000","last_in_change":"2023-07-14T19:14:58.961702+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-07-14T19:12:03.271333+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-07-14T19:15:56.073464+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-07-14T19:15:57.018726+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"0d79356a-95d2-49ad-857b-2092ae9f7e13","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":3599703125},{"type":"v1","addr":"172.21.15.40:6803","nonce":3599703125}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":3599703125},{"type":"v1","addr":"172.21.15.40:6805","nonce":3599703125}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":3599703125},{"type":"v1","addr":"172.21.15.40:6809","nonce":3599703125}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":3599703125},{"type":"v1","addr":"172.21.15.40:6807","nonce":3599703125}]},"public_addr":"172.21.15.40:6803/3599703125","cluster_addr":"172.21.15.40:6805/3599703125","heartbeat_back_addr":"172.21.15.40:6809/3599703125","heartbeat_front_addr":"172.21.15.40:6807/3599703125","state":["exists","up"]},{"osd":1,"uuid":"40f8a3d1-a5e4-473c-b9f2-7600ec22be0d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6800","nonce":2512760702},{"type":"v1","addr":"172.21.15.130:6801","nonce":2512760702}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6802","nonce":2512760702},{"type":"v1","addr":"172.21.15.130:6803","nonce":2512760702}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6806","nonce":2512760702},{"type":"v1","addr":"172.21.15.130:6807","nonce":2512760702}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6804","nonce":2512760702},{"type":"v1","addr":"172.21.15.130:6805","nonce":2512760702}]},"public_addr":"172.21.15.130:6801/2512760702","cluster_addr":"172.21.15.130:6803/2512760702","heartbeat_back_addr":"172.21.15.130:6807/2512760702","heartbeat_front_addr":"172.21.15.130:6805/2512760702","state":["exists","up"]},{"osd":2,"uuid":"5a157891-cb7a-4ea6-9239-f48a7c67c003","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6800","nonce":589569135},{"type":"v1","addr":"172.21.15.191:6801","nonce":589569135}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6802","nonce":589569135},{"type":"v1","addr":"172.21.15.191:6803","nonce":589569135}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6806","nonce":589569135},{"type":"v1","addr":"172.21.15.191:6807","nonce":589569135}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6804","nonce":589569135},{"type":"v1","addr":"172.21.15.191:6805","nonce":589569135}]},"public_addr":"172.21.15.191:6801/589569135","cluster_addr":"172.21.15.191:6803/589569135","heartbeat_back_addr":"172.21.15.191:6807/589569135","heartbeat_front_addr":"172.21.15.191:6805/589569135","state":["exists","up"]},{"osd":3,"uuid":"8020e7dc-ac40-487f-857a-8da40bccb459","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":996287860},{"type":"v1","addr":"172.21.15.40:6811","nonce":996287860}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6812","nonce":996287860},{"type":"v1","addr":"172.21.15.40:6813","nonce":996287860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6816","nonce":996287860},{"type":"v1","addr":"172.21.15.40:6817","nonce":996287860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":996287860},{"type":"v1","addr":"172.21.15.40:6815","nonce":996287860}]},"public_addr":"172.21.15.40:6811/996287860","cluster_addr":"172.21.15.40:6813/996287860","heartbeat_back_addr":"172.21.15.40:6817/996287860","heartbeat_front_addr":"172.21.15.40:6815/996287860","state":["exists","up"]},{"osd":4,"uuid":"c5501b35-f62d-482d-8d22-115c88f0298e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6808","nonce":3507448720},{"type":"v1","addr":"172.21.15.130:6809","nonce":3507448720}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6810","nonce":3507448720},{"type":"v1","addr":"172.21.15.130:6811","nonce":3507448720}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6814","nonce":3507448720},{"type":"v1","addr":"172.21.15.130:6815","nonce":3507448720}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6812","nonce":3507448720},{"type":"v1","addr":"172.21.15.130:6813","nonce":3507448720}]},"public_addr":"172.21.15.130:6809/3507448720","cluster_addr":"172.21.15.130:6811/3507448720","heartbeat_back_addr":"172.21.15.130:6815/3507448720","heartbeat_front_addr":"172.21.15.130:6813/3507448720","state":["exists","up"]},{"osd":5,"uuid":"7770b6d6-5874-493a-986a-84db482cee42","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6808","nonce":1781944663},{"type":"v1","addr":"172.21.15.191:6809","nonce":1781944663}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6810","nonce":1781944663},{"type":"v1","addr":"172.21.15.191:6811","nonce":1781944663}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6814","nonce":1781944663},{"type":"v1","addr":"172.21.15.191:6815","nonce":1781944663}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6812","nonce":1781944663},{"type":"v1","addr":"172.21.15.191:6813","nonce":1781944663}]},"public_addr":"172.21.15.191:6809/1781944663","cluster_addr":"172.21.15.191:6811/1781944663","heartbeat_back_addr":"172.21.15.191:6815/1781944663","heartbeat_front_addr":"172.21.15.191:6813/1781944663","state":["exists","up"]},{"osd":6,"uuid":"ef65ad00-dfee-4300-9961-dfc8573aab06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6818","nonce":693186773},{"type":"v1","addr":"172.21.15.40:6819","nonce":693186773}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6820","nonce":693186773},{"type":"v1","addr":"172.21.15.40:6821","nonce":693186773}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6824","nonce":693186773},{"type":"v1","addr":"172.21.15.40:6825","nonce":693186773}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6822","nonce":693186773},{"type":"v1","addr":"172.21.15.40:6823","nonce":693186773}]},"public_addr":"172.21.15.40:6819/693186773","cluster_addr":"172.21.15.40:6821/693186773","heartbeat_back_addr":"172.21.15.40:6825/693186773","heartbeat_front_addr":"172.21.15.40:6823/693186773","state":["exists","up"]},{"osd":7,"uuid":"1700fbfa-cbfc-4f76-97be-727a3b100740","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":50,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6816","nonce":2384019528},{"type":"v1","addr":"172.21.15.130:6817","nonce":2384019528}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6818","nonce":2384019528},{"type":"v1","addr":"172.21.15.130:6819","nonce":2384019528}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6822","nonce":2384019528},{"type":"v1","addr":"172.21.15.130:6823","nonce":2384019528}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6820","nonce":2384019528},{"type":"v1","addr":"172.21.15.130:6821","nonce":2384019528}]},"public_addr":"172.21.15.130:6817/2384019528","cluster_addr":"172.21.15.130:6819/2384019528","heartbeat_back_addr":"172.21.15.130:6823/2384019528","heartbeat_front_addr":"172.21.15.130:6821/2384019528","state":["exists","up"]},{"osd":8,"uuid":"b76d4033-c1e2-457b-8571-c8a49fe443f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":57,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6816","nonce":2386818417},{"type":"v1","addr":"172.21.15.191:6817","nonce":2386818417}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6818","nonce":2386818417},{"type":"v1","addr":"172.21.15.191:6819","nonce":2386818417}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6822","nonce":2386818417},{"type":"v1","addr":"172.21.15.191:6823","nonce":2386818417}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6820","nonce":2386818417},{"type":"v1","addr":"172.21.15.191:6821","nonce":2386818417}]},"public_addr":"172.21.15.191:6817/2386818417","cluster_addr":"172.21.15.191:6819/2386818417","heartbeat_back_addr":"172.21.15.191:6823/2386818417","heartbeat_front_addr":"172.21.15.191:6821/2386818417","state":["exists","up"]},{"osd":9,"uuid":"df31bf2f-e30e-41b6-b2b9-0e26a07a88bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":64,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6826","nonce":2228380263},{"type":"v1","addr":"172.21.15.40:6827","nonce":2228380263}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6828","nonce":2228380263},{"type":"v1","addr":"172.21.15.40:6829","nonce":2228380263}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":2228380263},{"type":"v1","addr":"172.21.15.40:6833","nonce":2228380263}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6830","nonce":2228380263},{"type":"v1","addr":"172.21.15.40:6831","nonce":2228380263}]},"public_addr":"172.21.15.40:6827/2228380263","cluster_addr":"172.21.15.40:6829/2228380263","heartbeat_back_addr":"172.21.15.40:6833/2228380263","heartbeat_front_addr":"172.21.15.40:6831/2228380263","state":["exists","up"]},{"osd":10,"uuid":"4a93d621-49ee-4ad7-9e49-750614faf912","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6824","nonce":2417067798},{"type":"v1","addr":"172.21.15.130:6825","nonce":2417067798}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6826","nonce":2417067798},{"type":"v1","addr":"172.21.15.130:6827","nonce":2417067798}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6830","nonce":2417067798},{"type":"v1","addr":"172.21.15.130:6831","nonce":2417067798}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6828","nonce":2417067798},{"type":"v1","addr":"172.21.15.130:6829","nonce":2417067798}]},"public_addr":"172.21.15.130:6825/2417067798","cluster_addr":"172.21.15.130:6827/2417067798","heartbeat_back_addr":"172.21.15.130:6831/2417067798","heartbeat_front_addr":"172.21.15.130:6829/2417067798","state":["exists","up"]},{"osd":11,"uuid":"f43a2a6f-82f3-42ac-b77c-4f1b34da6918","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":76,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6824","nonce":2886059446},{"type":"v1","addr":"172.21.15.191:6825","nonce":2886059446}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6826","nonce":2886059446},{"type":"v1","addr":"172.21.15.191:6827","nonce":2886059446}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6830","nonce":2886059446},{"type":"v1","addr":"172.21.15.191:6831","nonce":2886059446}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6828","nonce":2886059446},{"type":"v1","addr":"172.21.15.191:6829","nonce":2886059446}]},"public_addr":"172.21.15.191:6825/2886059446","cluster_addr":"172.21.15.191:6827/2886059446","heartbeat_back_addr":"172.21.15.191:6831/2886059446","heartbeat_front_addr":"172.21.15.191:6829/2886059446","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:11:18.320495+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:11:39.901199+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:12:00.869042+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:12:23.292742+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:12:44.587019+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:13:06.494893+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:13:28.378374+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:13:50.033491+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:14:10.937062+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:14:32.358498+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:14:53.666584+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:15:14.837566+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.40:6801/3698757264":"2023-07-15T19:09:50.732458+0000","172.21.15.40:6800/3698757264":"2023-07-15T19:09:50.732458+0000","172.21.15.40:6800/2569015667":"2023-07-15T19:09:24.441543+0000","172.21.15.40:0/3653961003":"2023-07-15T19:09:24.441543+0000","172.21.15.40:0/3568861680":"2023-07-15T19:09:24.441543+0000","172.21.15.40:0/3249773172":"2023-07-15T19:09:50.732458+0000","172.21.15.40:0/4084215480":"2023-07-15T19:09:24.441543+0000","172.21.15.40:0/1163941760":"2023-07-15T19:09:50.732458+0000","172.21.15.40:6801/2569015667":"2023-07-15T19:09:24.441543+0000","172.21.15.40:0/2326639960":"2023-07-15T19:09:50.732458+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-07-14T19:16:14.340 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:14.338+0000 7faf714f6700 1 -- 172.21.15.40:0/4020848162 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7faf540a11e0 msgr2=0x7faf540a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:16:14.340 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:14.338+0000 7faf714f6700 1 --2- 172.21.15.40:0/4020848162 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7faf540a11e0 0x7faf540a3690 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7faf6c074130 tx=0x7faf5c00b410 comp rx=0 tx=0).stop 2023-07-14T19:16:14.340 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:14.338+0000 7faf714f6700 1 -- 172.21.15.40:0/4020848162 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7faf6c0fff40 msgr2=0x7faf6c072940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:16:14.340 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:14.338+0000 7faf714f6700 1 --2- 172.21.15.40:0/4020848162 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7faf6c0fff40 0x7faf6c072940 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7faf60009990 tx=0x7faf60009580 comp rx=0 tx=0).stop 2023-07-14T19:16:14.340 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:14.338+0000 7faf714f6700 1 -- 172.21.15.40:0/4020848162 shutdown_connections 2023-07-14T19:16:14.340 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:14.338+0000 7faf714f6700 1 --2- 172.21.15.40:0/4020848162 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7faf540a11e0 0x7faf540a3690 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:14.341 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:14.338+0000 7faf714f6700 1 --2- 172.21.15.40:0/4020848162 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7faf6c104fe0 0x7faf6c073400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:14.341 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:14.338+0000 7faf714f6700 1 --2- 172.21.15.40:0/4020848162 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7faf6c0fff40 0x7faf6c072940 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:14.341 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:14.338+0000 7faf714f6700 1 --2- 172.21.15.40:0/4020848162 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7faf6c1008e0 0x7faf6c072e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:14.341 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:14.338+0000 7faf714f6700 1 -- 172.21.15.40:0/4020848162 >> 172.21.15.40:0/4020848162 conn(0x7faf6c0fb7c0 msgr2=0x7faf6c103890 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:16:14.341 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:14.338+0000 7faf714f6700 1 -- 172.21.15.40:0/4020848162 shutdown_connections 2023-07-14T19:16:14.341 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:14.338+0000 7faf714f6700 1 -- 172.21.15.40:0/4020848162 wait complete. 2023-07-14T19:16:14.730 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2023-07-14T19:16:14.731 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell --fsid adc52f04-2279-11ee-9b33-001a4aab830c -- ceph osd dump --format=json 2023-07-14T19:16:14.971 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:16:14 smithi040 ceph-mon[96518]: pgmap v262: 97 pgs: 97 active+clean; 623 KiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail; 7.9 KiB/s rd, 7.6 KiB/s wr, 21 op/s 2023-07-14T19:16:14.971 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:16:14 smithi040 ceph-mon[96518]: from='client.? 172.21.15.40:0/4020848162' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T19:16:14.972 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:16:14 smithi191 ceph-mon[98311]: pgmap v262: 97 pgs: 97 active+clean; 623 KiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail; 7.9 KiB/s rd, 7.6 KiB/s wr, 21 op/s 2023-07-14T19:16:14.972 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:16:14 smithi191 ceph-mon[98311]: from='client.? 172.21.15.40:0/4020848162' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T19:16:14.992 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:16:14 smithi130 ceph-mon[97832]: pgmap v262: 97 pgs: 97 active+clean; 623 KiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail; 7.9 KiB/s rd, 7.6 KiB/s wr, 21 op/s 2023-07-14T19:16:14.992 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:16:14 smithi130 ceph-mon[97832]: from='client.? 172.21.15.40:0/4020848162' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T19:16:15.295 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/adc52f04-2279-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T19:16:16.522 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:16.518+0000 7fd48369e700 1 -- 172.21.15.40:0/3286744428 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd47c0ff7e0 msgr2=0x7fd47c1037b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:16:16.522 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:16.518+0000 7fd48369e700 1 --2- 172.21.15.40:0/3286744428 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd47c0ff7e0 0x7fd47c1037b0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fd46c0099c0 tx=0x7fd46c009cd0 comp rx=0 tx=0).stop 2023-07-14T19:16:16.522 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:16.518+0000 7fd48369e700 1 -- 172.21.15.40:0/3286744428 shutdown_connections 2023-07-14T19:16:16.522 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:16.518+0000 7fd48369e700 1 --2- 172.21.15.40:0/3286744428 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd47c0fee40 0x7fd47c0ff210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:16.522 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:16.518+0000 7fd48369e700 1 --2- 172.21.15.40:0/3286744428 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd47c103ee0 0x7fd47c10ba60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:16.523 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:16.518+0000 7fd48369e700 1 --2- 172.21.15.40:0/3286744428 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd47c0ff7e0 0x7fd47c1037b0 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:16.523 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:16.518+0000 7fd48369e700 1 -- 172.21.15.40:0/3286744428 >> 172.21.15.40:0/3286744428 conn(0x7fd47c075ca0 msgr2=0x7fd47c0760a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:16:16.523 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:16.518+0000 7fd48369e700 1 -- 172.21.15.40:0/3286744428 shutdown_connections 2023-07-14T19:16:16.523 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:16.518+0000 7fd48369e700 1 -- 172.21.15.40:0/3286744428 wait complete. 2023-07-14T19:16:16.523 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:16.519+0000 7fd48369e700 1 Processor -- start 2023-07-14T19:16:16.523 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:16.519+0000 7fd48369e700 1 -- start start 2023-07-14T19:16:16.524 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:16.519+0000 7fd48369e700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd47c0fee40 0x7fd47c100850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:16:16.524 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:16.519+0000 7fd48369e700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd47c0ff7e0 0x7fd47c100d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:16:16.524 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:16.519+0000 7fd48369e700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd47c103ee0 0x7fd47c101310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:16:16.524 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:16.519+0000 7fd48369e700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd47c1a4200 con 0x7fd47c0fee40 2023-07-14T19:16:16.524 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:16.519+0000 7fd48369e700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd47c1a4370 con 0x7fd47c0ff7e0 2023-07-14T19:16:16.524 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:16.519+0000 7fd48369e700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd47c1a4530 con 0x7fd47c103ee0 2023-07-14T19:16:16.524 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:16.519+0000 7fd48143a700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd47c0fee40 0x7fd47c100850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:16:16.525 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:16.519+0000 7fd48143a700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd47c0fee40 0x7fd47c100850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:33138/0 (socket says 172.21.15.40:33138) 2023-07-14T19:16:16.525 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:16.519+0000 7fd48143a700 1 -- 172.21.15.40:0/482870766 learned_addr learned my addr 172.21.15.40:0/482870766 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-07-14T19:16:16.525 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:16.519+0000 7fd480c39700 1 --2- 172.21.15.40:0/482870766 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd47c0ff7e0 0x7fd47c100d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:16:16.525 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:16.519+0000 7fd481c3b700 1 --2- 172.21.15.40:0/482870766 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd47c103ee0 0x7fd47c101310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:16:16.525 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:16.520+0000 7fd48143a700 1 -- 172.21.15.40:0/482870766 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd47c103ee0 msgr2=0x7fd47c101310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:16:16.526 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:16.520+0000 7fd48143a700 1 --2- 172.21.15.40:0/482870766 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd47c103ee0 0x7fd47c101310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:16.526 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:16.520+0000 7fd48143a700 1 -- 172.21.15.40:0/482870766 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd47c0ff7e0 msgr2=0x7fd47c100d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:16:16.526 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:16.520+0000 7fd48143a700 1 --2- 172.21.15.40:0/482870766 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd47c0ff7e0 0x7fd47c100d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:16.526 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:16.520+0000 7fd48143a700 1 -- 172.21.15.40:0/482870766 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd46c0095f0 con 0x7fd47c0fee40 2023-07-14T19:16:16.526 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:16.520+0000 7fd48143a700 1 --2- 172.21.15.40:0/482870766 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd47c0fee40 0x7fd47c100850 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fd47800d950 tx=0x7fd47800dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:16:16.526 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:16.520+0000 7fd4727fc700 1 -- 172.21.15.40:0/482870766 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd47800f890 con 0x7fd47c0fee40 2023-07-14T19:16:16.526 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:16.520+0000 7fd48369e700 1 -- 172.21.15.40:0/482870766 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd47c1a4810 con 0x7fd47c0fee40 2023-07-14T19:16:16.527 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:16.520+0000 7fd4727fc700 1 -- 172.21.15.40:0/482870766 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd478010460 con 0x7fd47c0fee40 2023-07-14T19:16:16.527 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:16.520+0000 7fd4727fc700 1 -- 172.21.15.40:0/482870766 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd47800e610 con 0x7fd47c0fee40 2023-07-14T19:16:16.527 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:16.520+0000 7fd48369e700 1 -- 172.21.15.40:0/482870766 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd47c1a4d30 con 0x7fd47c0fee40 2023-07-14T19:16:16.528 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:16.521+0000 7fd48369e700 1 -- 172.21.15.40:0/482870766 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd47c04e340 con 0x7fd47c0fee40 2023-07-14T19:16:16.528 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:16.522+0000 7fd4727fc700 1 -- 172.21.15.40:0/482870766 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7fd4780108d0 con 0x7fd47c0fee40 2023-07-14T19:16:16.528 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:16.523+0000 7fd4727fc700 1 --2- 172.21.15.40:0/482870766 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7fd4680a11e0 0x7fd4680a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:16:16.528 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:16.523+0000 7fd480c39700 1 --2- 172.21.15.40:0/482870766 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7fd4680a11e0 0x7fd4680a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:16:16.529 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:16.523+0000 7fd4727fc700 1 -- 172.21.15.40:0/482870766 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7979+0+0 (secure 0 0 0) 0x7fd4780cc6b0 con 0x7fd47c0fee40 2023-07-14T19:16:16.529 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:16.523+0000 7fd480c39700 1 --2- 172.21.15.40:0/482870766 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7fd4680a11e0 0x7fd4680a3690 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fd46c00b580 tx=0x7fd46c003680 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:16:16.529 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:16.526+0000 7fd4727fc700 1 -- 172.21.15.40:0/482870766 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fd478095e50 con 0x7fd47c0fee40 2023-07-14T19:16:16.728 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:16.726+0000 7fd48369e700 1 -- 172.21.15.40:0/482870766 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd47c1a4f50 con 0x7fd47c0fee40 2023-07-14T19:16:16.729 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:16.727+0000 7fd4727fc700 1 -- 172.21.15.40:0/482870766 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v81) v1 ==== 74+0+22121 (secure 0 0 0) 0x7fd478016090 con 0x7fd47c0fee40 2023-07-14T19:16:16.729 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-07-14T19:16:16.730 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":81,"fsid":"adc52f04-2279-11ee-9b33-001a4aab830c","created":"2023-07-14T19:08:56.893730+0000","modified":"2023-07-14T19:15:59.225912+0000","last_up_change":"2023-07-14T19:15:16.632979+0000","last_in_change":"2023-07-14T19:14:58.961702+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-07-14T19:12:03.271333+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-07-14T19:15:56.073464+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-07-14T19:15:57.018726+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"0d79356a-95d2-49ad-857b-2092ae9f7e13","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":3599703125},{"type":"v1","addr":"172.21.15.40:6803","nonce":3599703125}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":3599703125},{"type":"v1","addr":"172.21.15.40:6805","nonce":3599703125}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":3599703125},{"type":"v1","addr":"172.21.15.40:6809","nonce":3599703125}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":3599703125},{"type":"v1","addr":"172.21.15.40:6807","nonce":3599703125}]},"public_addr":"172.21.15.40:6803/3599703125","cluster_addr":"172.21.15.40:6805/3599703125","heartbeat_back_addr":"172.21.15.40:6809/3599703125","heartbeat_front_addr":"172.21.15.40:6807/3599703125","state":["exists","up"]},{"osd":1,"uuid":"40f8a3d1-a5e4-473c-b9f2-7600ec22be0d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6800","nonce":2512760702},{"type":"v1","addr":"172.21.15.130:6801","nonce":2512760702}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6802","nonce":2512760702},{"type":"v1","addr":"172.21.15.130:6803","nonce":2512760702}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6806","nonce":2512760702},{"type":"v1","addr":"172.21.15.130:6807","nonce":2512760702}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6804","nonce":2512760702},{"type":"v1","addr":"172.21.15.130:6805","nonce":2512760702}]},"public_addr":"172.21.15.130:6801/2512760702","cluster_addr":"172.21.15.130:6803/2512760702","heartbeat_back_addr":"172.21.15.130:6807/2512760702","heartbeat_front_addr":"172.21.15.130:6805/2512760702","state":["exists","up"]},{"osd":2,"uuid":"5a157891-cb7a-4ea6-9239-f48a7c67c003","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6800","nonce":589569135},{"type":"v1","addr":"172.21.15.191:6801","nonce":589569135}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6802","nonce":589569135},{"type":"v1","addr":"172.21.15.191:6803","nonce":589569135}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6806","nonce":589569135},{"type":"v1","addr":"172.21.15.191:6807","nonce":589569135}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6804","nonce":589569135},{"type":"v1","addr":"172.21.15.191:6805","nonce":589569135}]},"public_addr":"172.21.15.191:6801/589569135","cluster_addr":"172.21.15.191:6803/589569135","heartbeat_back_addr":"172.21.15.191:6807/589569135","heartbeat_front_addr":"172.21.15.191:6805/589569135","state":["exists","up"]},{"osd":3,"uuid":"8020e7dc-ac40-487f-857a-8da40bccb459","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":996287860},{"type":"v1","addr":"172.21.15.40:6811","nonce":996287860}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6812","nonce":996287860},{"type":"v1","addr":"172.21.15.40:6813","nonce":996287860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6816","nonce":996287860},{"type":"v1","addr":"172.21.15.40:6817","nonce":996287860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":996287860},{"type":"v1","addr":"172.21.15.40:6815","nonce":996287860}]},"public_addr":"172.21.15.40:6811/996287860","cluster_addr":"172.21.15.40:6813/996287860","heartbeat_back_addr":"172.21.15.40:6817/996287860","heartbeat_front_addr":"172.21.15.40:6815/996287860","state":["exists","up"]},{"osd":4,"uuid":"c5501b35-f62d-482d-8d22-115c88f0298e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6808","nonce":3507448720},{"type":"v1","addr":"172.21.15.130:6809","nonce":3507448720}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6810","nonce":3507448720},{"type":"v1","addr":"172.21.15.130:6811","nonce":3507448720}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6814","nonce":3507448720},{"type":"v1","addr":"172.21.15.130:6815","nonce":3507448720}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6812","nonce":3507448720},{"type":"v1","addr":"172.21.15.130:6813","nonce":3507448720}]},"public_addr":"172.21.15.130:6809/3507448720","cluster_addr":"172.21.15.130:6811/3507448720","heartbeat_back_addr":"172.21.15.130:6815/3507448720","heartbeat_front_addr":"172.21.15.130:6813/3507448720","state":["exists","up"]},{"osd":5,"uuid":"7770b6d6-5874-493a-986a-84db482cee42","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6808","nonce":1781944663},{"type":"v1","addr":"172.21.15.191:6809","nonce":1781944663}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6810","nonce":1781944663},{"type":"v1","addr":"172.21.15.191:6811","nonce":1781944663}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6814","nonce":1781944663},{"type":"v1","addr":"172.21.15.191:6815","nonce":1781944663}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6812","nonce":1781944663},{"type":"v1","addr":"172.21.15.191:6813","nonce":1781944663}]},"public_addr":"172.21.15.191:6809/1781944663","cluster_addr":"172.21.15.191:6811/1781944663","heartbeat_back_addr":"172.21.15.191:6815/1781944663","heartbeat_front_addr":"172.21.15.191:6813/1781944663","state":["exists","up"]},{"osd":6,"uuid":"ef65ad00-dfee-4300-9961-dfc8573aab06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6818","nonce":693186773},{"type":"v1","addr":"172.21.15.40:6819","nonce":693186773}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6820","nonce":693186773},{"type":"v1","addr":"172.21.15.40:6821","nonce":693186773}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6824","nonce":693186773},{"type":"v1","addr":"172.21.15.40:6825","nonce":693186773}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6822","nonce":693186773},{"type":"v1","addr":"172.21.15.40:6823","nonce":693186773}]},"public_addr":"172.21.15.40:6819/693186773","cluster_addr":"172.21.15.40:6821/693186773","heartbeat_back_addr":"172.21.15.40:6825/693186773","heartbeat_front_addr":"172.21.15.40:6823/693186773","state":["exists","up"]},{"osd":7,"uuid":"1700fbfa-cbfc-4f76-97be-727a3b100740","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":50,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6816","nonce":2384019528},{"type":"v1","addr":"172.21.15.130:6817","nonce":2384019528}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6818","nonce":2384019528},{"type":"v1","addr":"172.21.15.130:6819","nonce":2384019528}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6822","nonce":2384019528},{"type":"v1","addr":"172.21.15.130:6823","nonce":2384019528}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6820","nonce":2384019528},{"type":"v1","addr":"172.21.15.130:6821","nonce":2384019528}]},"public_addr":"172.21.15.130:6817/2384019528","cluster_addr":"172.21.15.130:6819/2384019528","heartbeat_back_addr":"172.21.15.130:6823/2384019528","heartbeat_front_addr":"172.21.15.130:6821/2384019528","state":["exists","up"]},{"osd":8,"uuid":"b76d4033-c1e2-457b-8571-c8a49fe443f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":57,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6816","nonce":2386818417},{"type":"v1","addr":"172.21.15.191:6817","nonce":2386818417}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6818","nonce":2386818417},{"type":"v1","addr":"172.21.15.191:6819","nonce":2386818417}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6822","nonce":2386818417},{"type":"v1","addr":"172.21.15.191:6823","nonce":2386818417}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6820","nonce":2386818417},{"type":"v1","addr":"172.21.15.191:6821","nonce":2386818417}]},"public_addr":"172.21.15.191:6817/2386818417","cluster_addr":"172.21.15.191:6819/2386818417","heartbeat_back_addr":"172.21.15.191:6823/2386818417","heartbeat_front_addr":"172.21.15.191:6821/2386818417","state":["exists","up"]},{"osd":9,"uuid":"df31bf2f-e30e-41b6-b2b9-0e26a07a88bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":64,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6826","nonce":2228380263},{"type":"v1","addr":"172.21.15.40:6827","nonce":2228380263}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6828","nonce":2228380263},{"type":"v1","addr":"172.21.15.40:6829","nonce":2228380263}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":2228380263},{"type":"v1","addr":"172.21.15.40:6833","nonce":2228380263}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6830","nonce":2228380263},{"type":"v1","addr":"172.21.15.40:6831","nonce":2228380263}]},"public_addr":"172.21.15.40:6827/2228380263","cluster_addr":"172.21.15.40:6829/2228380263","heartbeat_back_addr":"172.21.15.40:6833/2228380263","heartbeat_front_addr":"172.21.15.40:6831/2228380263","state":["exists","up"]},{"osd":10,"uuid":"4a93d621-49ee-4ad7-9e49-750614faf912","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6824","nonce":2417067798},{"type":"v1","addr":"172.21.15.130:6825","nonce":2417067798}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6826","nonce":2417067798},{"type":"v1","addr":"172.21.15.130:6827","nonce":2417067798}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6830","nonce":2417067798},{"type":"v1","addr":"172.21.15.130:6831","nonce":2417067798}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6828","nonce":2417067798},{"type":"v1","addr":"172.21.15.130:6829","nonce":2417067798}]},"public_addr":"172.21.15.130:6825/2417067798","cluster_addr":"172.21.15.130:6827/2417067798","heartbeat_back_addr":"172.21.15.130:6831/2417067798","heartbeat_front_addr":"172.21.15.130:6829/2417067798","state":["exists","up"]},{"osd":11,"uuid":"f43a2a6f-82f3-42ac-b77c-4f1b34da6918","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":76,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6824","nonce":2886059446},{"type":"v1","addr":"172.21.15.191:6825","nonce":2886059446}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6826","nonce":2886059446},{"type":"v1","addr":"172.21.15.191:6827","nonce":2886059446}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6830","nonce":2886059446},{"type":"v1","addr":"172.21.15.191:6831","nonce":2886059446}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6828","nonce":2886059446},{"type":"v1","addr":"172.21.15.191:6829","nonce":2886059446}]},"public_addr":"172.21.15.191:6825/2886059446","cluster_addr":"172.21.15.191:6827/2886059446","heartbeat_back_addr":"172.21.15.191:6831/2886059446","heartbeat_front_addr":"172.21.15.191:6829/2886059446","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:11:18.320495+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:11:39.901199+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:12:00.869042+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:12:23.292742+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:12:44.587019+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:13:06.494893+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:13:28.378374+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:13:50.033491+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:14:10.937062+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:14:32.358498+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:14:53.666584+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:15:14.837566+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.40:6801/3698757264":"2023-07-15T19:09:50.732458+0000","172.21.15.40:6800/3698757264":"2023-07-15T19:09:50.732458+0000","172.21.15.40:6800/2569015667":"2023-07-15T19:09:24.441543+0000","172.21.15.40:0/3653961003":"2023-07-15T19:09:24.441543+0000","172.21.15.40:0/3568861680":"2023-07-15T19:09:24.441543+0000","172.21.15.40:0/3249773172":"2023-07-15T19:09:50.732458+0000","172.21.15.40:0/4084215480":"2023-07-15T19:09:24.441543+0000","172.21.15.40:0/1163941760":"2023-07-15T19:09:50.732458+0000","172.21.15.40:6801/2569015667":"2023-07-15T19:09:24.441543+0000","172.21.15.40:0/2326639960":"2023-07-15T19:09:50.732458+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-07-14T19:16:16.731 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:16.729+0000 7fd48369e700 1 -- 172.21.15.40:0/482870766 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7fd4680a11e0 msgr2=0x7fd4680a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:16:16.731 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:16.729+0000 7fd48369e700 1 --2- 172.21.15.40:0/482870766 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7fd4680a11e0 0x7fd4680a3690 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fd46c00b580 tx=0x7fd46c003680 comp rx=0 tx=0).stop 2023-07-14T19:16:16.732 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:16.729+0000 7fd48369e700 1 -- 172.21.15.40:0/482870766 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd47c0fee40 msgr2=0x7fd47c100850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:16:16.732 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:16.729+0000 7fd48369e700 1 --2- 172.21.15.40:0/482870766 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd47c0fee40 0x7fd47c100850 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fd47800d950 tx=0x7fd47800dc60 comp rx=0 tx=0).stop 2023-07-14T19:16:16.732 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:16.730+0000 7fd48369e700 1 -- 172.21.15.40:0/482870766 shutdown_connections 2023-07-14T19:16:16.732 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:16.730+0000 7fd48369e700 1 --2- 172.21.15.40:0/482870766 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7fd4680a11e0 0x7fd4680a3690 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:16.732 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:16.730+0000 7fd48369e700 1 --2- 172.21.15.40:0/482870766 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fd47c103ee0 0x7fd47c101310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:16.732 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:16.730+0000 7fd48369e700 1 --2- 172.21.15.40:0/482870766 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd47c0fee40 0x7fd47c100850 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:16.732 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:16.730+0000 7fd48369e700 1 --2- 172.21.15.40:0/482870766 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd47c0ff7e0 0x7fd47c100d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:16.733 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:16.730+0000 7fd48369e700 1 -- 172.21.15.40:0/482870766 >> 172.21.15.40:0/482870766 conn(0x7fd47c075ca0 msgr2=0x7fd47c068100 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:16:16.733 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:16.730+0000 7fd48369e700 1 -- 172.21.15.40:0/482870766 shutdown_connections 2023-07-14T19:16:16.733 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:16.730+0000 7fd48369e700 1 -- 172.21.15.40:0/482870766 wait complete. 2023-07-14T19:16:16.971 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:16:16 smithi040 ceph-mon[96518]: pgmap v263: 97 pgs: 97 active+clean; 623 KiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail; 8.3 KiB/s rd, 7.1 KiB/s wr, 21 op/s 2023-07-14T19:16:16.971 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:16:16 smithi191 ceph-mon[98311]: pgmap v263: 97 pgs: 97 active+clean; 623 KiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail; 8.3 KiB/s rd, 7.1 KiB/s wr, 21 op/s 2023-07-14T19:16:16.992 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:16:16 smithi130 ceph-mon[97832]: pgmap v263: 97 pgs: 97 active+clean; 623 KiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail; 8.3 KiB/s rd, 7.1 KiB/s wr, 21 op/s 2023-07-14T19:16:17.244 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell --fsid adc52f04-2279-11ee-9b33-001a4aab830c -- ceph tell osd.0 flush_pg_stats 2023-07-14T19:16:17.245 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell --fsid adc52f04-2279-11ee-9b33-001a4aab830c -- ceph tell osd.1 flush_pg_stats 2023-07-14T19:16:17.245 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell --fsid adc52f04-2279-11ee-9b33-001a4aab830c -- ceph tell osd.2 flush_pg_stats 2023-07-14T19:16:17.246 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell --fsid adc52f04-2279-11ee-9b33-001a4aab830c -- ceph tell osd.3 flush_pg_stats 2023-07-14T19:16:17.246 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell --fsid adc52f04-2279-11ee-9b33-001a4aab830c -- ceph tell osd.4 flush_pg_stats 2023-07-14T19:16:17.247 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell --fsid adc52f04-2279-11ee-9b33-001a4aab830c -- ceph tell osd.5 flush_pg_stats 2023-07-14T19:16:17.247 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell --fsid adc52f04-2279-11ee-9b33-001a4aab830c -- ceph tell osd.6 flush_pg_stats 2023-07-14T19:16:17.248 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell --fsid adc52f04-2279-11ee-9b33-001a4aab830c -- ceph tell osd.7 flush_pg_stats 2023-07-14T19:16:17.248 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell --fsid adc52f04-2279-11ee-9b33-001a4aab830c -- ceph tell osd.8 flush_pg_stats 2023-07-14T19:16:17.249 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell --fsid adc52f04-2279-11ee-9b33-001a4aab830c -- ceph tell osd.9 flush_pg_stats 2023-07-14T19:16:17.249 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell --fsid adc52f04-2279-11ee-9b33-001a4aab830c -- ceph tell osd.10 flush_pg_stats 2023-07-14T19:16:17.250 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell --fsid adc52f04-2279-11ee-9b33-001a4aab830c -- ceph tell osd.11 flush_pg_stats 2023-07-14T19:16:17.694 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:16:17 smithi040 ceph-mon[96518]: from='client.? 172.21.15.40:0/482870766' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T19:16:17.972 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:16:17 smithi191 ceph-mon[98311]: from='client.? 172.21.15.40:0/482870766' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T19:16:17.992 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:16:17 smithi130 ceph-mon[97832]: from='client.? 172.21.15.40:0/482870766' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T19:16:18.166 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/adc52f04-2279-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T19:16:18.185 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/adc52f04-2279-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T19:16:18.200 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/adc52f04-2279-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T19:16:18.538 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/adc52f04-2279-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T19:16:18.764 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/adc52f04-2279-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T19:16:18.782 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/adc52f04-2279-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T19:16:18.918 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/adc52f04-2279-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T19:16:18.971 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:16:18 smithi040 ceph-mon[96518]: pgmap v264: 97 pgs: 97 active+clean; 637 KiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 7.5 KiB/s wr, 20 op/s 2023-07-14T19:16:18.972 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:16:18 smithi191 ceph-mon[98311]: pgmap v264: 97 pgs: 97 active+clean; 637 KiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 7.5 KiB/s wr, 20 op/s 2023-07-14T19:16:18.992 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:16:18 smithi130 ceph-mon[97832]: pgmap v264: 97 pgs: 97 active+clean; 637 KiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 7.5 KiB/s wr, 20 op/s 2023-07-14T19:16:19.121 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/adc52f04-2279-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T19:16:19.261 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/adc52f04-2279-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T19:16:19.589 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/adc52f04-2279-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T19:16:20.311 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/adc52f04-2279-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T19:16:20.971 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:16:20 smithi040 ceph-mon[96518]: pgmap v265: 97 pgs: 97 active+clean; 637 KiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 3.4 KiB/s wr, 15 op/s 2023-07-14T19:16:20.972 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:16:20 smithi191 ceph-mon[98311]: pgmap v265: 97 pgs: 97 active+clean; 637 KiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 3.4 KiB/s wr, 15 op/s 2023-07-14T19:16:20.992 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:16:20 smithi130 ceph-mon[97832]: pgmap v265: 97 pgs: 97 active+clean; 637 KiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 3.4 KiB/s wr, 15 op/s 2023-07-14T19:16:22.971 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:16:22 smithi040 ceph-mon[96518]: pgmap v266: 97 pgs: 97 active+clean; 637 KiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 2.1 KiB/s wr, 12 op/s 2023-07-14T19:16:22.972 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:16:22 smithi191 ceph-mon[98311]: pgmap v266: 97 pgs: 97 active+clean; 637 KiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 2.1 KiB/s wr, 12 op/s 2023-07-14T19:16:22.992 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:16:22 smithi130 ceph-mon[97832]: pgmap v266: 97 pgs: 97 active+clean; 637 KiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 2.1 KiB/s wr, 12 op/s 2023-07-14T19:16:23.509 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/adc52f04-2279-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T19:16:23.510 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:23.507+0000 7f6e56e46700 1 -- 172.21.15.40:0/210491744 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6e500fee70 msgr2=0x7f6e500ff240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:16:23.511 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:23.507+0000 7f6e56e46700 1 --2- 172.21.15.40:0/210491744 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6e500fee70 0x7f6e500ff240 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f6e440099c0 tx=0x7f6e44009cd0 comp rx=0 tx=0).stop 2023-07-14T19:16:23.511 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:23.507+0000 7f6e56e46700 1 -- 172.21.15.40:0/210491744 shutdown_connections 2023-07-14T19:16:23.511 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:23.507+0000 7f6e56e46700 1 --2- 172.21.15.40:0/210491744 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6e50103f10 0x7f6e5010ba90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:23.511 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:23.507+0000 7f6e56e46700 1 --2- 172.21.15.40:0/210491744 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6e500fee70 0x7f6e500ff240 secure :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f6e440099c0 tx=0x7f6e44009cd0 comp rx=0 tx=0).stop 2023-07-14T19:16:23.511 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:23.507+0000 7f6e56e46700 1 --2- 172.21.15.40:0/210491744 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6e500ff810 0x7f6e501037e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:23.511 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:23.507+0000 7f6e56e46700 1 -- 172.21.15.40:0/210491744 >> 172.21.15.40:0/210491744 conn(0x7f6e50075ab0 msgr2=0x7f6e50075eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:16:23.512 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:23.508+0000 7f6e56e46700 1 -- 172.21.15.40:0/210491744 shutdown_connections 2023-07-14T19:16:23.512 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:23.508+0000 7f6e56e46700 1 -- 172.21.15.40:0/210491744 wait complete. 2023-07-14T19:16:23.512 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:23.508+0000 7f6e56e46700 1 Processor -- start 2023-07-14T19:16:23.512 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:23.508+0000 7f6e56e46700 1 -- start start 2023-07-14T19:16:23.512 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:23.508+0000 7f6e56e46700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6e500ff810 0x7f6e50198200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:16:23.512 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:23.508+0000 7f6e56e46700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6e50103f10 0x7f6e50198740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:16:23.512 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:23.509+0000 7f6e56e46700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6e5019cbb0 0x7f6e5019d020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:16:23.512 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:23.509+0000 7f6e56e46700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e5019d590 con 0x7f6e5019cbb0 2023-07-14T19:16:23.513 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:23.509+0000 7f6e56e46700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e5019d700 con 0x7f6e50103f10 2023-07-14T19:16:23.513 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:23.509+0000 7f6e56e46700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e5019d8c0 con 0x7f6e500ff810 2023-07-14T19:16:23.513 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:23.509+0000 7f6e553e3700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6e5019cbb0 0x7f6e5019d020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:16:23.513 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:23.509+0000 7f6e553e3700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6e5019cbb0 0x7f6e5019d020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:33154/0 (socket says 172.21.15.40:33154) 2023-07-14T19:16:23.513 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:23.509+0000 7f6e553e3700 1 -- 172.21.15.40:0/524128932 learned_addr learned my addr 172.21.15.40:0/524128932 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-07-14T19:16:23.513 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:23.509+0000 7f6e4ffff700 1 --2- 172.21.15.40:0/524128932 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6e50103f10 0x7f6e50198740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:16:23.513 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:23.509+0000 7f6e54be2700 1 --2- 172.21.15.40:0/524128932 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6e500ff810 0x7f6e50198200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:16:23.514 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:23.509+0000 7f6e553e3700 1 -- 172.21.15.40:0/524128932 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6e500ff810 msgr2=0x7f6e50198200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:16:23.514 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:23.509+0000 7f6e553e3700 1 --2- 172.21.15.40:0/524128932 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6e500ff810 0x7f6e50198200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:23.514 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:23.509+0000 7f6e553e3700 1 -- 172.21.15.40:0/524128932 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6e50103f10 msgr2=0x7f6e50198740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:16:23.514 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:23.509+0000 7f6e553e3700 1 --2- 172.21.15.40:0/524128932 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6e50103f10 0x7f6e50198740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:23.514 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:23.509+0000 7f6e553e3700 1 -- 172.21.15.40:0/524128932 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e440095f0 con 0x7f6e5019cbb0 2023-07-14T19:16:23.515 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:23.509+0000 7f6e553e3700 1 --2- 172.21.15.40:0/524128932 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6e5019cbb0 0x7f6e5019d020 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f6e4800ef30 tx=0x7f6e4800c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:16:23.516 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:23.509+0000 7f6e4dffb700 1 -- 172.21.15.40:0/524128932 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e48018070 con 0x7f6e5019cbb0 2023-07-14T19:16:23.516 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:23.509+0000 7f6e56e46700 1 -- 172.21.15.40:0/524128932 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6e501a4e60 con 0x7f6e5019cbb0 2023-07-14T19:16:23.516 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:23.509+0000 7f6e4dffb700 1 -- 172.21.15.40:0/524128932 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f6e4800f040 con 0x7f6e5019cbb0 2023-07-14T19:16:23.516 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:23.509+0000 7f6e4dffb700 1 -- 172.21.15.40:0/524128932 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e4800cdb0 con 0x7f6e5019cbb0 2023-07-14T19:16:23.516 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:23.510+0000 7f6e56e46700 1 -- 172.21.15.40:0/524128932 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e501a52d0 con 0x7f6e5019cbb0 2023-07-14T19:16:23.516 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:23.511+0000 7f6e56e46700 1 -- 172.21.15.40:0/524128932 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f6e5004e340 con 0x7f6e5019cbb0 2023-07-14T19:16:23.516 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:23.512+0000 7f6e4dffb700 1 -- 172.21.15.40:0/524128932 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7f6e48003a00 con 0x7f6e5019cbb0 2023-07-14T19:16:23.517 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:23.512+0000 7f6e4dffb700 1 --2- 172.21.15.40:0/524128932 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f6e400a11c0 0x7f6e400a3670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:16:23.517 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:23.512+0000 7f6e4dffb700 1 -- 172.21.15.40:0/524128932 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f6e480cc430 con 0x7f6e5019cbb0 2023-07-14T19:16:23.517 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:23.512+0000 7f6e54be2700 1 --2- 172.21.15.40:0/524128932 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f6e400a11c0 0x7f6e400a3670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:16:23.517 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:23.512+0000 7f6e4dffb700 1 --2- 172.21.15.40:0/524128932 >> [v2:172.21.15.40:6802/3599703125,v1:172.21.15.40:6803/3599703125] conn(0x7f6e400aa990 0x7f6e400acda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:16:23.517 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:23.512+0000 7f6e4dffb700 1 -- 172.21.15.40:0/524128932 --> [v2:172.21.15.40:6802/3599703125,v1:172.21.15.40:6803/3599703125] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f6e400ad450 con 0x7f6e400aa990 2023-07-14T19:16:23.517 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:23.512+0000 7f6e4dffb700 1 -- 172.21.15.40:0/524128932 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_get_version_reply(handle=1 version=81) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6e480ceb40 con 0x7f6e5019cbb0 2023-07-14T19:16:23.517 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:23.513+0000 7f6e4ffff700 1 --2- 172.21.15.40:0/524128932 >> [v2:172.21.15.40:6802/3599703125,v1:172.21.15.40:6803/3599703125] conn(0x7f6e400aa990 0x7f6e400acda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:16:23.517 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:23.513+0000 7f6e54be2700 1 --2- 172.21.15.40:0/524128932 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f6e400a11c0 0x7f6e400a3670 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f6e440098e0 tx=0x7f6e4401a040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:16:23.518 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:23.513+0000 7f6e4ffff700 1 --2- 172.21.15.40:0/524128932 >> [v2:172.21.15.40:6802/3599703125,v1:172.21.15.40:6803/3599703125] conn(0x7f6e400aa990 0x7f6e400acda0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:16:23.518 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:23.514+0000 7f6e4dffb700 1 -- 172.21.15.40:0/524128932 <== osd.0 v2:172.21.15.40:6802/3599703125 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7f6e400ad450 con 0x7f6e400aa990 2023-07-14T19:16:23.537 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:23.536+0000 7f6e56e46700 1 -- 172.21.15.40:0/524128932 --> [v2:172.21.15.40:6802/3599703125,v1:172.21.15.40:6803/3599703125] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f6e501120d0 con 0x7f6e400aa990 2023-07-14T19:16:23.538 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:23.536+0000 7f6e4dffb700 1 -- 172.21.15.40:0/524128932 <== osd.0 v2:172.21.15.40:6802/3599703125 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f6e3c007c50 con 0x7f6e400aa990 2023-07-14T19:16:23.539 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:23.537+0000 7f6e56e46700 1 -- 172.21.15.40:0/524128932 >> [v2:172.21.15.40:6802/3599703125,v1:172.21.15.40:6803/3599703125] conn(0x7f6e400aa990 msgr2=0x7f6e400acda0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:16:23.539 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:23.537+0000 7f6e56e46700 1 --2- 172.21.15.40:0/524128932 >> [v2:172.21.15.40:6802/3599703125,v1:172.21.15.40:6803/3599703125] conn(0x7f6e400aa990 0x7f6e400acda0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:23.539 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:23.537+0000 7f6e56e46700 1 -- 172.21.15.40:0/524128932 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f6e400a11c0 msgr2=0x7f6e400a3670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:16:23.539 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:23.537+0000 7f6e56e46700 1 --2- 172.21.15.40:0/524128932 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f6e400a11c0 0x7f6e400a3670 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f6e440098e0 tx=0x7f6e4401a040 comp rx=0 tx=0).stop 2023-07-14T19:16:23.539 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:23.537+0000 7f6e56e46700 1 -- 172.21.15.40:0/524128932 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6e5019cbb0 msgr2=0x7f6e5019d020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:16:23.540 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:23.537+0000 7f6e56e46700 1 --2- 172.21.15.40:0/524128932 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6e5019cbb0 0x7f6e5019d020 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f6e4800ef30 tx=0x7f6e4800c3a0 comp rx=0 tx=0).stop 2023-07-14T19:16:23.540 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:23.537+0000 7f6e553e3700 1 -- 172.21.15.40:0/524128932 reap_dead start 2023-07-14T19:16:23.540 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:23.538+0000 7f6e56e46700 1 -- 172.21.15.40:0/524128932 shutdown_connections 2023-07-14T19:16:23.540 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:23.538+0000 7f6e56e46700 1 -- 172.21.15.40:0/524128932 >> 172.21.15.40:0/524128932 conn(0x7f6e50075ab0 msgr2=0x7f6e500fc850 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:16:23.541 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:23.538+0000 7f6e56e46700 1 -- 172.21.15.40:0/524128932 shutdown_connections 2023-07-14T19:16:23.541 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:23.538+0000 7f6e56e46700 1 -- 172.21.15.40:0/524128932 wait complete. 2023-07-14T19:16:24.413 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:24.409+0000 7f937a8ee700 1 -- 172.21.15.40:0/397309865 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9374101830 msgr2=0x7f9374109d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:16:24.413 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:24.409+0000 7f937a8ee700 1 --2- 172.21.15.40:0/397309865 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9374101830 0x7f9374109d30 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f9368009990 tx=0x7f9368009ca0 comp rx=0 tx=0).stop 2023-07-14T19:16:24.414 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:24.409+0000 7f937a8ee700 1 -- 172.21.15.40:0/397309865 shutdown_connections 2023-07-14T19:16:24.414 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:24.409+0000 7f937a8ee700 1 --2- 172.21.15.40:0/397309865 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f937410a270 0x7f937410c6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:24.414 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:24.409+0000 7f937a8ee700 1 --2- 172.21.15.40:0/397309865 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9374100f20 0x7f93741012f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:24.414 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:24.409+0000 7f937a8ee700 1 --2- 172.21.15.40:0/397309865 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9374101830 0x7f9374109d30 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:24.415 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:24.409+0000 7f937a8ee700 1 -- 172.21.15.40:0/397309865 >> 172.21.15.40:0/397309865 conn(0x7f9374075ca0 msgr2=0x7f93740760a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:16:24.415 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:24.409+0000 7f937a8ee700 1 -- 172.21.15.40:0/397309865 shutdown_connections 2023-07-14T19:16:24.415 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:24.409+0000 7f937a8ee700 1 -- 172.21.15.40:0/397309865 wait complete. 2023-07-14T19:16:24.415 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:24.409+0000 7f937a8ee700 1 Processor -- start 2023-07-14T19:16:24.416 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:24.410+0000 7f937a8ee700 1 -- start start 2023-07-14T19:16:24.416 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:24.410+0000 7f937a8ee700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9374100f20 0x7f93740feb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:16:24.416 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:24.410+0000 7f937a8ee700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9374101830 0x7f93740ff0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:16:24.417 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:24.410+0000 7f937a8ee700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f937410a270 0x7f93740ff640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:16:24.417 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:24.410+0000 7f937a8ee700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93741a4140 con 0x7f937410a270 2023-07-14T19:16:24.417 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:24.410+0000 7f937a8ee700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93741a42b0 con 0x7f9374100f20 2023-07-14T19:16:24.418 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:24.410+0000 7f937a8ee700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93741a4470 con 0x7f9374101830 2023-07-14T19:16:24.418 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:24.410+0000 7f9378e8b700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f937410a270 0x7f93740ff640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:16:24.418 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:24.410+0000 7f9378e8b700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f937410a270 0x7f93740ff640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:33168/0 (socket says 172.21.15.40:33168) 2023-07-14T19:16:24.418 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:24.410+0000 7f9378e8b700 1 -- 172.21.15.40:0/3718842464 learned_addr learned my addr 172.21.15.40:0/3718842464 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-07-14T19:16:24.418 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:24.410+0000 7f9373fff700 1 --2- 172.21.15.40:0/3718842464 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9374100f20 0x7f93740feb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:16:24.419 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:24.410+0000 7f93737fe700 1 --2- 172.21.15.40:0/3718842464 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9374101830 0x7f93740ff0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:16:24.419 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:24.410+0000 7f9378e8b700 1 -- 172.21.15.40:0/3718842464 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9374101830 msgr2=0x7f93740ff0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:16:24.419 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:24.410+0000 7f9378e8b700 1 --2- 172.21.15.40:0/3718842464 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9374101830 0x7f93740ff0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:24.419 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:24.410+0000 7f9378e8b700 1 -- 172.21.15.40:0/3718842464 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9374100f20 msgr2=0x7f93740feb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:16:24.419 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:24.410+0000 7f9378e8b700 1 --2- 172.21.15.40:0/3718842464 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9374100f20 0x7f93740feb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:24.419 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:24.410+0000 7f9378e8b700 1 -- 172.21.15.40:0/3718842464 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93680095f0 con 0x7f937410a270 2023-07-14T19:16:24.419 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:24.411+0000 7f9378e8b700 1 --2- 172.21.15.40:0/3718842464 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f937410a270 0x7f93740ff640 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f936c00cca0 tx=0x7f936c00a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:16:24.421 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:24.411+0000 7f93717fa700 1 -- 172.21.15.40:0/3718842464 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f936c00aba0 con 0x7f937410a270 2023-07-14T19:16:24.421 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:24.411+0000 7f937a8ee700 1 -- 172.21.15.40:0/3718842464 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f93741a4750 con 0x7f937410a270 2023-07-14T19:16:24.421 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:24.411+0000 7f93717fa700 1 -- 172.21.15.40:0/3718842464 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f936c00ad00 con 0x7f937410a270 2023-07-14T19:16:24.421 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:24.411+0000 7f93717fa700 1 -- 172.21.15.40:0/3718842464 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f936c0115e0 con 0x7f937410a270 2023-07-14T19:16:24.421 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:24.411+0000 7f937a8ee700 1 -- 172.21.15.40:0/3718842464 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f93741a4c50 con 0x7f937410a270 2023-07-14T19:16:24.421 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:24.412+0000 7f937a8ee700 1 -- 172.21.15.40:0/3718842464 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f937404e340 con 0x7f937410a270 2023-07-14T19:16:24.422 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:24.413+0000 7f93717fa700 1 -- 172.21.15.40:0/3718842464 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7f936c012420 con 0x7f937410a270 2023-07-14T19:16:24.422 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:24.414+0000 7f93717fa700 1 --2- 172.21.15.40:0/3718842464 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f935c0a11e0 0x7f935c0a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:16:24.422 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:24.414+0000 7f93717fa700 1 -- 172.21.15.40:0/3718842464 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f936c0cc920 con 0x7f937410a270 2023-07-14T19:16:24.422 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:24.414+0000 7f9373fff700 1 --2- 172.21.15.40:0/3718842464 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f935c0a11e0 0x7f935c0a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:16:24.422 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:24.414+0000 7f93717fa700 1 --2- 172.21.15.40:0/3718842464 >> [v2:172.21.15.40:6818/693186773,v1:172.21.15.40:6819/693186773] conn(0x7f935c0aa9b0 0x7f935c0acdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:16:24.422 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:24.414+0000 7f93717fa700 1 -- 172.21.15.40:0/3718842464 --> [v2:172.21.15.40:6818/693186773,v1:172.21.15.40:6819/693186773] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f935c0ad470 con 0x7f935c0aa9b0 2023-07-14T19:16:24.422 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:24.414+0000 7f93717fa700 1 -- 172.21.15.40:0/3718842464 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_get_version_reply(handle=1 version=81) v2 ==== 24+0+0 (secure 0 0 0) 0x7f936c0d12c0 con 0x7f937410a270 2023-07-14T19:16:24.423 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:24.414+0000 7f93737fe700 1 --2- 172.21.15.40:0/3718842464 >> [v2:172.21.15.40:6818/693186773,v1:172.21.15.40:6819/693186773] conn(0x7f935c0aa9b0 0x7f935c0acdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:16:24.423 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:24.414+0000 7f9373fff700 1 --2- 172.21.15.40:0/3718842464 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f935c0a11e0 0x7f935c0a3690 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f9364005950 tx=0x7f936400b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:16:24.423 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:24.415+0000 7f93737fe700 1 --2- 172.21.15.40:0/3718842464 >> [v2:172.21.15.40:6818/693186773,v1:172.21.15.40:6819/693186773] conn(0x7f935c0aa9b0 0x7f935c0acdc0 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:16:24.423 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:24.415+0000 7f93717fa700 1 -- 172.21.15.40:0/3718842464 <== osd.6 v2:172.21.15.40:6818/693186773 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7f935c0ad470 con 0x7f935c0aa9b0 2023-07-14T19:16:24.443 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:24.442+0000 7f937a8ee700 1 -- 172.21.15.40:0/3718842464 --> [v2:172.21.15.40:6818/693186773,v1:172.21.15.40:6819/693186773] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f937410fe50 con 0x7f935c0aa9b0 2023-07-14T19:16:24.444 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:24.442+0000 7f93717fa700 1 -- 172.21.15.40:0/3718842464 <== osd.6 v2:172.21.15.40:6818/693186773 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f936801e040 con 0x7f935c0aa9b0 2023-07-14T19:16:24.445 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:24.443+0000 7f937a8ee700 1 -- 172.21.15.40:0/3718842464 >> [v2:172.21.15.40:6818/693186773,v1:172.21.15.40:6819/693186773] conn(0x7f935c0aa9b0 msgr2=0x7f935c0acdc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:16:24.445 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:24.443+0000 7f937a8ee700 1 --2- 172.21.15.40:0/3718842464 >> [v2:172.21.15.40:6818/693186773,v1:172.21.15.40:6819/693186773] conn(0x7f935c0aa9b0 0x7f935c0acdc0 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:24.446 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:24.442+0000 7f0ca24f9700 1 -- 172.21.15.40:0/1531861852 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0c9c0fe5e0 msgr2=0x7f0c9c0fe9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:16:24.446 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:24.442+0000 7f0ca24f9700 1 --2- 172.21.15.40:0/1531861852 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0c9c0fe5e0 0x7f0c9c0fe9b0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f0c900098f0 tx=0x7f0c90009c00 comp rx=0 tx=0).stop 2023-07-14T19:16:24.446 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:24.442+0000 7f0ca24f9700 1 -- 172.21.15.40:0/1531861852 shutdown_connections 2023-07-14T19:16:24.446 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:24.442+0000 7f0ca24f9700 1 --2- 172.21.15.40:0/1531861852 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0c9c103680 0x7f0c9c10b200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:24.447 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:24.442+0000 7f0ca24f9700 1 --2- 172.21.15.40:0/1531861852 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0c9c0fe5e0 0x7f0c9c0fe9b0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:24.447 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:24.442+0000 7f0ca24f9700 1 --2- 172.21.15.40:0/1531861852 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0c9c0fef80 0x7f0c9c102f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:24.447 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:24.442+0000 7f0ca24f9700 1 -- 172.21.15.40:0/1531861852 >> 172.21.15.40:0/1531861852 conn(0x7f0c9c075280 msgr2=0x7f0c9c075680 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:16:24.447 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:24.442+0000 7f0ca24f9700 1 -- 172.21.15.40:0/1531861852 shutdown_connections 2023-07-14T19:16:24.447 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:24.442+0000 7f0ca24f9700 1 -- 172.21.15.40:0/1531861852 wait complete. 2023-07-14T19:16:24.448 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:24.443+0000 7f937a8ee700 1 -- 172.21.15.40:0/3718842464 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f935c0a11e0 msgr2=0x7f935c0a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:16:24.448 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:24.443+0000 7f937a8ee700 1 --2- 172.21.15.40:0/3718842464 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f935c0a11e0 0x7f935c0a3690 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f9364005950 tx=0x7f936400b410 comp rx=0 tx=0).stop 2023-07-14T19:16:24.449 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:24.443+0000 7f937a8ee700 1 -- 172.21.15.40:0/3718842464 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f937410a270 msgr2=0x7f93740ff640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:16:24.449 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:24.443+0000 7f937a8ee700 1 --2- 172.21.15.40:0/3718842464 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f937410a270 0x7f93740ff640 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f936c00cca0 tx=0x7f936c00a060 comp rx=0 tx=0).stop 2023-07-14T19:16:24.449 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:24.443+0000 7f9378e8b700 1 -- 172.21.15.40:0/3718842464 reap_dead start 2023-07-14T19:16:24.449 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:24.443+0000 7f937a8ee700 1 -- 172.21.15.40:0/3718842464 shutdown_connections 2023-07-14T19:16:24.449 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:24.443+0000 7f937a8ee700 1 -- 172.21.15.40:0/3718842464 >> 172.21.15.40:0/3718842464 conn(0x7f9374075ca0 msgr2=0x7f93740fd420 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:16:24.449 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:24.443+0000 7f937a8ee700 1 -- 172.21.15.40:0/3718842464 shutdown_connections 2023-07-14T19:16:24.450 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:24.443+0000 7f937a8ee700 1 -- 172.21.15.40:0/3718842464 wait complete. 2023-07-14T19:16:24.451 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:24.443+0000 7f0ca24f9700 1 Processor -- start 2023-07-14T19:16:24.451 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:24.443+0000 7f0ca24f9700 1 -- start start 2023-07-14T19:16:24.451 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:24.444+0000 7f0ca24f9700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0c9c0fe5e0 0x7f0c9c1a66c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:16:24.451 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:24.444+0000 7f0ca24f9700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0c9c0fef80 0x7f0c9c1a6c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:16:24.451 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:24.444+0000 7f0ca24f9700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0c9c103680 0x7f0c9c06cbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:16:24.452 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:24.444+0000 7f0ca24f9700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c9c06d380 con 0x7f0c9c0fef80 2023-07-14T19:16:24.452 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:24.444+0000 7f0ca24f9700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c9c06d4f0 con 0x7f0c9c0fe5e0 2023-07-14T19:16:24.452 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:24.444+0000 7f0ca24f9700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c9c06d6b0 con 0x7f0c9c103680 2023-07-14T19:16:24.452 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:24.444+0000 7f0c9bfff700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0c9c0fe5e0 0x7f0c9c1a66c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:16:24.452 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:24.444+0000 7f0c9b7fe700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0c9c0fef80 0x7f0c9c1a6c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:16:24.452 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:24.444+0000 7f0c9b7fe700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0c9c0fef80 0x7f0c9c1a6c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:33180/0 (socket says 172.21.15.40:33180) 2023-07-14T19:16:24.452 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:24.444+0000 7f0c9b7fe700 1 -- 172.21.15.40:0/3565393654 learned_addr learned my addr 172.21.15.40:0/3565393654 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-07-14T19:16:24.453 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:24.444+0000 7f0ca0a96700 1 --2- 172.21.15.40:0/3565393654 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0c9c103680 0x7f0c9c06cbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:16:24.453 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:24.444+0000 7f0c9b7fe700 1 -- 172.21.15.40:0/3565393654 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0c9c103680 msgr2=0x7f0c9c06cbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:16:24.453 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:24.444+0000 7f0c9b7fe700 1 --2- 172.21.15.40:0/3565393654 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0c9c103680 0x7f0c9c06cbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:24.453 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:24.444+0000 7f0c9b7fe700 1 -- 172.21.15.40:0/3565393654 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0c9c0fe5e0 msgr2=0x7f0c9c1a66c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:16:24.453 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:24.445+0000 7f0c9b7fe700 1 --2- 172.21.15.40:0/3565393654 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0c9c0fe5e0 0x7f0c9c1a66c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:24.453 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:24.445+0000 7f0c9b7fe700 1 -- 172.21.15.40:0/3565393654 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c90009520 con 0x7f0c9c0fef80 2023-07-14T19:16:24.453 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:24.445+0000 7f0c9b7fe700 1 --2- 172.21.15.40:0/3565393654 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0c9c0fef80 0x7f0c9c1a6c00 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f0c8c00e9f0 tx=0x7f0c8c00ed00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:16:24.453 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:24.445+0000 7f0c997fa700 1 -- 172.21.15.40:0/3565393654 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c8c009690 con 0x7f0c9c0fef80 2023-07-14T19:16:24.454 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:24.445+0000 7f0c997fa700 1 -- 172.21.15.40:0/3565393654 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f0c8c00ce80 con 0x7f0c9c0fef80 2023-07-14T19:16:24.454 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:24.445+0000 7f0ca24f9700 1 -- 172.21.15.40:0/3565393654 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0c9c06d930 con 0x7f0c9c0fef80 2023-07-14T19:16:24.454 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:24.445+0000 7f0c997fa700 1 -- 172.21.15.40:0/3565393654 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c8c018680 con 0x7f0c9c0fef80 2023-07-14T19:16:24.454 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:24.445+0000 7f0ca24f9700 1 -- 172.21.15.40:0/3565393654 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0c9c1053f0 con 0x7f0c9c0fef80 2023-07-14T19:16:24.454 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:24.446+0000 7f0ca24f9700 1 -- 172.21.15.40:0/3565393654 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f0c9c10ff00 con 0x7f0c9c0fef80 2023-07-14T19:16:24.454 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:24.448+0000 7f0c997fa700 1 -- 172.21.15.40:0/3565393654 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7f0c8c010430 con 0x7f0c9c0fef80 2023-07-14T19:16:24.454 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:24.449+0000 7f0c997fa700 1 --2- 172.21.15.40:0/3565393654 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f0c840a11e0 0x7f0c840a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:16:24.455 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:24.449+0000 7f0c9bfff700 1 --2- 172.21.15.40:0/3565393654 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f0c840a11e0 0x7f0c840a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:16:24.455 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:24.450+0000 7f0c997fa700 1 -- 172.21.15.40:0/3565393654 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f0c8c014070 con 0x7f0c9c0fef80 2023-07-14T19:16:24.456 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:24.450+0000 7f0c997fa700 1 --2- 172.21.15.40:0/3565393654 >> [v2:172.21.15.40:6826/2228380263,v1:172.21.15.40:6827/2228380263] conn(0x7f0c840aa9a0 0x7f0c840acdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:16:24.456 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:24.450+0000 7f0c997fa700 1 -- 172.21.15.40:0/3565393654 --> [v2:172.21.15.40:6826/2228380263,v1:172.21.15.40:6827/2228380263] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f0c840ad460 con 0x7f0c840aa9a0 2023-07-14T19:16:24.456 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:24.450+0000 7f0c997fa700 1 -- 172.21.15.40:0/3565393654 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_get_version_reply(handle=1 version=81) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0c8c0cd620 con 0x7f0c9c0fef80 2023-07-14T19:16:24.456 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:24.450+0000 7f0ca0a96700 1 --2- 172.21.15.40:0/3565393654 >> [v2:172.21.15.40:6826/2228380263,v1:172.21.15.40:6827/2228380263] conn(0x7f0c840aa9a0 0x7f0c840acdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:16:24.456 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:24.450+0000 7f0c9bfff700 1 --2- 172.21.15.40:0/3565393654 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f0c840a11e0 0x7f0c840a3690 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f0c90005e90 tx=0x7f0c90019040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:16:24.456 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:24.450+0000 7f0ca0a96700 1 --2- 172.21.15.40:0/3565393654 >> [v2:172.21.15.40:6826/2228380263,v1:172.21.15.40:6827/2228380263] conn(0x7f0c840aa9a0 0x7f0c840acdb0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.9 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:16:24.457 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:24.451+0000 7f0c997fa700 1 -- 172.21.15.40:0/3565393654 <== osd.9 v2:172.21.15.40:6826/2228380263 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7f0c840ad460 con 0x7f0c840aa9a0 2023-07-14T19:16:24.475 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:24.473+0000 7f0ca24f9700 1 -- 172.21.15.40:0/3565393654 --> [v2:172.21.15.40:6826/2228380263,v1:172.21.15.40:6827/2228380263] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f0c9c04e340 con 0x7f0c840aa9a0 2023-07-14T19:16:24.480 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:24.478+0000 7f0c997fa700 1 -- 172.21.15.40:0/3565393654 <== osd.9 v2:172.21.15.40:6826/2228380263 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f0c9c04e340 con 0x7f0c840aa9a0 2023-07-14T19:16:24.481 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:24.478+0000 7f0ca24f9700 1 -- 172.21.15.40:0/3565393654 >> [v2:172.21.15.40:6826/2228380263,v1:172.21.15.40:6827/2228380263] conn(0x7f0c840aa9a0 msgr2=0x7f0c840acdb0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:16:24.481 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:24.478+0000 7f0ca24f9700 1 --2- 172.21.15.40:0/3565393654 >> [v2:172.21.15.40:6826/2228380263,v1:172.21.15.40:6827/2228380263] conn(0x7f0c840aa9a0 0x7f0c840acdb0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:24.481 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:24.478+0000 7f0ca24f9700 1 -- 172.21.15.40:0/3565393654 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f0c840a11e0 msgr2=0x7f0c840a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:16:24.481 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:24.478+0000 7f0ca24f9700 1 --2- 172.21.15.40:0/3565393654 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f0c840a11e0 0x7f0c840a3690 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f0c90005e90 tx=0x7f0c90019040 comp rx=0 tx=0).stop 2023-07-14T19:16:24.481 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:24.478+0000 7f0ca24f9700 1 -- 172.21.15.40:0/3565393654 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0c9c0fef80 msgr2=0x7f0c9c1a6c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:16:24.481 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:24.478+0000 7f0ca24f9700 1 --2- 172.21.15.40:0/3565393654 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0c9c0fef80 0x7f0c9c1a6c00 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f0c8c00e9f0 tx=0x7f0c8c00ed00 comp rx=0 tx=0).stop 2023-07-14T19:16:24.482 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:24.478+0000 7f0ca0a96700 1 -- 172.21.15.40:0/3565393654 reap_dead start 2023-07-14T19:16:24.482 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:24.479+0000 7f0ca24f9700 1 -- 172.21.15.40:0/3565393654 shutdown_connections 2023-07-14T19:16:24.482 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:24.479+0000 7f0ca24f9700 1 -- 172.21.15.40:0/3565393654 >> 172.21.15.40:0/3565393654 conn(0x7f0c9c075280 msgr2=0x7f0c9c10a710 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:16:24.482 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:24.479+0000 7f0ca24f9700 1 -- 172.21.15.40:0/3565393654 shutdown_connections 2023-07-14T19:16:24.482 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:24.479+0000 7f0ca24f9700 1 -- 172.21.15.40:0/3565393654 wait complete. 2023-07-14T19:16:24.669 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:24.582+0000 7fa151511700 1 -- 172.21.15.40:0/2350938021 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa14c0fee40 msgr2=0x7fa14c0ff210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:16:24.669 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:24.582+0000 7fa151511700 1 --2- 172.21.15.40:0/2350938021 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa14c0fee40 0x7fa14c0ff210 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fa13c009920 tx=0x7fa13c009c30 comp rx=0 tx=0).stop 2023-07-14T19:16:24.670 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:24.583+0000 7fa151511700 1 -- 172.21.15.40:0/2350938021 shutdown_connections 2023-07-14T19:16:24.670 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:24.583+0000 7fa151511700 1 --2- 172.21.15.40:0/2350938021 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa14c103ee0 0x7fa14c10ba60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:24.670 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:24.583+0000 7fa151511700 1 --2- 172.21.15.40:0/2350938021 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa14c0fee40 0x7fa14c0ff210 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:24.670 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:24.583+0000 7fa151511700 1 --2- 172.21.15.40:0/2350938021 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa14c0ff7e0 0x7fa14c1037b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:24.670 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:24.583+0000 7fa151511700 1 -- 172.21.15.40:0/2350938021 >> 172.21.15.40:0/2350938021 conn(0x7fa14c075ca0 msgr2=0x7fa14c0760a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:16:24.670 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:24.583+0000 7fa151511700 1 -- 172.21.15.40:0/2350938021 shutdown_connections 2023-07-14T19:16:24.671 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:24.583+0000 7fa151511700 1 -- 172.21.15.40:0/2350938021 wait complete. 2023-07-14T19:16:24.671 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:24.583+0000 7fa151511700 1 Processor -- start 2023-07-14T19:16:24.671 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:24.584+0000 7fa151511700 1 -- start start 2023-07-14T19:16:24.671 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:24.584+0000 7fa151511700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa14c0fee40 0x7fa14c06bec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:16:24.671 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:24.584+0000 7fa151511700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa14c0ff7e0 0x7fa14c06c400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:16:24.671 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:24.584+0000 7fa151511700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa14c103ee0 0x7fa14c071830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:16:24.671 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:24.584+0000 7fa151511700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa14c071f30 con 0x7fa14c0fee40 2023-07-14T19:16:24.671 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:24.584+0000 7fa151511700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa14c0720a0 con 0x7fa14c103ee0 2023-07-14T19:16:24.672 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:24.584+0000 7fa151511700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa14c072260 con 0x7fa14c0ff7e0 2023-07-14T19:16:24.672 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:24.584+0000 7fa14affd700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa14c0fee40 0x7fa14c06bec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:16:24.672 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:24.584+0000 7fa14affd700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa14c0fee40 0x7fa14c06bec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:33194/0 (socket says 172.21.15.40:33194) 2023-07-14T19:16:24.672 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:24.584+0000 7fa14affd700 1 -- 172.21.15.40:0/3331698109 learned_addr learned my addr 172.21.15.40:0/3331698109 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-07-14T19:16:24.672 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:24.584+0000 7fa14a7fc700 1 --2- 172.21.15.40:0/3331698109 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa14c0ff7e0 0x7fa14c06c400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:16:24.672 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:24.584+0000 7fa14affd700 1 -- 172.21.15.40:0/3331698109 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa14c0ff7e0 msgr2=0x7fa14c06c400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:16:24.672 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:24.584+0000 7fa14affd700 1 --2- 172.21.15.40:0/3331698109 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa14c0ff7e0 0x7fa14c06c400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:24.673 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:24.584+0000 7fa14affd700 1 -- 172.21.15.40:0/3331698109 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa14c103ee0 msgr2=0x7fa14c071830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:16:24.673 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:24.585+0000 7fa14affd700 1 --2- 172.21.15.40:0/3331698109 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa14c103ee0 0x7fa14c071830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:24.673 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:24.585+0000 7fa14affd700 1 -- 172.21.15.40:0/3331698109 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa140009520 con 0x7fa14c0fee40 2023-07-14T19:16:24.673 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:24.585+0000 7fa14a7fc700 1 --2- 172.21.15.40:0/3331698109 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa14c0ff7e0 0x7fa14c06c400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-14T19:16:24.673 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:24.585+0000 7fa14affd700 1 --2- 172.21.15.40:0/3331698109 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa14c0fee40 0x7fa14c06bec0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fa13c009230 tx=0x7fa13c00bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:16:24.673 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:24.585+0000 7fa133fff700 1 -- 172.21.15.40:0/3331698109 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa13c01c070 con 0x7fa14c0fee40 2023-07-14T19:16:24.673 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:24.585+0000 7fa151511700 1 -- 172.21.15.40:0/3331698109 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa144009d00 con 0x7fa14c0fee40 2023-07-14T19:16:24.674 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:24.585+0000 7fa133fff700 1 -- 172.21.15.40:0/3331698109 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa13c00bc00 con 0x7fa14c0fee40 2023-07-14T19:16:24.674 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:24.585+0000 7fa133fff700 1 -- 172.21.15.40:0/3331698109 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa13c004a30 con 0x7fa14c0fee40 2023-07-14T19:16:24.674 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:24.585+0000 7fa151511700 1 -- 172.21.15.40:0/3331698109 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa13c009520 con 0x7fa14c0fee40 2023-07-14T19:16:24.674 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:24.586+0000 7fa151511700 1 -- 172.21.15.40:0/3331698109 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fa14c10ed20 con 0x7fa14c0fee40 2023-07-14T19:16:24.674 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:24.587+0000 7fa133fff700 1 -- 172.21.15.40:0/3331698109 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7fa13c020070 con 0x7fa14c0fee40 2023-07-14T19:16:24.674 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:24.588+0000 7fa133fff700 1 --2- 172.21.15.40:0/3331698109 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7fa1340a11e0 0x7fa1340a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:16:24.674 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:24.588+0000 7fa14a7fc700 1 --2- 172.21.15.40:0/3331698109 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7fa1340a11e0 0x7fa1340a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:16:24.674 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:24.588+0000 7fa133fff700 1 -- 172.21.15.40:0/3331698109 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7979+0+0 (secure 0 0 0) 0x7fa13c0cdcc0 con 0x7fa14c0fee40 2023-07-14T19:16:24.675 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:24.588+0000 7fa133fff700 1 --2- 172.21.15.40:0/3331698109 >> [v2:172.21.15.191:6800/589569135,v1:172.21.15.191:6801/589569135] conn(0x7fa1340aa9b0 0x7fa1340acdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:16:24.675 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:24.588+0000 7fa133fff700 1 -- 172.21.15.40:0/3331698109 --> [v2:172.21.15.191:6800/589569135,v1:172.21.15.191:6801/589569135] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa1340ad470 con 0x7fa1340aa9b0 2023-07-14T19:16:24.675 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:24.588+0000 7fa133fff700 1 -- 172.21.15.40:0/3331698109 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_get_version_reply(handle=1 version=81) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa13c0ce160 con 0x7fa14c0fee40 2023-07-14T19:16:24.675 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:24.588+0000 7fa14a7fc700 1 --2- 172.21.15.40:0/3331698109 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7fa1340a11e0 0x7fa1340a3690 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fa14c06d370 tx=0x7fa140009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:16:24.676 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:24.588+0000 7fa14b7fe700 1 --2- 172.21.15.40:0/3331698109 >> [v2:172.21.15.191:6800/589569135,v1:172.21.15.191:6801/589569135] conn(0x7fa1340aa9b0 0x7fa1340acdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:16:24.676 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:24.590+0000 7fa14b7fe700 1 --2- 172.21.15.40:0/3331698109 >> [v2:172.21.15.191:6800/589569135,v1:172.21.15.191:6801/589569135] conn(0x7fa1340aa9b0 0x7fa1340acdc0 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:16:24.676 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:24.591+0000 7fa133fff700 1 -- 172.21.15.40:0/3331698109 <== osd.2 v2:172.21.15.191:6800/589569135 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7fa1340ad470 con 0x7fa1340aa9b0 2023-07-14T19:16:24.676 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:24.607+0000 7fa151511700 1 -- 172.21.15.40:0/3331698109 --> [v2:172.21.15.191:6800/589569135,v1:172.21.15.191:6801/589569135] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fa14c04e340 con 0x7fa1340aa9b0 2023-07-14T19:16:24.676 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:24.608+0000 7fa133fff700 1 -- 172.21.15.40:0/3331698109 <== osd.2 v2:172.21.15.191:6800/589569135 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fa14c04e340 con 0x7fa1340aa9b0 2023-07-14T19:16:24.676 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:24.608+0000 7fa151511700 1 -- 172.21.15.40:0/3331698109 >> [v2:172.21.15.191:6800/589569135,v1:172.21.15.191:6801/589569135] conn(0x7fa1340aa9b0 msgr2=0x7fa1340acdc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:16:24.676 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:24.608+0000 7fa151511700 1 --2- 172.21.15.40:0/3331698109 >> [v2:172.21.15.191:6800/589569135,v1:172.21.15.191:6801/589569135] conn(0x7fa1340aa9b0 0x7fa1340acdc0 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:24.677 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:24.608+0000 7fa151511700 1 -- 172.21.15.40:0/3331698109 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7fa1340a11e0 msgr2=0x7fa1340a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:16:24.678 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:24.608+0000 7fa151511700 1 --2- 172.21.15.40:0/3331698109 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7fa1340a11e0 0x7fa1340a3690 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fa14c06d370 tx=0x7fa140009260 comp rx=0 tx=0).stop 2023-07-14T19:16:24.678 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:24.608+0000 7fa151511700 1 -- 172.21.15.40:0/3331698109 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa14c0fee40 msgr2=0x7fa14c06bec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:16:24.678 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:24.608+0000 7fa151511700 1 --2- 172.21.15.40:0/3331698109 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa14c0fee40 0x7fa14c06bec0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fa13c009230 tx=0x7fa13c00bf60 comp rx=0 tx=0).stop 2023-07-14T19:16:24.678 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:24.608+0000 7fa14b7fe700 1 -- 172.21.15.40:0/3331698109 reap_dead start 2023-07-14T19:16:24.678 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:24.608+0000 7fa151511700 1 -- 172.21.15.40:0/3331698109 shutdown_connections 2023-07-14T19:16:24.678 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:24.608+0000 7fa151511700 1 -- 172.21.15.40:0/3331698109 >> 172.21.15.40:0/3331698109 conn(0x7fa14c075ca0 msgr2=0x7fa14c102790 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:16:24.678 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:24.608+0000 7fa151511700 1 -- 172.21.15.40:0/3331698109 shutdown_connections 2023-07-14T19:16:24.679 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:24.609+0000 7fa151511700 1 -- 172.21.15.40:0/3331698109 wait complete. 2023-07-14T19:16:24.742 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:16:24 smithi130 ceph-mon[97832]: pgmap v267: 97 pgs: 97 active+clean; 637 KiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 1.5 KiB/s wr, 9 op/s 2023-07-14T19:16:24.793 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:16:24 smithi040 ceph-mon[96518]: pgmap v267: 97 pgs: 97 active+clean; 637 KiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 1.5 KiB/s wr, 9 op/s 2023-07-14T19:16:24.972 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:16:24 smithi191 ceph-mon[98311]: pgmap v267: 97 pgs: 97 active+clean; 637 KiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 1.5 KiB/s wr, 9 op/s 2023-07-14T19:16:25.388 INFO:teuthology.orchestra.run.smithi040.stdout:34359738432 2023-07-14T19:16:25.389 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell --fsid adc52f04-2279-11ee-9b33-001a4aab830c -- ceph osd last-stat-seq osd.0 2023-07-14T19:16:25.391 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:25.115+0000 7f827d3cf700 1 -- 172.21.15.40:0/3502279148 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f82780fee40 msgr2=0x7f82780ff210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:16:25.392 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:25.115+0000 7f827d3cf700 1 --2- 172.21.15.40:0/3502279148 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f82780fee40 0x7f82780ff210 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f826c0099c0 tx=0x7f826c009cd0 comp rx=0 tx=0).stop 2023-07-14T19:16:25.392 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:25.115+0000 7f827d3cf700 1 -- 172.21.15.40:0/3502279148 shutdown_connections 2023-07-14T19:16:25.392 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:25.115+0000 7f827d3cf700 1 --2- 172.21.15.40:0/3502279148 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8278103ee0 0x7f827810ba60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:25.392 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:25.115+0000 7f827d3cf700 1 --2- 172.21.15.40:0/3502279148 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f82780fee40 0x7f82780ff210 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:25.392 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:25.115+0000 7f827d3cf700 1 --2- 172.21.15.40:0/3502279148 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f82780ff7e0 0x7f82781037b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:25.393 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:25.115+0000 7f827d3cf700 1 -- 172.21.15.40:0/3502279148 >> 172.21.15.40:0/3502279148 conn(0x7f8278075ca0 msgr2=0x7f82780760a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:16:25.393 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:25.115+0000 7f827d3cf700 1 -- 172.21.15.40:0/3502279148 shutdown_connections 2023-07-14T19:16:25.393 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:25.115+0000 7f827d3cf700 1 -- 172.21.15.40:0/3502279148 wait complete. 2023-07-14T19:16:25.393 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:25.116+0000 7f827d3cf700 1 Processor -- start 2023-07-14T19:16:25.393 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:25.116+0000 7f827d3cf700 1 -- start start 2023-07-14T19:16:25.393 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:25.116+0000 7f827d3cf700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f82780fee40 0x7f8278196c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:16:25.393 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:25.116+0000 7f827d3cf700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f82780ff7e0 0x7f8278197140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:16:25.394 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:25.116+0000 7f827d3cf700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8278103ee0 0x7f82781a1c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:16:25.394 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:25.116+0000 7f827d3cf700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8278197a50 con 0x7f8278103ee0 2023-07-14T19:16:25.394 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:25.116+0000 7f827d3cf700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82781a21c0 con 0x7f82780fee40 2023-07-14T19:16:25.394 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:25.116+0000 7f827d3cf700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82781a2330 con 0x7f82780ff7e0 2023-07-14T19:16:25.394 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:25.116+0000 7f82777fe700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8278103ee0 0x7f82781a1c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:16:25.394 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:25.116+0000 7f82777fe700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8278103ee0 0x7f82781a1c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:33214/0 (socket says 172.21.15.40:33214) 2023-07-14T19:16:25.395 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:25.116+0000 7f82777fe700 1 -- 172.21.15.40:0/1477443443 learned_addr learned my addr 172.21.15.40:0/1477443443 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-07-14T19:16:25.395 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:25.116+0000 7f82767fc700 1 --2- 172.21.15.40:0/1477443443 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f82780ff7e0 0x7f8278197140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:16:25.395 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:25.117+0000 7f8276ffd700 1 --2- 172.21.15.40:0/1477443443 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f82780fee40 0x7f8278196c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:16:25.395 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:25.117+0000 7f82777fe700 1 -- 172.21.15.40:0/1477443443 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f82780ff7e0 msgr2=0x7f8278197140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:16:25.395 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:25.117+0000 7f82777fe700 1 --2- 172.21.15.40:0/1477443443 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f82780ff7e0 0x7f8278197140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:25.395 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:25.117+0000 7f82777fe700 1 -- 172.21.15.40:0/1477443443 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f82780fee40 msgr2=0x7f8278196c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:16:25.395 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:25.117+0000 7f82777fe700 1 --2- 172.21.15.40:0/1477443443 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f82780fee40 0x7f8278196c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:25.395 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:25.117+0000 7f82777fe700 1 -- 172.21.15.40:0/1477443443 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f826c0095f0 con 0x7f8278103ee0 2023-07-14T19:16:25.396 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:25.117+0000 7f82777fe700 1 --2- 172.21.15.40:0/1477443443 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8278103ee0 0x7f82781a1c80 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f8270007f10 tx=0x7f827000d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:16:25.396 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:25.117+0000 7f825ffff700 1 -- 172.21.15.40:0/1477443443 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8270017070 con 0x7f8278103ee0 2023-07-14T19:16:25.396 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:25.117+0000 7f827d3cf700 1 -- 172.21.15.40:0/1477443443 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f82781a2610 con 0x7f8278103ee0 2023-07-14T19:16:25.396 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:25.117+0000 7f825ffff700 1 -- 172.21.15.40:0/1477443443 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f827000f040 con 0x7f8278103ee0 2023-07-14T19:16:25.396 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:25.117+0000 7f825ffff700 1 -- 172.21.15.40:0/1477443443 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8270005740 con 0x7f8278103ee0 2023-07-14T19:16:25.396 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:25.117+0000 7f827d3cf700 1 -- 172.21.15.40:0/1477443443 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f82781a2ac0 con 0x7f8278103ee0 2023-07-14T19:16:25.396 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:25.118+0000 7f827d3cf700 1 -- 172.21.15.40:0/1477443443 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f827810eda0 con 0x7f8278103ee0 2023-07-14T19:16:25.397 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:25.119+0000 7f825ffff700 1 -- 172.21.15.40:0/1477443443 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7f827000dc60 con 0x7f8278103ee0 2023-07-14T19:16:25.397 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:25.120+0000 7f825ffff700 1 --2- 172.21.15.40:0/1477443443 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f82600a11e0 0x7f82600a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:16:25.397 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:25.120+0000 7f8276ffd700 1 --2- 172.21.15.40:0/1477443443 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f82600a11e0 0x7f82600a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:16:25.397 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:25.120+0000 7f825ffff700 1 -- 172.21.15.40:0/1477443443 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f82700cc6b0 con 0x7f8278103ee0 2023-07-14T19:16:25.397 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:25.120+0000 7f825ffff700 1 --2- 172.21.15.40:0/1477443443 >> [v2:172.21.15.191:6816/2386818417,v1:172.21.15.191:6817/2386818417] conn(0x7f82600aa9b0 0x7f82600acdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:16:25.397 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:25.120+0000 7f825ffff700 1 -- 172.21.15.40:0/1477443443 --> [v2:172.21.15.191:6816/2386818417,v1:172.21.15.191:6817/2386818417] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f82600ad470 con 0x7f82600aa9b0 2023-07-14T19:16:25.397 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:25.120+0000 7f825ffff700 1 -- 172.21.15.40:0/1477443443 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_get_version_reply(handle=1 version=81) v2 ==== 24+0+0 (secure 0 0 0) 0x7f82700ccb50 con 0x7f8278103ee0 2023-07-14T19:16:25.398 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:25.120+0000 7f8276ffd700 1 --2- 172.21.15.40:0/1477443443 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f82600a11e0 0x7f82600a3690 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f826c000c00 tx=0x7f826c005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:16:25.398 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:25.120+0000 7f82767fc700 1 --2- 172.21.15.40:0/1477443443 >> [v2:172.21.15.191:6816/2386818417,v1:172.21.15.191:6817/2386818417] conn(0x7f82600aa9b0 0x7f82600acdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:16:25.398 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:25.121+0000 7f82767fc700 1 --2- 172.21.15.40:0/1477443443 >> [v2:172.21.15.191:6816/2386818417,v1:172.21.15.191:6817/2386818417] conn(0x7f82600aa9b0 0x7f82600acdc0 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:16:25.398 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:25.121+0000 7f825ffff700 1 -- 172.21.15.40:0/1477443443 <== osd.8 v2:172.21.15.191:6816/2386818417 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7f82600ad470 con 0x7f82600aa9b0 2023-07-14T19:16:25.399 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:25.137+0000 7f827d3cf700 1 -- 172.21.15.40:0/1477443443 --> [v2:172.21.15.191:6816/2386818417,v1:172.21.15.191:6817/2386818417] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f827804e340 con 0x7f82600aa9b0 2023-07-14T19:16:25.399 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:25.138+0000 7f825ffff700 1 -- 172.21.15.40:0/1477443443 <== osd.8 v2:172.21.15.191:6816/2386818417 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f827804e340 con 0x7f82600aa9b0 2023-07-14T19:16:25.399 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:25.138+0000 7f827d3cf700 1 -- 172.21.15.40:0/1477443443 >> [v2:172.21.15.191:6816/2386818417,v1:172.21.15.191:6817/2386818417] conn(0x7f82600aa9b0 msgr2=0x7f82600acdc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:16:25.399 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:25.138+0000 7f827d3cf700 1 --2- 172.21.15.40:0/1477443443 >> [v2:172.21.15.191:6816/2386818417,v1:172.21.15.191:6817/2386818417] conn(0x7f82600aa9b0 0x7f82600acdc0 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:25.400 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:25.138+0000 7f827d3cf700 1 -- 172.21.15.40:0/1477443443 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f82600a11e0 msgr2=0x7f82600a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:16:25.400 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:25.138+0000 7f827d3cf700 1 --2- 172.21.15.40:0/1477443443 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f82600a11e0 0x7f82600a3690 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f826c000c00 tx=0x7f826c005fb0 comp rx=0 tx=0).stop 2023-07-14T19:16:25.400 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:25.138+0000 7f827d3cf700 1 -- 172.21.15.40:0/1477443443 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8278103ee0 msgr2=0x7f82781a1c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:16:25.400 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:25.138+0000 7f827d3cf700 1 --2- 172.21.15.40:0/1477443443 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8278103ee0 0x7f82781a1c80 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f8270007f10 tx=0x7f827000d3b0 comp rx=0 tx=0).stop 2023-07-14T19:16:25.400 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:25.138+0000 7f82777fe700 1 -- 172.21.15.40:0/1477443443 reap_dead start 2023-07-14T19:16:25.400 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:25.139+0000 7f827d3cf700 1 -- 172.21.15.40:0/1477443443 shutdown_connections 2023-07-14T19:16:25.401 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:25.139+0000 7f827d3cf700 1 -- 172.21.15.40:0/1477443443 >> 172.21.15.40:0/1477443443 conn(0x7f8278075ca0 msgr2=0x7f82780fc4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:16:25.401 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:25.139+0000 7f827d3cf700 1 -- 172.21.15.40:0/1477443443 shutdown_connections 2023-07-14T19:16:25.401 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:25.139+0000 7f827d3cf700 1 -- 172.21.15.40:0/1477443443 wait complete. 2023-07-14T19:16:26.198 INFO:teuthology.orchestra.run.smithi040.stdout:274877906969 2023-07-14T19:16:26.198 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell --fsid adc52f04-2279-11ee-9b33-001a4aab830c -- ceph osd last-stat-seq osd.9 2023-07-14T19:16:26.659 INFO:teuthology.orchestra.run.smithi040.stdout:193273528358 2023-07-14T19:16:26.659 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell --fsid adc52f04-2279-11ee-9b33-001a4aab830c -- ceph osd last-stat-seq osd.6 2023-07-14T19:16:26.760 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:26.758+0000 7f41e603a700 1 -- 172.21.15.40:0/2436580052 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f41e0100e00 msgr2=0x7f41e01011d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:16:26.761 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:26.758+0000 7f41e603a700 1 --2- 172.21.15.40:0/2436580052 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f41e0100e00 0x7f41e01011d0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f41d0009990 tx=0x7f41d0009ca0 comp rx=0 tx=0).stop 2023-07-14T19:16:26.761 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:26.759+0000 7f41e603a700 1 -- 172.21.15.40:0/2436580052 shutdown_connections 2023-07-14T19:16:26.761 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:26.759+0000 7f41e603a700 1 --2- 172.21.15.40:0/2436580052 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f41e010a320 0x7f41e010c740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:26.761 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:26.759+0000 7f41e603a700 1 --2- 172.21.15.40:0/2436580052 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f41e0100e00 0x7f41e01011d0 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:26.761 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:26.759+0000 7f41e603a700 1 --2- 172.21.15.40:0/2436580052 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f41e0101710 0x7f41e0109de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:26.761 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:26.759+0000 7f41e603a700 1 -- 172.21.15.40:0/2436580052 >> 172.21.15.40:0/2436580052 conn(0x7f41e0075ab0 msgr2=0x7f41e0075eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:16:26.762 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:26.759+0000 7f41e603a700 1 -- 172.21.15.40:0/2436580052 shutdown_connections 2023-07-14T19:16:26.762 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:26.759+0000 7f41e603a700 1 -- 172.21.15.40:0/2436580052 wait complete. 2023-07-14T19:16:26.763 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:26.759+0000 7f41e603a700 1 Processor -- start 2023-07-14T19:16:26.763 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:26.760+0000 7f41e603a700 1 -- start start 2023-07-14T19:16:26.763 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:26.760+0000 7f41e603a700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f41e0100e00 0x7f41e00fec30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:16:26.763 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:26.760+0000 7f41e603a700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f41e0101710 0x7f41e00ff170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:16:26.763 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:26.760+0000 7f41e603a700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f41e010a320 0x7f41e00ff6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:16:26.763 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:26.760+0000 7f41e603a700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41e01a8780 con 0x7f41e010a320 2023-07-14T19:16:26.764 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:26.760+0000 7f41e603a700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41e0199f40 con 0x7f41e0101710 2023-07-14T19:16:26.764 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:26.760+0000 7f41e603a700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41e019a100 con 0x7f41e0100e00 2023-07-14T19:16:26.764 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:26.760+0000 7f41df7fe700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f41e0100e00 0x7f41e00fec30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:16:26.764 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:26.760+0000 7f41deffd700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f41e0101710 0x7f41e00ff170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:16:26.764 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:26.760+0000 7f41deffd700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f41e0101710 0x7f41e00ff170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.40:48384/0 (socket says 172.21.15.40:48384) 2023-07-14T19:16:26.764 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:26.760+0000 7f41deffd700 1 -- 172.21.15.40:0/2087579920 learned_addr learned my addr 172.21.15.40:0/2087579920 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-07-14T19:16:26.764 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:26.760+0000 7f41df7fe700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f41e0100e00 0x7f41e00fec30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.40:49832/0 (socket says 172.21.15.40:49832) 2023-07-14T19:16:26.765 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:26.761+0000 7f41dffff700 1 --2- 172.21.15.40:0/2087579920 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f41e010a320 0x7f41e00ff6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:16:26.765 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:26.761+0000 7f41df7fe700 1 -- 172.21.15.40:0/2087579920 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f41e0101710 msgr2=0x7f41e00ff170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:16:26.765 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:26.761+0000 7f41df7fe700 1 --2- 172.21.15.40:0/2087579920 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f41e0101710 0x7f41e00ff170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:26.765 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:26.761+0000 7f41df7fe700 1 -- 172.21.15.40:0/2087579920 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f41e010a320 msgr2=0x7f41e00ff6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:16:26.765 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:26.761+0000 7f41df7fe700 1 --2- 172.21.15.40:0/2087579920 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f41e010a320 0x7f41e00ff6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:26.765 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:26.761+0000 7f41df7fe700 1 -- 172.21.15.40:0/2087579920 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41d00095f0 con 0x7f41e0100e00 2023-07-14T19:16:26.765 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:26.761+0000 7f41df7fe700 1 --2- 172.21.15.40:0/2087579920 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f41e0100e00 0x7f41e00fec30 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f41d00059d0 tx=0x7f41d000f690 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:16:26.766 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:26.761+0000 7f41dcff9700 1 -- 172.21.15.40:0/2087579920 <== mon.1 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41d001d070 con 0x7f41e0100e00 2023-07-14T19:16:26.766 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:26.761+0000 7f41dcff9700 1 -- 172.21.15.40:0/2087579920 <== mon.1 v2:172.21.15.191:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f41d0005510 con 0x7f41e0100e00 2023-07-14T19:16:26.766 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:26.761+0000 7f41e603a700 1 -- 172.21.15.40:0/2087579920 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f41e019a380 con 0x7f41e0100e00 2023-07-14T19:16:26.767 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:26.761+0000 7f41dcff9700 1 -- 172.21.15.40:0/2087579920 <== mon.1 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41d0004ab0 con 0x7f41e0100e00 2023-07-14T19:16:26.767 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:26.762+0000 7f41e603a700 1 -- 172.21.15.40:0/2087579920 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f41e019a840 con 0x7f41e0100e00 2023-07-14T19:16:26.767 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:26.762+0000 7f41e603a700 1 -- 172.21.15.40:0/2087579920 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f41e010fec0 con 0x7f41e0100e00 2023-07-14T19:16:26.767 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:26.764+0000 7f41dcff9700 1 -- 172.21.15.40:0/2087579920 <== mon.1 v2:172.21.15.191:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7f41d000fb50 con 0x7f41e0100e00 2023-07-14T19:16:26.768 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:26.765+0000 7f41dcff9700 1 --2- 172.21.15.40:0/2087579920 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f41c80a1260 0x7f41c80a3710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:16:26.768 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:26.765+0000 7f41dcff9700 1 -- 172.21.15.40:0/2087579920 <== mon.1 v2:172.21.15.191:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f41d00cde70 con 0x7f41e0100e00 2023-07-14T19:16:26.768 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:26.765+0000 7f41dcff9700 1 --2- 172.21.15.40:0/2087579920 >> [v2:172.21.15.191:6824/2886059446,v1:172.21.15.191:6825/2886059446] conn(0x7f41c80aaa30 0x7f41c80ace40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:16:26.768 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:26.765+0000 7f41dcff9700 1 -- 172.21.15.40:0/2087579920 --> [v2:172.21.15.191:6824/2886059446,v1:172.21.15.191:6825/2886059446] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f41c80ad4f0 con 0x7f41c80aaa30 2023-07-14T19:16:26.769 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:26.765+0000 7f41dcff9700 1 -- 172.21.15.40:0/2087579920 <== mon.1 v2:172.21.15.191:3300/0 6 ==== mon_get_version_reply(handle=1 version=81) v2 ==== 24+0+0 (secure 0 0 0) 0x7f41d00ce310 con 0x7f41e0100e00 2023-07-14T19:16:26.769 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:26.766+0000 7f41deffd700 1 --2- 172.21.15.40:0/2087579920 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f41c80a1260 0x7f41c80a3710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:16:26.769 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:26.766+0000 7f41dffff700 1 --2- 172.21.15.40:0/2087579920 >> [v2:172.21.15.191:6824/2886059446,v1:172.21.15.191:6825/2886059446] conn(0x7f41c80aaa30 0x7f41c80ace40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:16:26.770 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:26.767+0000 7f41deffd700 1 --2- 172.21.15.40:0/2087579920 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f41c80a1260 0x7f41c80a3710 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f41e0100420 tx=0x7f41d4004080 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:16:26.770 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:26.767+0000 7f41dffff700 1 --2- 172.21.15.40:0/2087579920 >> [v2:172.21.15.191:6824/2886059446,v1:172.21.15.191:6825/2886059446] conn(0x7f41c80aaa30 0x7f41c80ace40 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.11 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:16:26.770 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:26.768+0000 7f41dcff9700 1 -- 172.21.15.40:0/2087579920 <== osd.11 v2:172.21.15.191:6824/2886059446 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7f41c80ad4f0 con 0x7f41c80aaa30 2023-07-14T19:16:26.786 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:26.784+0000 7f41e603a700 1 -- 172.21.15.40:0/2087579920 --> [v2:172.21.15.191:6824/2886059446,v1:172.21.15.191:6825/2886059446] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f41e0068230 con 0x7f41c80aaa30 2023-07-14T19:16:26.787 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:26.785+0000 7f41dcff9700 1 -- 172.21.15.40:0/2087579920 <== osd.11 v2:172.21.15.191:6824/2886059446 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f41e0068230 con 0x7f41c80aaa30 2023-07-14T19:16:26.788 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:26.785+0000 7f41e603a700 1 -- 172.21.15.40:0/2087579920 >> [v2:172.21.15.191:6824/2886059446,v1:172.21.15.191:6825/2886059446] conn(0x7f41c80aaa30 msgr2=0x7f41c80ace40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:16:26.788 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:26.785+0000 7f41e603a700 1 --2- 172.21.15.40:0/2087579920 >> [v2:172.21.15.191:6824/2886059446,v1:172.21.15.191:6825/2886059446] conn(0x7f41c80aaa30 0x7f41c80ace40 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:26.789 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:26.785+0000 7f41e603a700 1 -- 172.21.15.40:0/2087579920 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f41c80a1260 msgr2=0x7f41c80a3710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:16:26.789 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:26.785+0000 7f41e603a700 1 --2- 172.21.15.40:0/2087579920 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f41c80a1260 0x7f41c80a3710 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f41e0100420 tx=0x7f41d4004080 comp rx=0 tx=0).stop 2023-07-14T19:16:26.789 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:26.785+0000 7f41e603a700 1 -- 172.21.15.40:0/2087579920 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f41e0100e00 msgr2=0x7f41e00fec30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:16:26.789 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:26.785+0000 7f41e603a700 1 --2- 172.21.15.40:0/2087579920 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f41e0100e00 0x7f41e00fec30 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f41d00059d0 tx=0x7f41d000f690 comp rx=0 tx=0).stop 2023-07-14T19:16:26.790 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:26.785+0000 7f41dffff700 1 -- 172.21.15.40:0/2087579920 reap_dead start 2023-07-14T19:16:26.790 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:26.785+0000 7f41e603a700 1 -- 172.21.15.40:0/2087579920 shutdown_connections 2023-07-14T19:16:26.790 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:26.785+0000 7f41e603a700 1 -- 172.21.15.40:0/2087579920 >> 172.21.15.40:0/2087579920 conn(0x7f41e0075ab0 msgr2=0x7f41e0101d80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:16:26.790 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:26.785+0000 7f41e603a700 1 -- 172.21.15.40:0/2087579920 shutdown_connections 2023-07-14T19:16:26.791 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:26.786+0000 7f41e603a700 1 -- 172.21.15.40:0/2087579920 wait complete. 2023-07-14T19:16:26.808 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:16:26 smithi040 ceph-mon[96518]: pgmap v268: 97 pgs: 97 active+clean; 637 KiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 1.5 KiB/s wr, 10 op/s 2023-07-14T19:16:26.972 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:16:26 smithi191 ceph-mon[98311]: pgmap v268: 97 pgs: 97 active+clean; 637 KiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 1.5 KiB/s wr, 10 op/s 2023-07-14T19:16:26.992 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:16:26 smithi130 ceph-mon[97832]: pgmap v268: 97 pgs: 97 active+clean; 637 KiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 1.5 KiB/s wr, 10 op/s 2023-07-14T19:16:27.164 INFO:teuthology.orchestra.run.smithi040.stdout:77309411384 2023-07-14T19:16:27.165 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell --fsid adc52f04-2279-11ee-9b33-001a4aab830c -- ceph osd last-stat-seq osd.2 2023-07-14T19:16:27.235 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:27.233+0000 7f9ea5d06700 1 -- 172.21.15.40:0/2023032765 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9ea00fe210 msgr2=0x7f9ea00fe680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:16:27.235 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:27.233+0000 7f9ea5d06700 1 --2- 172.21.15.40:0/2023032765 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9ea00fe210 0x7f9ea00fe680 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f9e94009990 tx=0x7f9e94009ca0 comp rx=0 tx=0).stop 2023-07-14T19:16:27.236 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:27.233+0000 7f9ea5d06700 1 -- 172.21.15.40:0/2023032765 shutdown_connections 2023-07-14T19:16:27.236 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:27.233+0000 7f9ea5d06700 1 --2- 172.21.15.40:0/2023032765 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9ea0104c70 0x7f9ea0105040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:27.236 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:27.233+0000 7f9ea5d06700 1 --2- 172.21.15.40:0/2023032765 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9ea00febc0 0x7f9ea010cb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:27.236 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:27.233+0000 7f9ea5d06700 1 --2- 172.21.15.40:0/2023032765 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9ea00fe210 0x7f9ea00fe680 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:27.236 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:27.233+0000 7f9ea5d06700 1 -- 172.21.15.40:0/2023032765 >> 172.21.15.40:0/2023032765 conn(0x7f9ea0075280 msgr2=0x7f9ea0075680 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:16:27.236 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:27.233+0000 7f9ea5d06700 1 -- 172.21.15.40:0/2023032765 shutdown_connections 2023-07-14T19:16:27.237 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:27.233+0000 7f9ea5d06700 1 -- 172.21.15.40:0/2023032765 wait complete. 2023-07-14T19:16:27.237 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:27.234+0000 7f9ea5d06700 1 Processor -- start 2023-07-14T19:16:27.238 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:27.234+0000 7f9ea5d06700 1 -- start start 2023-07-14T19:16:27.238 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:27.234+0000 7f9ea5d06700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9ea00fe210 0x7f9ea0195d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:16:27.238 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:27.234+0000 7f9ea5d06700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9ea00febc0 0x7f9ea0196260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:16:27.238 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:27.234+0000 7f9ea5d06700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9ea0104c70 0x7f9ea019a6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:16:27.239 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:27.234+0000 7f9ea5d06700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ea019ace0 con 0x7f9ea00febc0 2023-07-14T19:16:27.239 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:27.234+0000 7f9ea5d06700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ea019ae50 con 0x7f9ea0104c70 2023-07-14T19:16:27.239 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:27.234+0000 7f9ea5d06700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ea019b010 con 0x7f9ea00fe210 2023-07-14T19:16:27.239 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:27.234+0000 7f9e9effd700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9ea00febc0 0x7f9ea0196260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:16:27.239 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:27.234+0000 7f9e9effd700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9ea00febc0 0x7f9ea0196260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:33240/0 (socket says 172.21.15.40:33240) 2023-07-14T19:16:27.239 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:27.234+0000 7f9e9effd700 1 -- 172.21.15.40:0/3597683677 learned_addr learned my addr 172.21.15.40:0/3597683677 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-07-14T19:16:27.239 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:27.234+0000 7f9e9ffff700 1 --2- 172.21.15.40:0/3597683677 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9ea0104c70 0x7f9ea019a6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:16:27.239 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:27.234+0000 7f9e9f7fe700 1 --2- 172.21.15.40:0/3597683677 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9ea00fe210 0x7f9ea0195d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:16:27.240 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:27.235+0000 7f9e9effd700 1 -- 172.21.15.40:0/3597683677 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9ea00fe210 msgr2=0x7f9ea0195d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:16:27.240 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:27.235+0000 7f9e9effd700 1 --2- 172.21.15.40:0/3597683677 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9ea00fe210 0x7f9ea0195d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:27.240 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:27.235+0000 7f9e9effd700 1 -- 172.21.15.40:0/3597683677 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9ea0104c70 msgr2=0x7f9ea019a6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:16:27.240 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:27.235+0000 7f9e9effd700 1 --2- 172.21.15.40:0/3597683677 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9ea0104c70 0x7f9ea019a6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:27.240 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:27.235+0000 7f9e9effd700 1 -- 172.21.15.40:0/3597683677 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e940095f0 con 0x7f9ea00febc0 2023-07-14T19:16:27.240 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:27.235+0000 7f9e9effd700 1 --2- 172.21.15.40:0/3597683677 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9ea00febc0 0x7f9ea0196260 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f9e94000c00 tx=0x7f9e940047b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:16:27.240 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:27.235+0000 7f9e9cff9700 1 -- 172.21.15.40:0/3597683677 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e9401d070 con 0x7f9ea00febc0 2023-07-14T19:16:27.241 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:27.235+0000 7f9ea5d06700 1 -- 172.21.15.40:0/3597683677 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9ea019b290 con 0x7f9ea00febc0 2023-07-14T19:16:27.241 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:27.235+0000 7f9e9cff9700 1 -- 172.21.15.40:0/3597683677 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f9e94021410 con 0x7f9ea00febc0 2023-07-14T19:16:27.241 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:27.235+0000 7f9e9cff9700 1 -- 172.21.15.40:0/3597683677 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e94017650 con 0x7f9ea00febc0 2023-07-14T19:16:27.241 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:27.235+0000 7f9ea5d06700 1 -- 172.21.15.40:0/3597683677 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ea01a2b50 con 0x7f9ea00febc0 2023-07-14T19:16:27.241 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:27.236+0000 7f9ea5d06700 1 -- 172.21.15.40:0/3597683677 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f9ea010ffc0 con 0x7f9ea00febc0 2023-07-14T19:16:27.242 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:27.237+0000 7f9e9cff9700 1 -- 172.21.15.40:0/3597683677 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7f9e940217f0 con 0x7f9ea00febc0 2023-07-14T19:16:27.242 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:27.238+0000 7f9e9cff9700 1 --2- 172.21.15.40:0/3597683677 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f9e880a11e0 0x7f9e880a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:16:27.242 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:27.238+0000 7f9e9f7fe700 1 --2- 172.21.15.40:0/3597683677 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f9e880a11e0 0x7f9e880a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:16:27.243 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:27.238+0000 7f9e9cff9700 1 -- 172.21.15.40:0/3597683677 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f9e940ce7b0 con 0x7f9ea00febc0 2023-07-14T19:16:27.243 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:27.238+0000 7f9e9cff9700 1 --2- 172.21.15.40:0/3597683677 >> [v2:172.21.15.130:6816/2384019528,v1:172.21.15.130:6817/2384019528] conn(0x7f9e880aa9b0 0x7f9e880acdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:16:27.243 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:27.238+0000 7f9e9cff9700 1 -- 172.21.15.40:0/3597683677 --> [v2:172.21.15.130:6816/2384019528,v1:172.21.15.130:6817/2384019528] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f9e880ad470 con 0x7f9e880aa9b0 2023-07-14T19:16:27.243 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:27.238+0000 7f9e9cff9700 1 -- 172.21.15.40:0/3597683677 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_get_version_reply(handle=1 version=81) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9e940cec50 con 0x7f9ea00febc0 2023-07-14T19:16:27.243 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:27.238+0000 7f9e9f7fe700 1 --2- 172.21.15.40:0/3597683677 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f9e880a11e0 0x7f9e880a3690 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f9e90009c60 tx=0x7f9e90009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:16:27.243 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:27.238+0000 7f9e9ffff700 1 --2- 172.21.15.40:0/3597683677 >> [v2:172.21.15.130:6816/2384019528,v1:172.21.15.130:6817/2384019528] conn(0x7f9e880aa9b0 0x7f9e880acdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:16:27.243 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:27.239+0000 7f9e9ffff700 1 --2- 172.21.15.40:0/3597683677 >> [v2:172.21.15.130:6816/2384019528,v1:172.21.15.130:6817/2384019528] conn(0x7f9e880aa9b0 0x7f9e880acdc0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:16:27.244 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:27.240+0000 7f9e9cff9700 1 -- 172.21.15.40:0/3597683677 <== osd.7 v2:172.21.15.130:6816/2384019528 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7f9e880ad470 con 0x7f9e880aa9b0 2023-07-14T19:16:27.258 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:27.255+0000 7f9ea5d06700 1 -- 172.21.15.40:0/3597683677 --> [v2:172.21.15.130:6816/2384019528,v1:172.21.15.130:6817/2384019528] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f9ea004e340 con 0x7f9e880aa9b0 2023-07-14T19:16:27.258 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:27.256+0000 7f9e9cff9700 1 -- 172.21.15.40:0/3597683677 <== osd.7 v2:172.21.15.130:6816/2384019528 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f9ea004e340 con 0x7f9e880aa9b0 2023-07-14T19:16:27.259 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:27.256+0000 7f9ea5d06700 1 -- 172.21.15.40:0/3597683677 >> [v2:172.21.15.130:6816/2384019528,v1:172.21.15.130:6817/2384019528] conn(0x7f9e880aa9b0 msgr2=0x7f9e880acdc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:16:27.259 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:27.256+0000 7f9ea5d06700 1 --2- 172.21.15.40:0/3597683677 >> [v2:172.21.15.130:6816/2384019528,v1:172.21.15.130:6817/2384019528] conn(0x7f9e880aa9b0 0x7f9e880acdc0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:27.259 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:27.257+0000 7f9ea5d06700 1 -- 172.21.15.40:0/3597683677 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f9e880a11e0 msgr2=0x7f9e880a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:16:27.259 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:27.257+0000 7f9ea5d06700 1 --2- 172.21.15.40:0/3597683677 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f9e880a11e0 0x7f9e880a3690 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f9e90009c60 tx=0x7f9e90009260 comp rx=0 tx=0).stop 2023-07-14T19:16:27.260 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:27.257+0000 7f9ea5d06700 1 -- 172.21.15.40:0/3597683677 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9ea00febc0 msgr2=0x7f9ea0196260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:16:27.260 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:27.257+0000 7f9ea5d06700 1 --2- 172.21.15.40:0/3597683677 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9ea00febc0 0x7f9ea0196260 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f9e94000c00 tx=0x7f9e940047b0 comp rx=0 tx=0).stop 2023-07-14T19:16:27.260 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:27.257+0000 7f9e9ffff700 1 -- 172.21.15.40:0/3597683677 reap_dead start 2023-07-14T19:16:27.260 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:27.257+0000 7f9ea5d06700 1 -- 172.21.15.40:0/3597683677 shutdown_connections 2023-07-14T19:16:27.260 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:27.257+0000 7f9ea5d06700 1 -- 172.21.15.40:0/3597683677 >> 172.21.15.40:0/3597683677 conn(0x7f9ea0075280 msgr2=0x7f9ea00fca10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:16:27.260 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:27.257+0000 7f9ea5d06700 1 -- 172.21.15.40:0/3597683677 shutdown_connections 2023-07-14T19:16:27.261 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:27.257+0000 7f9ea5d06700 1 -- 172.21.15.40:0/3597683677 wait complete. 2023-07-14T19:16:28.033 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:27.939+0000 7f45a37cb700 1 -- 172.21.15.40:0/1392947123 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f459c103ee0 msgr2=0x7f459c10ba60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:16:28.033 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:27.939+0000 7f45a37cb700 1 --2- 172.21.15.40:0/1392947123 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f459c103ee0 0x7f459c10ba60 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f459400b0d0 tx=0x7f459400b3e0 comp rx=0 tx=0).stop 2023-07-14T19:16:28.033 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:27.939+0000 7f45a37cb700 1 -- 172.21.15.40:0/1392947123 shutdown_connections 2023-07-14T19:16:28.033 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:27.939+0000 7f45a37cb700 1 --2- 172.21.15.40:0/1392947123 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f459c103ee0 0x7f459c10ba60 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:28.033 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:27.939+0000 7f45a37cb700 1 --2- 172.21.15.40:0/1392947123 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f459c0fee40 0x7f459c0ff210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:28.033 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:27.939+0000 7f45a37cb700 1 --2- 172.21.15.40:0/1392947123 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f459c0ff7e0 0x7f459c1037b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:28.034 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:27.939+0000 7f45a37cb700 1 -- 172.21.15.40:0/1392947123 >> 172.21.15.40:0/1392947123 conn(0x7f459c075ca0 msgr2=0x7f459c0760a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:16:28.034 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:27.939+0000 7f45a37cb700 1 -- 172.21.15.40:0/1392947123 shutdown_connections 2023-07-14T19:16:28.034 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:27.940+0000 7f45a37cb700 1 -- 172.21.15.40:0/1392947123 wait complete. 2023-07-14T19:16:28.034 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:27.940+0000 7f45a37cb700 1 Processor -- start 2023-07-14T19:16:28.034 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:27.940+0000 7f45a37cb700 1 -- start start 2023-07-14T19:16:28.034 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:27.940+0000 7f45a37cb700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f459c0fee40 0x7f459c196cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:16:28.034 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:27.940+0000 7f45a37cb700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f459c0ff7e0 0x7f459c197200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:16:28.035 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:27.940+0000 7f45a37cb700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f459c103ee0 0x7f459c1a1d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:16:28.035 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:27.940+0000 7f45a37cb700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f459c1a2280 con 0x7f459c103ee0 2023-07-14T19:16:28.035 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:27.940+0000 7f45a37cb700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f459c1a23c0 con 0x7f459c0fee40 2023-07-14T19:16:28.035 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:27.940+0000 7f45a37cb700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f459c1a2500 con 0x7f459c0ff7e0 2023-07-14T19:16:28.035 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:27.940+0000 7f45a1d68700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f459c103ee0 0x7f459c1a1d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:16:28.035 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:27.940+0000 7f45a1d68700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f459c103ee0 0x7f459c1a1d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:33258/0 (socket says 172.21.15.40:33258) 2023-07-14T19:16:28.035 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:27.940+0000 7f45a1d68700 1 -- 172.21.15.40:0/2403047862 learned_addr learned my addr 172.21.15.40:0/2403047862 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-07-14T19:16:28.036 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:27.941+0000 7f45a1567700 1 --2- 172.21.15.40:0/2403047862 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f459c0fee40 0x7f459c196cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:16:28.036 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:27.941+0000 7f45a0d66700 1 --2- 172.21.15.40:0/2403047862 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f459c0ff7e0 0x7f459c197200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:16:28.036 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:27.941+0000 7f45a1d68700 1 -- 172.21.15.40:0/2403047862 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f459c0ff7e0 msgr2=0x7f459c197200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:16:28.036 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:27.941+0000 7f45a1d68700 1 --2- 172.21.15.40:0/2403047862 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f459c0ff7e0 0x7f459c197200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:28.036 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:27.941+0000 7f45a1d68700 1 -- 172.21.15.40:0/2403047862 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f459c0fee40 msgr2=0x7f459c196cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:16:28.036 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:27.941+0000 7f45a1d68700 1 --2- 172.21.15.40:0/2403047862 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f459c0fee40 0x7f459c196cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:28.036 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:27.941+0000 7f45a1d68700 1 -- 172.21.15.40:0/2403047862 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4594009d00 con 0x7f459c103ee0 2023-07-14T19:16:28.037 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:27.941+0000 7f45a1d68700 1 --2- 172.21.15.40:0/2403047862 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f459c103ee0 0x7f459c1a1d40 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f4594007870 tx=0x7f45940078a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:16:28.037 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:27.941+0000 7f45927fc700 1 -- 172.21.15.40:0/2403047862 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f459400e040 con 0x7f459c103ee0 2023-07-14T19:16:28.037 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:27.941+0000 7f45927fc700 1 -- 172.21.15.40:0/2403047862 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f4594003e10 con 0x7f459c103ee0 2023-07-14T19:16:28.037 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:27.941+0000 7f45a37cb700 1 -- 172.21.15.40:0/2403047862 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f459c1a2730 con 0x7f459c103ee0 2023-07-14T19:16:28.037 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:27.941+0000 7f45927fc700 1 -- 172.21.15.40:0/2403047862 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f459401b8a0 con 0x7f459c103ee0 2023-07-14T19:16:28.037 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:27.941+0000 7f45a37cb700 1 -- 172.21.15.40:0/2403047862 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f459c1a2ba0 con 0x7f459c103ee0 2023-07-14T19:16:28.037 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:27.942+0000 7f45a37cb700 1 -- 172.21.15.40:0/2403047862 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f459c10ed80 con 0x7f459c103ee0 2023-07-14T19:16:28.037 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:27.943+0000 7f45927fc700 1 -- 172.21.15.40:0/2403047862 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7f4594005090 con 0x7f459c103ee0 2023-07-14T19:16:28.038 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:27.944+0000 7f45927fc700 1 --2- 172.21.15.40:0/2403047862 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f45880a1260 0x7f45880a3710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:16:28.038 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:27.944+0000 7f45a1567700 1 --2- 172.21.15.40:0/2403047862 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f45880a1260 0x7f45880a3710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:16:28.038 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:27.944+0000 7f45927fc700 1 -- 172.21.15.40:0/2403047862 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f45940ce240 con 0x7f459c103ee0 2023-07-14T19:16:28.038 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:27.944+0000 7f45927fc700 1 --2- 172.21.15.40:0/2403047862 >> [v2:172.21.15.130:6808/3507448720,v1:172.21.15.130:6809/3507448720] conn(0x7f45880aaa30 0x7f45880ace40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:16:28.038 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:27.944+0000 7f45927fc700 1 -- 172.21.15.40:0/2403047862 --> [v2:172.21.15.130:6808/3507448720,v1:172.21.15.130:6809/3507448720] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f45880ad4f0 con 0x7f45880aaa30 2023-07-14T19:16:28.038 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:27.944+0000 7f45927fc700 1 -- 172.21.15.40:0/2403047862 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_get_version_reply(handle=1 version=81) v2 ==== 24+0+0 (secure 0 0 0) 0x7f45940ce6e0 con 0x7f459c103ee0 2023-07-14T19:16:28.038 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:27.944+0000 7f45a1567700 1 --2- 172.21.15.40:0/2403047862 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f45880a1260 0x7f45880a3710 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f4598005950 tx=0x7f459800b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:16:28.039 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:27.944+0000 7f45a0d66700 1 --2- 172.21.15.40:0/2403047862 >> [v2:172.21.15.130:6808/3507448720,v1:172.21.15.130:6809/3507448720] conn(0x7f45880aaa30 0x7f45880ace40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:16:28.039 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:27.945+0000 7f45a0d66700 1 --2- 172.21.15.40:0/2403047862 >> [v2:172.21.15.130:6808/3507448720,v1:172.21.15.130:6809/3507448720] conn(0x7f45880aaa30 0x7f45880ace40 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:16:28.039 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:27.946+0000 7f45927fc700 1 -- 172.21.15.40:0/2403047862 <== osd.4 v2:172.21.15.130:6808/3507448720 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7f45880ad4f0 con 0x7f45880aaa30 2023-07-14T19:16:28.039 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:27.963+0000 7f45a37cb700 1 -- 172.21.15.40:0/2403047862 --> [v2:172.21.15.130:6808/3507448720,v1:172.21.15.130:6809/3507448720] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f459c04e340 con 0x7f45880aaa30 2023-07-14T19:16:28.039 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:27.963+0000 7f45927fc700 1 -- 172.21.15.40:0/2403047862 <== osd.4 v2:172.21.15.130:6808/3507448720 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f459c04e340 con 0x7f45880aaa30 2023-07-14T19:16:28.039 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:27.963+0000 7f45a37cb700 1 -- 172.21.15.40:0/2403047862 >> [v2:172.21.15.130:6808/3507448720,v1:172.21.15.130:6809/3507448720] conn(0x7f45880aaa30 msgr2=0x7f45880ace40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:16:28.039 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:27.963+0000 7f45a37cb700 1 --2- 172.21.15.40:0/2403047862 >> [v2:172.21.15.130:6808/3507448720,v1:172.21.15.130:6809/3507448720] conn(0x7f45880aaa30 0x7f45880ace40 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:28.040 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:27.964+0000 7f45a37cb700 1 -- 172.21.15.40:0/2403047862 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f45880a1260 msgr2=0x7f45880a3710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:16:28.040 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:27.964+0000 7f45a37cb700 1 --2- 172.21.15.40:0/2403047862 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f45880a1260 0x7f45880a3710 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f4598005950 tx=0x7f459800b410 comp rx=0 tx=0).stop 2023-07-14T19:16:28.040 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:27.964+0000 7f45a37cb700 1 -- 172.21.15.40:0/2403047862 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f459c103ee0 msgr2=0x7f459c1a1d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:16:28.040 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:27.964+0000 7f45a37cb700 1 --2- 172.21.15.40:0/2403047862 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f459c103ee0 0x7f459c1a1d40 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f4594007870 tx=0x7f45940078a0 comp rx=0 tx=0).stop 2023-07-14T19:16:28.041 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:27.964+0000 7f45a1d68700 1 -- 172.21.15.40:0/2403047862 reap_dead start 2023-07-14T19:16:28.041 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:27.964+0000 7f45a37cb700 1 -- 172.21.15.40:0/2403047862 shutdown_connections 2023-07-14T19:16:28.041 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:27.964+0000 7f45a37cb700 1 -- 172.21.15.40:0/2403047862 >> 172.21.15.40:0/2403047862 conn(0x7f459c075ca0 msgr2=0x7f459c0fc4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:16:28.041 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:27.964+0000 7f45a37cb700 1 -- 172.21.15.40:0/2403047862 shutdown_connections 2023-07-14T19:16:28.041 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:27.964+0000 7f45a37cb700 1 -- 172.21.15.40:0/2403047862 wait complete. 2023-07-14T19:16:28.685 INFO:teuthology.orchestra.run.smithi040.stdout:326417514513 2023-07-14T19:16:28.686 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell --fsid adc52f04-2279-11ee-9b33-001a4aab830c -- ceph osd last-stat-seq osd.11 2023-07-14T19:16:28.760 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:28.758+0000 7f236e2ea700 1 -- 172.21.15.40:0/3392275226 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2368068990 msgr2=0x7f2368068d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:16:28.761 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:28.758+0000 7f236e2ea700 1 --2- 172.21.15.40:0/3392275226 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2368068990 0x7f2368068d60 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f2358009990 tx=0x7f2358009ca0 comp rx=0 tx=0).stop 2023-07-14T19:16:28.761 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:28.758+0000 7f236e2ea700 1 -- 172.21.15.40:0/3392275226 shutdown_connections 2023-07-14T19:16:28.761 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:28.758+0000 7f236e2ea700 1 --2- 172.21.15.40:0/3392275226 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f236810cad0 0x7f236810eeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:28.761 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:28.758+0000 7f236e2ea700 1 --2- 172.21.15.40:0/3392275226 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2368068990 0x7f2368068d60 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:28.761 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:28.758+0000 7f236e2ea700 1 --2- 172.21.15.40:0/3392275226 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f23680692a0 0x7f236810c590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:28.761 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:28.758+0000 7f236e2ea700 1 -- 172.21.15.40:0/3392275226 >> 172.21.15.40:0/3392275226 conn(0x7f2368074d90 msgr2=0x7f2368075190 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:16:28.761 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:28.759+0000 7f236e2ea700 1 -- 172.21.15.40:0/3392275226 shutdown_connections 2023-07-14T19:16:28.762 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:28.759+0000 7f236e2ea700 1 -- 172.21.15.40:0/3392275226 wait complete. 2023-07-14T19:16:28.762 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:28.759+0000 7f236e2ea700 1 Processor -- start 2023-07-14T19:16:28.763 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:28.759+0000 7f236e2ea700 1 -- start start 2023-07-14T19:16:28.763 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:28.760+0000 7f236e2ea700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2368068990 0x7f2368198190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:16:28.763 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:28.760+0000 7f236e2ea700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f23680692a0 0x7f23681986d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:16:28.763 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:28.760+0000 7f236e2ea700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f236810cad0 0x7f236819cb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:16:28.763 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:28.760+0000 7f236e2ea700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f236819d150 con 0x7f236810cad0 2023-07-14T19:16:28.763 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:28.760+0000 7f236e2ea700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f236819d2c0 con 0x7f2368068990 2023-07-14T19:16:28.764 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:28.760+0000 7f236e2ea700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f236819d480 con 0x7f23680692a0 2023-07-14T19:16:28.764 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:28.760+0000 7f236c887700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f236810cad0 0x7f236819cb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:16:28.764 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:28.760+0000 7f236c887700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f236810cad0 0x7f236819cb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:33270/0 (socket says 172.21.15.40:33270) 2023-07-14T19:16:28.764 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:28.760+0000 7f236c887700 1 -- 172.21.15.40:0/1727222416 learned_addr learned my addr 172.21.15.40:0/1727222416 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-07-14T19:16:28.764 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:28.760+0000 7f2367fff700 1 --2- 172.21.15.40:0/1727222416 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2368068990 0x7f2368198190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:16:28.764 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:28.761+0000 7f236c887700 1 -- 172.21.15.40:0/1727222416 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f23680692a0 msgr2=0x7f23681986d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-07-14T19:16:28.764 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:28.761+0000 7f236c887700 1 --2- 172.21.15.40:0/1727222416 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f23680692a0 0x7f23681986d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:28.765 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:28.761+0000 7f236c887700 1 -- 172.21.15.40:0/1727222416 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2368068990 msgr2=0x7f2368198190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:16:28.765 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:28.761+0000 7f236c887700 1 --2- 172.21.15.40:0/1727222416 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2368068990 0x7f2368198190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:28.765 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:28.761+0000 7f236c887700 1 -- 172.21.15.40:0/1727222416 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23580095f0 con 0x7f236810cad0 2023-07-14T19:16:28.765 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:28.761+0000 7f236c887700 1 --2- 172.21.15.40:0/1727222416 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f236810cad0 0x7f236819cb40 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f236000ef90 tx=0x7f236000c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:16:28.765 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:28.761+0000 7f23657fa700 1 -- 172.21.15.40:0/1727222416 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2360018070 con 0x7f236810cad0 2023-07-14T19:16:28.765 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:28.761+0000 7f236e2ea700 1 -- 172.21.15.40:0/1727222416 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f236819d700 con 0x7f236810cad0 2023-07-14T19:16:28.766 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:28.761+0000 7f23657fa700 1 -- 172.21.15.40:0/1727222416 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f236000f040 con 0x7f236810cad0 2023-07-14T19:16:28.766 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:28.761+0000 7f236e2ea700 1 -- 172.21.15.40:0/1727222416 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f23681a4fc0 con 0x7f236810cad0 2023-07-14T19:16:28.766 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:28.761+0000 7f23657fa700 1 -- 172.21.15.40:0/1727222416 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2360003e00 con 0x7f236810cad0 2023-07-14T19:16:28.766 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:28.762+0000 7f236e2ea700 1 -- 172.21.15.40:0/1727222416 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f2354000ff0 con 0x7f236810cad0 2023-07-14T19:16:28.766 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:28.764+0000 7f23657fa700 1 -- 172.21.15.40:0/1727222416 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7f236000ca50 con 0x7f236810cad0 2023-07-14T19:16:28.767 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:28.765+0000 7f23657fa700 1 --2- 172.21.15.40:0/1727222416 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f23500a11e0 0x7f23500a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:16:28.767 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:28.765+0000 7f23657fa700 1 -- 172.21.15.40:0/1727222416 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f23600cc460 con 0x7f236810cad0 2023-07-14T19:16:28.767 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:28.766+0000 7f23657fa700 1 --2- 172.21.15.40:0/1727222416 >> [v2:172.21.15.130:6824/2417067798,v1:172.21.15.130:6825/2417067798] conn(0x7f23500aa9b0 0x7f23500acdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:16:28.768 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:28.766+0000 7f23657fa700 1 -- 172.21.15.40:0/1727222416 --> [v2:172.21.15.130:6824/2417067798,v1:172.21.15.130:6825/2417067798] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f23500ad470 con 0x7f23500aa9b0 2023-07-14T19:16:28.768 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:28.766+0000 7f23657fa700 1 -- 172.21.15.40:0/1727222416 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_get_version_reply(handle=1 version=81) v2 ==== 24+0+0 (secure 0 0 0) 0x7f23600ceb70 con 0x7f236810cad0 2023-07-14T19:16:28.768 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:28.766+0000 7f2367fff700 1 --2- 172.21.15.40:0/1727222416 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f23500a11e0 0x7f23500a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:16:28.768 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:28.766+0000 7f2367fff700 1 --2- 172.21.15.40:0/1727222416 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f23500a11e0 0x7f23500a3690 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f235800b2e0 tx=0x7f2358005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:16:28.769 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:28.766+0000 7f23677fe700 1 --2- 172.21.15.40:0/1727222416 >> [v2:172.21.15.130:6824/2417067798,v1:172.21.15.130:6825/2417067798] conn(0x7f23500aa9b0 0x7f23500acdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:16:28.769 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:28.767+0000 7f23677fe700 1 --2- 172.21.15.40:0/1727222416 >> [v2:172.21.15.130:6824/2417067798,v1:172.21.15.130:6825/2417067798] conn(0x7f23500aa9b0 0x7f23500acdc0 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.10 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:16:28.769 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:28.767+0000 7f23657fa700 1 -- 172.21.15.40:0/1727222416 <== osd.10 v2:172.21.15.130:6824/2417067798 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7f23500ad470 con 0x7f23500aa9b0 2023-07-14T19:16:28.785 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:28.783+0000 7f236e2ea700 1 -- 172.21.15.40:0/1727222416 --> [v2:172.21.15.130:6824/2417067798,v1:172.21.15.130:6825/2417067798] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f2354002d70 con 0x7f23500aa9b0 2023-07-14T19:16:28.785 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:28.784+0000 7f23657fa700 1 -- 172.21.15.40:0/1727222416 <== osd.10 v2:172.21.15.130:6824/2417067798 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f2354002d70 con 0x7f23500aa9b0 2023-07-14T19:16:28.786 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:28.784+0000 7f236e2ea700 1 -- 172.21.15.40:0/1727222416 >> [v2:172.21.15.130:6824/2417067798,v1:172.21.15.130:6825/2417067798] conn(0x7f23500aa9b0 msgr2=0x7f23500acdc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:16:28.787 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:28.784+0000 7f236e2ea700 1 --2- 172.21.15.40:0/1727222416 >> [v2:172.21.15.130:6824/2417067798,v1:172.21.15.130:6825/2417067798] conn(0x7f23500aa9b0 0x7f23500acdc0 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:28.787 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:28.784+0000 7f236e2ea700 1 -- 172.21.15.40:0/1727222416 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f23500a11e0 msgr2=0x7f23500a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:16:28.787 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:28.784+0000 7f236e2ea700 1 --2- 172.21.15.40:0/1727222416 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f23500a11e0 0x7f23500a3690 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f235800b2e0 tx=0x7f2358005fd0 comp rx=0 tx=0).stop 2023-07-14T19:16:28.787 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:28.784+0000 7f236e2ea700 1 -- 172.21.15.40:0/1727222416 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f236810cad0 msgr2=0x7f236819cb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:16:28.787 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:28.784+0000 7f236e2ea700 1 --2- 172.21.15.40:0/1727222416 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f236810cad0 0x7f236819cb40 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f236000ef90 tx=0x7f236000c3a0 comp rx=0 tx=0).stop 2023-07-14T19:16:28.787 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:28.784+0000 7f236c887700 1 -- 172.21.15.40:0/1727222416 reap_dead start 2023-07-14T19:16:28.787 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:28.785+0000 7f236e2ea700 1 -- 172.21.15.40:0/1727222416 shutdown_connections 2023-07-14T19:16:28.788 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:28.785+0000 7f236e2ea700 1 -- 172.21.15.40:0/1727222416 >> 172.21.15.40:0/1727222416 conn(0x7f2368074d90 msgr2=0x7f23680fc960 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:16:28.788 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:28.785+0000 7f236e2ea700 1 -- 172.21.15.40:0/1727222416 shutdown_connections 2023-07-14T19:16:28.788 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:28.785+0000 7f236e2ea700 1 -- 172.21.15.40:0/1727222416 wait complete. 2023-07-14T19:16:28.822 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:28.820+0000 7fde76e04700 1 -- 172.21.15.40:0/1354365146 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fde701008d0 msgr2=0x7fde701048a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:16:28.822 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:28.820+0000 7fde76e04700 1 --2- 172.21.15.40:0/1354365146 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fde701008d0 0x7fde701048a0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fde5c0099c0 tx=0x7fde5c009cd0 comp rx=0 tx=0).stop 2023-07-14T19:16:28.822 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:28.820+0000 7fde76e04700 1 -- 172.21.15.40:0/1354365146 shutdown_connections 2023-07-14T19:16:28.822 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:28.820+0000 7fde76e04700 1 --2- 172.21.15.40:0/1354365146 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fde70104fd0 0x7fde7010cb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:28.823 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:28.820+0000 7fde76e04700 1 --2- 172.21.15.40:0/1354365146 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fde700fff30 0x7fde70100300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:28.823 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:28.820+0000 7fde76e04700 1 --2- 172.21.15.40:0/1354365146 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fde701008d0 0x7fde701048a0 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:28.823 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:28.820+0000 7fde76e04700 1 -- 172.21.15.40:0/1354365146 >> 172.21.15.40:0/1354365146 conn(0x7fde700fb790 msgr2=0x7fde700fdba0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:16:28.823 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:28.820+0000 7fde76e04700 1 -- 172.21.15.40:0/1354365146 shutdown_connections 2023-07-14T19:16:28.823 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:28.820+0000 7fde76e04700 1 -- 172.21.15.40:0/1354365146 wait complete. 2023-07-14T19:16:28.825 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:28.821+0000 7fde76e04700 1 Processor -- start 2023-07-14T19:16:28.825 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:28.821+0000 7fde76e04700 1 -- start start 2023-07-14T19:16:28.825 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:28.821+0000 7fde76e04700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fde700fff30 0x7fde70072940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:16:28.825 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:28.821+0000 7fde76e04700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fde701008d0 0x7fde70072e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:16:28.825 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:28.821+0000 7fde76e04700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fde70104fd0 0x7fde700733c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:16:28.826 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:28.821+0000 7fde76e04700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde701a41f0 con 0x7fde701008d0 2023-07-14T19:16:28.826 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:28.821+0000 7fde76e04700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde701a4360 con 0x7fde700fff30 2023-07-14T19:16:28.826 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:28.821+0000 7fde76e04700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde701a4520 con 0x7fde70104fd0 2023-07-14T19:16:28.826 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:28.821+0000 7fde6ffff700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fde701008d0 0x7fde70072e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:16:28.826 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:28.821+0000 7fde6ffff700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fde701008d0 0x7fde70072e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:33284/0 (socket says 172.21.15.40:33284) 2023-07-14T19:16:28.826 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:28.821+0000 7fde6ffff700 1 -- 172.21.15.40:0/2006992495 learned_addr learned my addr 172.21.15.40:0/2006992495 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-07-14T19:16:28.826 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:28.821+0000 7fde74ba0700 1 --2- 172.21.15.40:0/2006992495 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fde700fff30 0x7fde70072940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:16:28.826 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:28.821+0000 7fde753a1700 1 --2- 172.21.15.40:0/2006992495 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fde70104fd0 0x7fde700733c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:16:28.827 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:28.822+0000 7fde6ffff700 1 -- 172.21.15.40:0/2006992495 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fde70104fd0 msgr2=0x7fde700733c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:16:28.827 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:28.822+0000 7fde6ffff700 1 --2- 172.21.15.40:0/2006992495 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fde70104fd0 0x7fde700733c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:28.827 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:28.822+0000 7fde6ffff700 1 -- 172.21.15.40:0/2006992495 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fde700fff30 msgr2=0x7fde70072940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:16:28.827 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:28.822+0000 7fde6ffff700 1 --2- 172.21.15.40:0/2006992495 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fde700fff30 0x7fde70072940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:28.827 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:28.822+0000 7fde6ffff700 1 -- 172.21.15.40:0/2006992495 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde5c0095f0 con 0x7fde701008d0 2023-07-14T19:16:28.827 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:28.822+0000 7fde74ba0700 1 --2- 172.21.15.40:0/2006992495 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fde700fff30 0x7fde70072940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-14T19:16:28.827 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:28.822+0000 7fde6ffff700 1 --2- 172.21.15.40:0/2006992495 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fde701008d0 0x7fde70072e80 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fde5c009fd0 tx=0x7fde5c009580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:16:28.828 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:28.822+0000 7fde6dffb700 1 -- 172.21.15.40:0/2006992495 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde5c01d070 con 0x7fde701008d0 2023-07-14T19:16:28.828 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:28.822+0000 7fde76e04700 1 -- 172.21.15.40:0/2006992495 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fde701a47a0 con 0x7fde701008d0 2023-07-14T19:16:28.828 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:28.822+0000 7fde6dffb700 1 -- 172.21.15.40:0/2006992495 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fde5c005030 con 0x7fde701008d0 2023-07-14T19:16:28.828 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:28.822+0000 7fde6dffb700 1 -- 172.21.15.40:0/2006992495 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde5c004a90 con 0x7fde701008d0 2023-07-14T19:16:28.828 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:28.822+0000 7fde76e04700 1 -- 172.21.15.40:0/2006992495 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fde701a4bc0 con 0x7fde701008d0 2023-07-14T19:16:28.828 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:28.823+0000 7fde76e04700 1 -- 172.21.15.40:0/2006992495 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fde7004e340 con 0x7fde701008d0 2023-07-14T19:16:28.829 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:28.824+0000 7fde6dffb700 1 -- 172.21.15.40:0/2006992495 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7fde5c0051a0 con 0x7fde701008d0 2023-07-14T19:16:28.829 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:28.825+0000 7fde6dffb700 1 --2- 172.21.15.40:0/2006992495 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7fde600a11e0 0x7fde600a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:16:28.830 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:28.825+0000 7fde6dffb700 1 -- 172.21.15.40:0/2006992495 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7979+0+0 (secure 0 0 0) 0x7fde5c0cd5a0 con 0x7fde701008d0 2023-07-14T19:16:28.830 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:28.825+0000 7fde74ba0700 1 --2- 172.21.15.40:0/2006992495 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7fde600a11e0 0x7fde600a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:16:28.830 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:28.825+0000 7fde6dffb700 1 --2- 172.21.15.40:0/2006992495 >> [v2:172.21.15.130:6800/2512760702,v1:172.21.15.130:6801/2512760702] conn(0x7fde600aa9b0 0x7fde600acdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:16:28.830 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:28.825+0000 7fde6dffb700 1 -- 172.21.15.40:0/2006992495 --> [v2:172.21.15.130:6800/2512760702,v1:172.21.15.130:6801/2512760702] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fde600ad470 con 0x7fde600aa9b0 2023-07-14T19:16:28.830 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:28.825+0000 7fde6dffb700 1 -- 172.21.15.40:0/2006992495 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_get_version_reply(handle=1 version=81) v2 ==== 24+0+0 (secure 0 0 0) 0x7fde5c0cfcb0 con 0x7fde701008d0 2023-07-14T19:16:28.830 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:28.825+0000 7fde74ba0700 1 --2- 172.21.15.40:0/2006992495 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7fde600a11e0 0x7fde600a3690 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fde64011490 tx=0x7fde64009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:16:28.831 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:28.826+0000 7fde753a1700 1 --2- 172.21.15.40:0/2006992495 >> [v2:172.21.15.130:6800/2512760702,v1:172.21.15.130:6801/2512760702] conn(0x7fde600aa9b0 0x7fde600acdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:16:28.831 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:28.826+0000 7fde753a1700 1 --2- 172.21.15.40:0/2006992495 >> [v2:172.21.15.130:6800/2512760702,v1:172.21.15.130:6801/2512760702] conn(0x7fde600aa9b0 0x7fde600acdc0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:16:28.831 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:28.827+0000 7fde6dffb700 1 -- 172.21.15.40:0/2006992495 <== osd.1 v2:172.21.15.130:6800/2512760702 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7fde600ad470 con 0x7fde600aa9b0 2023-07-14T19:16:28.860 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:28.858+0000 7fde76e04700 1 -- 172.21.15.40:0/2006992495 --> [v2:172.21.15.130:6800/2512760702,v1:172.21.15.130:6801/2512760702] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fde7010fe70 con 0x7fde600aa9b0 2023-07-14T19:16:28.861 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:28.859+0000 7fde6dffb700 1 -- 172.21.15.40:0/2006992495 <== osd.1 v2:172.21.15.130:6800/2512760702 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fde6800f040 con 0x7fde600aa9b0 2023-07-14T19:16:28.862 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:28.859+0000 7fde76e04700 1 -- 172.21.15.40:0/2006992495 >> [v2:172.21.15.130:6800/2512760702,v1:172.21.15.130:6801/2512760702] conn(0x7fde600aa9b0 msgr2=0x7fde600acdc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:16:28.863 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:28.859+0000 7fde76e04700 1 --2- 172.21.15.40:0/2006992495 >> [v2:172.21.15.130:6800/2512760702,v1:172.21.15.130:6801/2512760702] conn(0x7fde600aa9b0 0x7fde600acdc0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:28.863 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:28.859+0000 7fde76e04700 1 -- 172.21.15.40:0/2006992495 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7fde600a11e0 msgr2=0x7fde600a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:16:28.863 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:28.859+0000 7fde76e04700 1 --2- 172.21.15.40:0/2006992495 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7fde600a11e0 0x7fde600a3690 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fde64011490 tx=0x7fde64009260 comp rx=0 tx=0).stop 2023-07-14T19:16:28.864 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:28.859+0000 7fde76e04700 1 -- 172.21.15.40:0/2006992495 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fde701008d0 msgr2=0x7fde70072e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:16:28.864 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:28.859+0000 7fde76e04700 1 --2- 172.21.15.40:0/2006992495 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fde701008d0 0x7fde70072e80 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fde5c009fd0 tx=0x7fde5c009580 comp rx=0 tx=0).stop 2023-07-14T19:16:28.864 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:28.859+0000 7fde753a1700 1 -- 172.21.15.40:0/2006992495 reap_dead start 2023-07-14T19:16:28.865 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:28.859+0000 7fde76e04700 1 -- 172.21.15.40:0/2006992495 shutdown_connections 2023-07-14T19:16:28.865 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:28.860+0000 7fde76e04700 1 -- 172.21.15.40:0/2006992495 >> 172.21.15.40:0/2006992495 conn(0x7fde700fb790 msgr2=0x7fde700fc230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:16:28.865 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:28.860+0000 7fde76e04700 1 -- 172.21.15.40:0/2006992495 shutdown_connections 2023-07-14T19:16:28.866 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:28.860+0000 7fde76e04700 1 -- 172.21.15.40:0/2006992495 wait complete. 2023-07-14T19:16:28.887 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:16:28 smithi040 ceph-mon[96518]: pgmap v269: 97 pgs: 97 active+clean; 638 KiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 1.7 KiB/s wr, 11 op/s 2023-07-14T19:16:28.972 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:16:28 smithi191 ceph-mon[98311]: pgmap v269: 97 pgs: 97 active+clean; 638 KiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 1.7 KiB/s wr, 11 op/s 2023-07-14T19:16:28.992 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:16:28 smithi130 ceph-mon[97832]: pgmap v269: 97 pgs: 97 active+clean; 638 KiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 1.7 KiB/s wr, 11 op/s 2023-07-14T19:16:29.818 INFO:teuthology.orchestra.run.smithi040.stdout:244813135902 2023-07-14T19:16:29.819 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell --fsid adc52f04-2279-11ee-9b33-001a4aab830c -- ceph osd last-stat-seq osd.8 2023-07-14T19:16:29.943 INFO:teuthology.orchestra.run.smithi040.stdout:296352743445 2023-07-14T19:16:29.943 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell --fsid adc52f04-2279-11ee-9b33-001a4aab830c -- ceph osd last-stat-seq osd.10 2023-07-14T19:16:29.944 INFO:teuthology.orchestra.run.smithi040.stdout:214748364834 2023-07-14T19:16:29.945 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell --fsid adc52f04-2279-11ee-9b33-001a4aab830c -- ceph osd last-stat-seq osd.7 2023-07-14T19:16:30.376 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:30.228+0000 7f94b9898700 1 -- 172.21.15.40:0/720832040 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f94b4104c70 msgr2=0x7f94b4105040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:16:30.377 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:30.228+0000 7f94b9898700 1 --2- 172.21.15.40:0/720832040 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f94b4104c70 0x7f94b4105040 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f94a4009990 tx=0x7f94a4009ca0 comp rx=0 tx=0).stop 2023-07-14T19:16:30.377 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:30.228+0000 7f94b9898700 1 -- 172.21.15.40:0/720832040 shutdown_connections 2023-07-14T19:16:30.377 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:30.228+0000 7f94b9898700 1 --2- 172.21.15.40:0/720832040 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f94b40febc0 0x7f94b410cb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:30.378 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:30.228+0000 7f94b9898700 1 --2- 172.21.15.40:0/720832040 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f94b4104c70 0x7f94b4105040 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:30.378 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:30.228+0000 7f94b9898700 1 --2- 172.21.15.40:0/720832040 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f94b40fe210 0x7f94b40fe680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:30.379 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:30.228+0000 7f94b9898700 1 -- 172.21.15.40:0/720832040 >> 172.21.15.40:0/720832040 conn(0x7f94b4075280 msgr2=0x7f94b4075680 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:16:30.379 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:30.228+0000 7f94b9898700 1 -- 172.21.15.40:0/720832040 shutdown_connections 2023-07-14T19:16:30.379 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:30.228+0000 7f94b9898700 1 -- 172.21.15.40:0/720832040 wait complete. 2023-07-14T19:16:30.379 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:30.229+0000 7f94b9898700 1 Processor -- start 2023-07-14T19:16:30.379 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:30.229+0000 7f94b9898700 1 -- start start 2023-07-14T19:16:30.379 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:30.229+0000 7f94b9898700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f94b40fe210 0x7f94b4195d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:16:30.379 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:30.229+0000 7f94b9898700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f94b40febc0 0x7f94b41962c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:16:30.380 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:30.229+0000 7f94b9898700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f94b4104c70 0x7f94b419a730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:16:30.380 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:30.229+0000 7f94b9898700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94b419ad40 con 0x7f94b4104c70 2023-07-14T19:16:30.380 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:30.229+0000 7f94b9898700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94b419aeb0 con 0x7f94b40febc0 2023-07-14T19:16:30.380 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:30.229+0000 7f94b9898700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94b419b070 con 0x7f94b40fe210 2023-07-14T19:16:30.380 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:30.229+0000 7f94b37fe700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f94b4104c70 0x7f94b419a730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:16:30.381 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:30.229+0000 7f94b37fe700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f94b4104c70 0x7f94b419a730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:33302/0 (socket says 172.21.15.40:33302) 2023-07-14T19:16:30.381 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:30.229+0000 7f94b37fe700 1 -- 172.21.15.40:0/2313049830 learned_addr learned my addr 172.21.15.40:0/2313049830 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-07-14T19:16:30.381 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:30.230+0000 7f94b27fc700 1 --2- 172.21.15.40:0/2313049830 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f94b40febc0 0x7f94b41962c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:16:30.381 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:30.230+0000 7f94b2ffd700 1 --2- 172.21.15.40:0/2313049830 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f94b40fe210 0x7f94b4195d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:16:30.381 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:30.230+0000 7f94b37fe700 1 -- 172.21.15.40:0/2313049830 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f94b40fe210 msgr2=0x7f94b4195d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:16:30.381 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:30.230+0000 7f94b37fe700 1 --2- 172.21.15.40:0/2313049830 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f94b40fe210 0x7f94b4195d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:30.382 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:30.230+0000 7f94b37fe700 1 -- 172.21.15.40:0/2313049830 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f94b40febc0 msgr2=0x7f94b41962c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:16:30.382 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:30.230+0000 7f94b37fe700 1 --2- 172.21.15.40:0/2313049830 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f94b40febc0 0x7f94b41962c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:30.382 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:30.230+0000 7f94b37fe700 1 -- 172.21.15.40:0/2313049830 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94a40095f0 con 0x7f94b4104c70 2023-07-14T19:16:30.382 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:30.230+0000 7f94b37fe700 1 --2- 172.21.15.40:0/2313049830 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f94b4104c70 0x7f94b419a730 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f94ac00ada0 tx=0x7f94ac00c330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:16:30.382 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:30.230+0000 7f94b8896700 1 -- 172.21.15.40:0/2313049830 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94ac00f070 con 0x7f94b4104c70 2023-07-14T19:16:30.382 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:30.230+0000 7f94b8896700 1 -- 172.21.15.40:0/2313049830 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f94ac007b30 con 0x7f94b4104c70 2023-07-14T19:16:30.382 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:30.230+0000 7f94b9898700 1 -- 172.21.15.40:0/2313049830 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f94b419b350 con 0x7f94b4104c70 2023-07-14T19:16:30.382 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:30.230+0000 7f94b8896700 1 -- 172.21.15.40:0/2313049830 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94ac005440 con 0x7f94b4104c70 2023-07-14T19:16:30.383 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:30.230+0000 7f94b9898700 1 -- 172.21.15.40:0/2313049830 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f94b41a2bb0 con 0x7f94b4104c70 2023-07-14T19:16:30.383 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:30.231+0000 7f94b9898700 1 -- 172.21.15.40:0/2313049830 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f94b410ff60 con 0x7f94b4104c70 2023-07-14T19:16:30.383 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:30.232+0000 7f94b8896700 1 -- 172.21.15.40:0/2313049830 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7f94ac004ad0 con 0x7f94b4104c70 2023-07-14T19:16:30.384 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:30.233+0000 7f94b8896700 1 --2- 172.21.15.40:0/2313049830 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f949c0a1260 0x7f949c0a3710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:16:30.384 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:30.233+0000 7f94b2ffd700 1 --2- 172.21.15.40:0/2313049830 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f949c0a1260 0x7f949c0a3710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:16:30.384 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:30.233+0000 7f94b8896700 1 -- 172.21.15.40:0/2313049830 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f94ac0cc910 con 0x7f94b4104c70 2023-07-14T19:16:30.384 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:30.233+0000 7f94b8896700 1 --2- 172.21.15.40:0/2313049830 >> [v2:172.21.15.40:6810/996287860,v1:172.21.15.40:6811/996287860] conn(0x7f949c0aaa30 0x7f949c0ace40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:16:30.384 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:30.233+0000 7f94b8896700 1 -- 172.21.15.40:0/2313049830 --> [v2:172.21.15.40:6810/996287860,v1:172.21.15.40:6811/996287860] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f949c0ad4f0 con 0x7f949c0aaa30 2023-07-14T19:16:30.384 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:30.233+0000 7f94b8896700 1 -- 172.21.15.40:0/2313049830 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_get_version_reply(handle=1 version=81) v2 ==== 24+0+0 (secure 0 0 0) 0x7f94ac0ccdb0 con 0x7f94b4104c70 2023-07-14T19:16:30.385 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:30.233+0000 7f94b27fc700 1 --2- 172.21.15.40:0/2313049830 >> [v2:172.21.15.40:6810/996287860,v1:172.21.15.40:6811/996287860] conn(0x7f949c0aaa30 0x7f949c0ace40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:16:30.385 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:30.233+0000 7f94b2ffd700 1 --2- 172.21.15.40:0/2313049830 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f949c0a1260 0x7f949c0a3710 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f94a4006010 tx=0x7f94a400b540 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:16:30.385 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:30.234+0000 7f94b27fc700 1 --2- 172.21.15.40:0/2313049830 >> [v2:172.21.15.40:6810/996287860,v1:172.21.15.40:6811/996287860] conn(0x7f949c0aaa30 0x7f949c0ace40 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:16:30.385 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:30.234+0000 7f94b8896700 1 -- 172.21.15.40:0/2313049830 <== osd.3 v2:172.21.15.40:6810/996287860 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7f949c0ad4f0 con 0x7f949c0aaa30 2023-07-14T19:16:30.385 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:30.253+0000 7f94b9898700 1 -- 172.21.15.40:0/2313049830 --> [v2:172.21.15.40:6810/996287860,v1:172.21.15.40:6811/996287860] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f94b404e340 con 0x7f949c0aaa30 2023-07-14T19:16:30.385 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:30.254+0000 7f94b8896700 1 -- 172.21.15.40:0/2313049830 <== osd.3 v2:172.21.15.40:6810/996287860 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f94b404e340 con 0x7f949c0aaa30 2023-07-14T19:16:30.385 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:30.254+0000 7f94b9898700 1 -- 172.21.15.40:0/2313049830 >> [v2:172.21.15.40:6810/996287860,v1:172.21.15.40:6811/996287860] conn(0x7f949c0aaa30 msgr2=0x7f949c0ace40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:16:30.385 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:30.254+0000 7f94b9898700 1 --2- 172.21.15.40:0/2313049830 >> [v2:172.21.15.40:6810/996287860,v1:172.21.15.40:6811/996287860] conn(0x7f949c0aaa30 0x7f949c0ace40 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:30.386 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:30.254+0000 7f94b9898700 1 -- 172.21.15.40:0/2313049830 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f949c0a1260 msgr2=0x7f949c0a3710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:16:30.386 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:30.254+0000 7f94b9898700 1 --2- 172.21.15.40:0/2313049830 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f949c0a1260 0x7f949c0a3710 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f94a4006010 tx=0x7f94a400b540 comp rx=0 tx=0).stop 2023-07-14T19:16:30.386 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:30.254+0000 7f94b9898700 1 -- 172.21.15.40:0/2313049830 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f94b4104c70 msgr2=0x7f94b419a730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:16:30.386 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:30.254+0000 7f94b9898700 1 --2- 172.21.15.40:0/2313049830 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f94b4104c70 0x7f94b419a730 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f94ac00ada0 tx=0x7f94ac00c330 comp rx=0 tx=0).stop 2023-07-14T19:16:30.386 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:30.254+0000 7f94b37fe700 1 -- 172.21.15.40:0/2313049830 reap_dead start 2023-07-14T19:16:30.386 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:30.254+0000 7f94b9898700 1 -- 172.21.15.40:0/2313049830 shutdown_connections 2023-07-14T19:16:30.386 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:30.254+0000 7f94b9898700 1 -- 172.21.15.40:0/2313049830 >> 172.21.15.40:0/2313049830 conn(0x7f94b4075280 msgr2=0x7f94b40fbce0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:16:30.387 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:30.254+0000 7f94b9898700 1 -- 172.21.15.40:0/2313049830 shutdown_connections 2023-07-14T19:16:30.387 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:30.255+0000 7f94b9898700 1 -- 172.21.15.40:0/2313049830 wait complete. 2023-07-14T19:16:30.432 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:30.260+0000 7f532b29e700 1 -- 172.21.15.40:0/1236053532 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5324103ee0 msgr2=0x7f532410ba60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:16:30.433 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:30.260+0000 7f532b29e700 1 --2- 172.21.15.40:0/1236053532 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5324103ee0 0x7f532410ba60 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f531c00b0d0 tx=0x7f531c00b3e0 comp rx=0 tx=0).stop 2023-07-14T19:16:30.433 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:30.260+0000 7f532b29e700 1 -- 172.21.15.40:0/1236053532 shutdown_connections 2023-07-14T19:16:30.433 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:30.260+0000 7f532b29e700 1 --2- 172.21.15.40:0/1236053532 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f53240fee40 0x7f53240ff210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:30.433 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:30.260+0000 7f532b29e700 1 --2- 172.21.15.40:0/1236053532 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5324103ee0 0x7f532410ba60 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:30.433 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:30.260+0000 7f532b29e700 1 --2- 172.21.15.40:0/1236053532 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f53240ff7e0 0x7f53241037b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:30.433 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:30.260+0000 7f532b29e700 1 -- 172.21.15.40:0/1236053532 >> 172.21.15.40:0/1236053532 conn(0x7f5324075ca0 msgr2=0x7f53240760a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:16:30.434 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:30.260+0000 7f532b29e700 1 -- 172.21.15.40:0/1236053532 shutdown_connections 2023-07-14T19:16:30.434 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:30.260+0000 7f532b29e700 1 -- 172.21.15.40:0/1236053532 wait complete. 2023-07-14T19:16:30.434 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:30.260+0000 7f532b29e700 1 Processor -- start 2023-07-14T19:16:30.434 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:30.261+0000 7f532b29e700 1 -- start start 2023-07-14T19:16:30.434 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:30.261+0000 7f532b29e700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f53240fee40 0x7f5324198080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:16:30.434 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:30.261+0000 7f532b29e700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f53240ff7e0 0x7f53241985c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:16:30.434 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:30.261+0000 7f532b29e700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5324103ee0 0x7f532419ca30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:16:30.434 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:30.261+0000 7f532b29e700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f532419d040 con 0x7f53240fee40 2023-07-14T19:16:30.435 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:30.261+0000 7f532b29e700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f532419d1b0 con 0x7f53240ff7e0 2023-07-14T19:16:30.435 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:30.261+0000 7f532b29e700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f532419d370 con 0x7f5324103ee0 2023-07-14T19:16:30.435 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:30.261+0000 7f532903a700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f53240fee40 0x7f5324198080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:16:30.435 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:30.261+0000 7f532903a700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f53240fee40 0x7f5324198080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:33314/0 (socket says 172.21.15.40:33314) 2023-07-14T19:16:30.435 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:30.261+0000 7f532903a700 1 -- 172.21.15.40:0/1591533628 learned_addr learned my addr 172.21.15.40:0/1591533628 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-07-14T19:16:30.435 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:30.261+0000 7f532983b700 1 --2- 172.21.15.40:0/1591533628 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5324103ee0 0x7f532419ca30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:16:30.435 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:30.261+0000 7f5328839700 1 --2- 172.21.15.40:0/1591533628 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f53240ff7e0 0x7f53241985c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:16:30.436 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:30.261+0000 7f532903a700 1 -- 172.21.15.40:0/1591533628 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5324103ee0 msgr2=0x7f532419ca30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:16:30.436 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:30.261+0000 7f532903a700 1 --2- 172.21.15.40:0/1591533628 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5324103ee0 0x7f532419ca30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:30.436 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:30.261+0000 7f532903a700 1 -- 172.21.15.40:0/1591533628 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f53240ff7e0 msgr2=0x7f53241985c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:16:30.436 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:30.261+0000 7f532903a700 1 --2- 172.21.15.40:0/1591533628 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f53240ff7e0 0x7f53241985c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:30.436 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:30.261+0000 7f532903a700 1 -- 172.21.15.40:0/1591533628 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f531c009d00 con 0x7f53240fee40 2023-07-14T19:16:30.436 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:30.261+0000 7f532903a700 1 --2- 172.21.15.40:0/1591533628 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f53240fee40 0x7f5324198080 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f532000d950 tx=0x7f532000dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:16:30.436 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:30.262+0000 7f531a7fc700 1 -- 172.21.15.40:0/1591533628 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f532000f890 con 0x7f53240fee40 2023-07-14T19:16:30.436 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:30.262+0000 7f531a7fc700 1 -- 172.21.15.40:0/1591533628 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f5320010460 con 0x7f53240fee40 2023-07-14T19:16:30.437 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:30.262+0000 7f532b29e700 1 -- 172.21.15.40:0/1591533628 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f532419d650 con 0x7f53240fee40 2023-07-14T19:16:30.437 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:30.262+0000 7f531a7fc700 1 -- 172.21.15.40:0/1591533628 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f532000e610 con 0x7f53240fee40 2023-07-14T19:16:30.437 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:30.262+0000 7f532b29e700 1 -- 172.21.15.40:0/1591533628 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f53241a4eb0 con 0x7f53240fee40 2023-07-14T19:16:30.437 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:30.262+0000 7f532b29e700 1 -- 172.21.15.40:0/1591533628 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f532410ed80 con 0x7f53240fee40 2023-07-14T19:16:30.437 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:30.264+0000 7f531a7fc700 1 -- 172.21.15.40:0/1591533628 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7f53200108d0 con 0x7f53240fee40 2023-07-14T19:16:30.437 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:30.265+0000 7f531a7fc700 1 --2- 172.21.15.40:0/1591533628 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f53100a11e0 0x7f53100a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:16:30.437 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:30.265+0000 7f531a7fc700 1 -- 172.21.15.40:0/1591533628 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f53200ccb70 con 0x7f53240fee40 2023-07-14T19:16:30.438 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:30.265+0000 7f5328839700 1 --2- 172.21.15.40:0/1591533628 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f53100a11e0 0x7f53100a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:16:30.438 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:30.265+0000 7f531a7fc700 1 --2- 172.21.15.40:0/1591533628 >> [v2:172.21.15.191:6808/1781944663,v1:172.21.15.191:6809/1781944663] conn(0x7f53100aa9b0 0x7f53100acdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:16:30.438 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:30.265+0000 7f531a7fc700 1 -- 172.21.15.40:0/1591533628 --> [v2:172.21.15.191:6808/1781944663,v1:172.21.15.191:6809/1781944663] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f53100ad470 con 0x7f53100aa9b0 2023-07-14T19:16:30.438 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:30.265+0000 7f531a7fc700 1 -- 172.21.15.40:0/1591533628 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_get_version_reply(handle=1 version=81) v2 ==== 24+0+0 (secure 0 0 0) 0x7f53200cd010 con 0x7f53240fee40 2023-07-14T19:16:30.439 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:30.266+0000 7f5328839700 1 --2- 172.21.15.40:0/1591533628 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f53100a11e0 0x7f53100a3690 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f53241996a0 tx=0x7f5314009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:16:30.439 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:30.266+0000 7f532983b700 1 --2- 172.21.15.40:0/1591533628 >> [v2:172.21.15.191:6808/1781944663,v1:172.21.15.191:6809/1781944663] conn(0x7f53100aa9b0 0x7f53100acdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:16:30.439 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:30.266+0000 7f532983b700 1 --2- 172.21.15.40:0/1591533628 >> [v2:172.21.15.191:6808/1781944663,v1:172.21.15.191:6809/1781944663] conn(0x7f53100aa9b0 0x7f53100acdc0 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:16:30.439 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:30.267+0000 7f531a7fc700 1 -- 172.21.15.40:0/1591533628 <== osd.5 v2:172.21.15.191:6808/1781944663 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7f53100ad470 con 0x7f53100aa9b0 2023-07-14T19:16:30.439 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:30.283+0000 7f532b29e700 1 -- 172.21.15.40:0/1591533628 --> [v2:172.21.15.191:6808/1781944663,v1:172.21.15.191:6809/1781944663] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f532404e340 con 0x7f53100aa9b0 2023-07-14T19:16:30.439 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:30.284+0000 7f531a7fc700 1 -- 172.21.15.40:0/1591533628 <== osd.5 v2:172.21.15.191:6808/1781944663 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f532404e340 con 0x7f53100aa9b0 2023-07-14T19:16:30.439 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:30.284+0000 7f532b29e700 1 -- 172.21.15.40:0/1591533628 >> [v2:172.21.15.191:6808/1781944663,v1:172.21.15.191:6809/1781944663] conn(0x7f53100aa9b0 msgr2=0x7f53100acdc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:16:30.440 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:30.284+0000 7f532b29e700 1 --2- 172.21.15.40:0/1591533628 >> [v2:172.21.15.191:6808/1781944663,v1:172.21.15.191:6809/1781944663] conn(0x7f53100aa9b0 0x7f53100acdc0 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:30.440 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:30.284+0000 7f532b29e700 1 -- 172.21.15.40:0/1591533628 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f53100a11e0 msgr2=0x7f53100a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:16:30.440 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:30.284+0000 7f532b29e700 1 --2- 172.21.15.40:0/1591533628 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f53100a11e0 0x7f53100a3690 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f53241996a0 tx=0x7f5314009260 comp rx=0 tx=0).stop 2023-07-14T19:16:30.440 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:30.284+0000 7f532b29e700 1 -- 172.21.15.40:0/1591533628 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f53240fee40 msgr2=0x7f5324198080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:16:30.440 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:30.284+0000 7f532b29e700 1 --2- 172.21.15.40:0/1591533628 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f53240fee40 0x7f5324198080 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f532000d950 tx=0x7f532000dc60 comp rx=0 tx=0).stop 2023-07-14T19:16:30.440 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:30.284+0000 7f532983b700 1 -- 172.21.15.40:0/1591533628 reap_dead start 2023-07-14T19:16:30.440 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:30.284+0000 7f532b29e700 1 -- 172.21.15.40:0/1591533628 shutdown_connections 2023-07-14T19:16:30.440 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:30.284+0000 7f532b29e700 1 -- 172.21.15.40:0/1591533628 >> 172.21.15.40:0/1591533628 conn(0x7f5324075ca0 msgr2=0x7f5324068990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:16:30.441 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:30.284+0000 7f532b29e700 1 -- 172.21.15.40:0/1591533628 shutdown_connections 2023-07-14T19:16:30.441 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:30.284+0000 7f532b29e700 1 -- 172.21.15.40:0/1591533628 wait complete. 2023-07-14T19:16:30.541 INFO:teuthology.orchestra.run.smithi040.stdout:55834574908 2023-07-14T19:16:30.542 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell --fsid adc52f04-2279-11ee-9b33-001a4aab830c -- ceph osd last-stat-seq osd.1 2023-07-14T19:16:30.904 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:16:30 smithi040 ceph-mon[96518]: pgmap v270: 97 pgs: 97 active+clean; 638 KiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 255 B/s wr, 11 op/s 2023-07-14T19:16:30.972 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:16:30 smithi191 ceph-mon[98311]: pgmap v270: 97 pgs: 97 active+clean; 638 KiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 255 B/s wr, 11 op/s 2023-07-14T19:16:30.992 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:16:30 smithi130 ceph-mon[97832]: pgmap v270: 97 pgs: 97 active+clean; 638 KiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 255 B/s wr, 11 op/s 2023-07-14T19:16:30.994 INFO:teuthology.orchestra.run.smithi040.stdout:137438953520 2023-07-14T19:16:30.995 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell --fsid adc52f04-2279-11ee-9b33-001a4aab830c -- ceph osd last-stat-seq osd.4 2023-07-14T19:16:31.168 INFO:teuthology.orchestra.run.smithi040.stdout:158913789996 2023-07-14T19:16:31.168 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell --fsid adc52f04-2279-11ee-9b33-001a4aab830c -- ceph osd last-stat-seq osd.5 2023-07-14T19:16:31.620 INFO:teuthology.orchestra.run.smithi040.stdout:107374182452 2023-07-14T19:16:31.620 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell --fsid adc52f04-2279-11ee-9b33-001a4aab830c -- ceph osd last-stat-seq osd.3 2023-07-14T19:16:31.629 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/adc52f04-2279-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T19:16:31.645 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/adc52f04-2279-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T19:16:31.668 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/adc52f04-2279-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T19:16:31.830 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/adc52f04-2279-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T19:16:32.135 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/adc52f04-2279-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T19:16:32.135 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/adc52f04-2279-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T19:16:32.502 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/adc52f04-2279-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T19:16:32.818 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/adc52f04-2279-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T19:16:32.849 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:16:32 smithi040 ceph-mon[96518]: pgmap v271: 97 pgs: 97 active+clean; 638 KiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 255 B/s wr, 11 op/s 2023-07-14T19:16:32.974 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:16:32 smithi191 ceph-mon[98311]: pgmap v271: 97 pgs: 97 active+clean; 638 KiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 255 B/s wr, 11 op/s 2023-07-14T19:16:32.992 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:16:32 smithi130 ceph-mon[97832]: pgmap v271: 97 pgs: 97 active+clean; 638 KiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 255 B/s wr, 11 op/s 2023-07-14T19:16:34.949 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:16:34 smithi040 ceph-mon[96518]: pgmap v272: 97 pgs: 97 active+clean; 638 KiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 255 B/s wr, 9 op/s 2023-07-14T19:16:34.972 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:16:34 smithi191 ceph-mon[98311]: pgmap v272: 97 pgs: 97 active+clean; 638 KiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 255 B/s wr, 9 op/s 2023-07-14T19:16:34.992 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:16:34 smithi130 ceph-mon[97832]: pgmap v272: 97 pgs: 97 active+clean; 638 KiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 255 B/s wr, 9 op/s 2023-07-14T19:16:35.385 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:35.383+0000 7fb3a3999700 1 -- 172.21.15.40:0/498039548 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb39c100920 msgr2=0x7fb39c1048f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:16:35.385 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:35.383+0000 7fb3a3999700 1 --2- 172.21.15.40:0/498039548 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb39c100920 0x7fb39c1048f0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fb38c0099c0 tx=0x7fb38c009cd0 comp rx=0 tx=0).stop 2023-07-14T19:16:35.385 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:35.383+0000 7fb3a3999700 1 -- 172.21.15.40:0/498039548 shutdown_connections 2023-07-14T19:16:35.385 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:35.383+0000 7fb3a3999700 1 --2- 172.21.15.40:0/498039548 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb39c105020 0x7fb39c10cba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:35.386 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:35.383+0000 7fb3a3999700 1 --2- 172.21.15.40:0/498039548 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb39c0fff80 0x7fb39c100350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:35.386 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:35.383+0000 7fb3a3999700 1 --2- 172.21.15.40:0/498039548 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb39c100920 0x7fb39c1048f0 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:35.386 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:35.383+0000 7fb3a3999700 1 -- 172.21.15.40:0/498039548 >> 172.21.15.40:0/498039548 conn(0x7fb39c0fb840 msgr2=0x7fb39c0fdc50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:16:35.386 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:35.383+0000 7fb3a3999700 1 -- 172.21.15.40:0/498039548 shutdown_connections 2023-07-14T19:16:35.386 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:35.383+0000 7fb3a3999700 1 -- 172.21.15.40:0/498039548 wait complete. 2023-07-14T19:16:35.388 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:35.384+0000 7fb3a3999700 1 Processor -- start 2023-07-14T19:16:35.388 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:35.384+0000 7fb3a3999700 1 -- start start 2023-07-14T19:16:35.388 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:35.384+0000 7fb3a3999700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb39c0fff80 0x7fb39c195db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:16:35.388 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:35.384+0000 7fb3a3999700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb39c100920 0x7fb39c1962f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:16:35.388 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:35.384+0000 7fb3a3999700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb39c105020 0x7fb39c19a760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:16:35.389 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:35.384+0000 7fb3a3999700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb39c19ad70 con 0x7fb39c100920 2023-07-14T19:16:35.389 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:35.384+0000 7fb3a3999700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb39c19aee0 con 0x7fb39c105020 2023-07-14T19:16:35.389 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:35.384+0000 7fb3a3999700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb39c19b0a0 con 0x7fb39c0fff80 2023-07-14T19:16:35.389 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:35.384+0000 7fb3a0f34700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb39c100920 0x7fb39c1962f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:16:35.389 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:35.384+0000 7fb3a0f34700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb39c100920 0x7fb39c1962f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:33332/0 (socket says 172.21.15.40:33332) 2023-07-14T19:16:35.389 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:35.384+0000 7fb3a0f34700 1 -- 172.21.15.40:0/1189564659 learned_addr learned my addr 172.21.15.40:0/1189564659 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-07-14T19:16:35.389 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:35.384+0000 7fb3a1735700 1 --2- 172.21.15.40:0/1189564659 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb39c0fff80 0x7fb39c195db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:16:35.390 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:35.384+0000 7fb3a1f36700 1 --2- 172.21.15.40:0/1189564659 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb39c105020 0x7fb39c19a760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:16:35.390 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:35.385+0000 7fb3a0f34700 1 -- 172.21.15.40:0/1189564659 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb39c0fff80 msgr2=0x7fb39c195db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:16:35.390 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:35.385+0000 7fb3a0f34700 1 --2- 172.21.15.40:0/1189564659 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb39c0fff80 0x7fb39c195db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:35.390 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:35.385+0000 7fb3a0f34700 1 -- 172.21.15.40:0/1189564659 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb39c105020 msgr2=0x7fb39c19a760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:16:35.390 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:35.385+0000 7fb3a0f34700 1 --2- 172.21.15.40:0/1189564659 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb39c105020 0x7fb39c19a760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:35.390 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:35.385+0000 7fb3a0f34700 1 -- 172.21.15.40:0/1189564659 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb38c0095f0 con 0x7fb39c100920 2023-07-14T19:16:35.390 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:35.385+0000 7fb3a1f36700 1 --2- 172.21.15.40:0/1189564659 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb39c105020 0x7fb39c19a760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-14T19:16:35.390 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:35.385+0000 7fb3a0f34700 1 --2- 172.21.15.40:0/1189564659 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb39c100920 0x7fb39c1962f0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7fb38c006010 tx=0x7fb38c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:16:35.391 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:35.385+0000 7fb3927fc700 1 -- 172.21.15.40:0/1189564659 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb38c01d070 con 0x7fb39c100920 2023-07-14T19:16:35.391 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:35.385+0000 7fb3927fc700 1 -- 172.21.15.40:0/1189564659 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb38c004090 con 0x7fb39c100920 2023-07-14T19:16:35.391 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:35.385+0000 7fb3927fc700 1 -- 172.21.15.40:0/1189564659 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb38c022ea0 con 0x7fb39c100920 2023-07-14T19:16:35.391 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:35.385+0000 7fb3a3999700 1 -- 172.21.15.40:0/1189564659 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb39c19b320 con 0x7fb39c100920 2023-07-14T19:16:35.391 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:35.385+0000 7fb3a3999700 1 -- 172.21.15.40:0/1189564659 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb39c1a2be0 con 0x7fb39c100920 2023-07-14T19:16:35.391 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:35.386+0000 7fb3a3999700 1 -- 172.21.15.40:0/1189564659 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb39c04e340 con 0x7fb39c100920 2023-07-14T19:16:35.392 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:35.387+0000 7fb3927fc700 1 -- 172.21.15.40:0/1189564659 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7fb38c003bd0 con 0x7fb39c100920 2023-07-14T19:16:35.392 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:35.388+0000 7fb3927fc700 1 --2- 172.21.15.40:0/1189564659 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7fb3880a1260 0x7fb3880a3710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:16:35.392 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:35.388+0000 7fb3a1735700 1 --2- 172.21.15.40:0/1189564659 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7fb3880a1260 0x7fb3880a3710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:16:35.393 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:35.388+0000 7fb3927fc700 1 -- 172.21.15.40:0/1189564659 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7979+0+0 (secure 0 0 0) 0x7fb38c0cdad0 con 0x7fb39c100920 2023-07-14T19:16:35.393 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:35.388+0000 7fb3a1735700 1 --2- 172.21.15.40:0/1189564659 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7fb3880a1260 0x7fb3880a3710 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fb398006fd0 tx=0x7fb398008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:16:35.394 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:35.392+0000 7fb3927fc700 1 -- 172.21.15.40:0/1189564659 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fb38c05c780 con 0x7fb39c100920 2023-07-14T19:16:35.620 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:35.618+0000 7fb3a3999700 1 -- 172.21.15.40:0/1189564659 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 11} v 0) v1 -- 0x7fb39c1a2d70 con 0x7fb39c100920 2023-07-14T19:16:35.620 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:35.618+0000 7fb3927fc700 1 -- 172.21.15.40:0/1189564659 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 11}]=0 v0) v1 ==== 75+0+13 (secure 0 0 0) 0x7fb38c027020 con 0x7fb39c100920 2023-07-14T19:16:35.620 INFO:teuthology.orchestra.run.smithi040.stdout:326417514515 2023-07-14T19:16:35.622 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:35.620+0000 7fb3a3999700 1 -- 172.21.15.40:0/1189564659 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7fb3880a1260 msgr2=0x7fb3880a3710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:16:35.622 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:35.620+0000 7fb3a3999700 1 --2- 172.21.15.40:0/1189564659 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7fb3880a1260 0x7fb3880a3710 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fb398006fd0 tx=0x7fb398008040 comp rx=0 tx=0).stop 2023-07-14T19:16:35.622 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:35.620+0000 7fb3a3999700 1 -- 172.21.15.40:0/1189564659 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb39c100920 msgr2=0x7fb39c1962f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:16:35.623 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:35.620+0000 7fb3a3999700 1 --2- 172.21.15.40:0/1189564659 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb39c100920 0x7fb39c1962f0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7fb38c006010 tx=0x7fb38c0057f0 comp rx=0 tx=0).stop 2023-07-14T19:16:35.623 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:35.620+0000 7fb3a3999700 1 -- 172.21.15.40:0/1189564659 shutdown_connections 2023-07-14T19:16:35.623 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:35.620+0000 7fb3a3999700 1 --2- 172.21.15.40:0/1189564659 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7fb3880a1260 0x7fb3880a3710 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:35.623 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:35.620+0000 7fb3a3999700 1 --2- 172.21.15.40:0/1189564659 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb39c0fff80 0x7fb39c195db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:35.623 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:35.620+0000 7fb3a3999700 1 --2- 172.21.15.40:0/1189564659 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb39c105020 0x7fb39c19a760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:35.623 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:35.620+0000 7fb3a3999700 1 --2- 172.21.15.40:0/1189564659 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb39c100920 0x7fb39c1962f0 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:35.624 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:35.620+0000 7fb3a3999700 1 -- 172.21.15.40:0/1189564659 >> 172.21.15.40:0/1189564659 conn(0x7fb39c0fb840 msgr2=0x7fb39c0fc310 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:16:35.624 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:35.620+0000 7fb3a3999700 1 -- 172.21.15.40:0/1189564659 shutdown_connections 2023-07-14T19:16:35.624 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:35.621+0000 7fb3a3999700 1 -- 172.21.15.40:0/1189564659 wait complete. 2023-07-14T19:16:35.637 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:35.635+0000 7fdda3e10700 1 -- 172.21.15.40:0/2533127250 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdd9c0fee70 msgr2=0x7fdd9c0ff240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:16:35.638 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:35.635+0000 7fdda3e10700 1 --2- 172.21.15.40:0/2533127250 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdd9c0fee70 0x7fdd9c0ff240 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7fdd980099c0 tx=0x7fdd98009cd0 comp rx=0 tx=0).stop 2023-07-14T19:16:35.638 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:35.636+0000 7fdda3e10700 1 -- 172.21.15.40:0/2533127250 shutdown_connections 2023-07-14T19:16:35.638 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:35.636+0000 7fdda3e10700 1 --2- 172.21.15.40:0/2533127250 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fdd9c103f10 0x7fdd9c10ba90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:35.638 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:35.636+0000 7fdda3e10700 1 --2- 172.21.15.40:0/2533127250 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdd9c0fee70 0x7fdd9c0ff240 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:35.638 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:35.636+0000 7fdda3e10700 1 --2- 172.21.15.40:0/2533127250 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fdd9c0ff810 0x7fdd9c1037e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:35.639 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:35.636+0000 7fdda3e10700 1 -- 172.21.15.40:0/2533127250 >> 172.21.15.40:0/2533127250 conn(0x7fdd9c075d20 msgr2=0x7fdd9c076120 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:16:35.639 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:35.636+0000 7fdda3e10700 1 -- 172.21.15.40:0/2533127250 shutdown_connections 2023-07-14T19:16:35.639 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:35.636+0000 7fdda3e10700 1 -- 172.21.15.40:0/2533127250 wait complete. 2023-07-14T19:16:35.641 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:35.636+0000 7fdda3e10700 1 Processor -- start 2023-07-14T19:16:35.641 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:35.636+0000 7fdda3e10700 1 -- start start 2023-07-14T19:16:35.641 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:35.637+0000 7fdda3e10700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fdd9c0fee70 0x7fdd9c194990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:16:35.641 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:35.637+0000 7fdda3e10700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fdd9c0ff810 0x7fdd9c194ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:16:35.641 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:35.637+0000 7fdda3e10700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdd9c103f10 0x7fdd9c195410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:16:35.641 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:35.637+0000 7fdda3e10700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd9c1a41d0 con 0x7fdd9c103f10 2023-07-14T19:16:35.642 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:35.637+0000 7fdda3e10700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd9c1a4340 con 0x7fdd9c0fee70 2023-07-14T19:16:35.642 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:35.637+0000 7fdda3e10700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd9c1a4500 con 0x7fdd9c0ff810 2023-07-14T19:16:35.642 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:35.637+0000 7fdda23ad700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdd9c103f10 0x7fdd9c195410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:16:35.642 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:35.637+0000 7fdda23ad700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdd9c103f10 0x7fdd9c195410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:33348/0 (socket says 172.21.15.40:33348) 2023-07-14T19:16:35.642 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:35.637+0000 7fdda23ad700 1 -- 172.21.15.40:0/2928387204 learned_addr learned my addr 172.21.15.40:0/2928387204 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-07-14T19:16:35.642 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:35.637+0000 7fdda1bac700 1 --2- 172.21.15.40:0/2928387204 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fdd9c0fee70 0x7fdd9c194990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:16:35.642 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:35.637+0000 7fdda13ab700 1 --2- 172.21.15.40:0/2928387204 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fdd9c0ff810 0x7fdd9c194ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:16:35.643 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:35.637+0000 7fdda23ad700 1 -- 172.21.15.40:0/2928387204 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fdd9c0ff810 msgr2=0x7fdd9c194ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:16:35.643 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:35.637+0000 7fdda23ad700 1 --2- 172.21.15.40:0/2928387204 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fdd9c0ff810 0x7fdd9c194ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:35.643 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:35.637+0000 7fdda23ad700 1 -- 172.21.15.40:0/2928387204 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fdd9c0fee70 msgr2=0x7fdd9c194990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:16:35.643 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:35.637+0000 7fdda23ad700 1 --2- 172.21.15.40:0/2928387204 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fdd9c0fee70 0x7fdd9c194990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:35.643 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:35.637+0000 7fdda23ad700 1 -- 172.21.15.40:0/2928387204 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd980095f0 con 0x7fdd9c103f10 2023-07-14T19:16:35.643 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:35.638+0000 7fdda23ad700 1 --2- 172.21.15.40:0/2928387204 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdd9c103f10 0x7fdd9c195410 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fdd9400ef30 tx=0x7fdd9400c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:16:35.643 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:35.638+0000 7fdd92ffd700 1 -- 172.21.15.40:0/2928387204 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd94018070 con 0x7fdd9c103f10 2023-07-14T19:16:35.644 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:35.638+0000 7fdd92ffd700 1 -- 172.21.15.40:0/2928387204 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fdd9400f040 con 0x7fdd9c103f10 2023-07-14T19:16:35.644 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:35.638+0000 7fdda3e10700 1 -- 172.21.15.40:0/2928387204 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdd9c1a4780 con 0x7fdd9c103f10 2023-07-14T19:16:35.644 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:35.638+0000 7fdd92ffd700 1 -- 172.21.15.40:0/2928387204 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd9400cdb0 con 0x7fdd9c103f10 2023-07-14T19:16:35.644 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:35.638+0000 7fdda3e10700 1 -- 172.21.15.40:0/2928387204 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd9c1a4c30 con 0x7fdd9c103f10 2023-07-14T19:16:35.644 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:35.639+0000 7fdda3e10700 1 -- 172.21.15.40:0/2928387204 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdd9c04e340 con 0x7fdd9c103f10 2023-07-14T19:16:35.645 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:35.641+0000 7fdd92ffd700 1 -- 172.21.15.40:0/2928387204 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7fdd94003a00 con 0x7fdd9c103f10 2023-07-14T19:16:35.645 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:35.642+0000 7fdd92ffd700 1 --2- 172.21.15.40:0/2928387204 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7fdd880a11e0 0x7fdd880a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:16:35.645 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:35.642+0000 7fdd92ffd700 1 -- 172.21.15.40:0/2928387204 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7979+0+0 (secure 0 0 0) 0x7fdd940cce20 con 0x7fdd9c103f10 2023-07-14T19:16:35.645 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:35.642+0000 7fdda1bac700 1 --2- 172.21.15.40:0/2928387204 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7fdd880a11e0 0x7fdd880a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:16:35.646 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:35.642+0000 7fdda1bac700 1 --2- 172.21.15.40:0/2928387204 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7fdd880a11e0 0x7fdd880a3690 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fdd98009fd0 tx=0x7fdd98005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:16:35.647 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:35.646+0000 7fdd92ffd700 1 -- 172.21.15.40:0/2928387204 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fdd94096650 con 0x7fdd9c103f10 2023-07-14T19:16:35.858 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:35.856+0000 7fdda3e10700 1 -- 172.21.15.40:0/2928387204 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7fdd9c1a4fe0 con 0x7fdd9c103f10 2023-07-14T19:16:35.858 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:35.857+0000 7fdd92ffd700 1 -- 172.21.15.40:0/2928387204 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fdd94099c70 con 0x7fdd9c103f10 2023-07-14T19:16:35.859 INFO:teuthology.orchestra.run.smithi040.stdout:34359738434 2023-07-14T19:16:35.861 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:35.859+0000 7fdda3e10700 1 -- 172.21.15.40:0/2928387204 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7fdd880a11e0 msgr2=0x7fdd880a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:16:35.861 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:35.859+0000 7fdda3e10700 1 --2- 172.21.15.40:0/2928387204 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7fdd880a11e0 0x7fdd880a3690 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fdd98009fd0 tx=0x7fdd98005fb0 comp rx=0 tx=0).stop 2023-07-14T19:16:35.861 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:35.859+0000 7fdda3e10700 1 -- 172.21.15.40:0/2928387204 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdd9c103f10 msgr2=0x7fdd9c195410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:16:35.862 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:35.859+0000 7fdda3e10700 1 --2- 172.21.15.40:0/2928387204 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdd9c103f10 0x7fdd9c195410 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fdd9400ef30 tx=0x7fdd9400c3a0 comp rx=0 tx=0).stop 2023-07-14T19:16:35.862 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:35.859+0000 7fdda3e10700 1 -- 172.21.15.40:0/2928387204 shutdown_connections 2023-07-14T19:16:35.862 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:35.859+0000 7fdda3e10700 1 --2- 172.21.15.40:0/2928387204 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7fdd880a11e0 0x7fdd880a3690 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:35.862 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:35.859+0000 7fdda3e10700 1 --2- 172.21.15.40:0/2928387204 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdd9c103f10 0x7fdd9c195410 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:35.862 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:35.859+0000 7fdda3e10700 1 --2- 172.21.15.40:0/2928387204 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fdd9c0fee70 0x7fdd9c194990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:35.862 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:35.859+0000 7fdda3e10700 1 --2- 172.21.15.40:0/2928387204 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fdd9c0ff810 0x7fdd9c194ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:35.862 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:35.859+0000 7fdda3e10700 1 -- 172.21.15.40:0/2928387204 >> 172.21.15.40:0/2928387204 conn(0x7fdd9c075d20 msgr2=0x7fdd9c0fd4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:16:35.863 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:35.860+0000 7fdda3e10700 1 -- 172.21.15.40:0/2928387204 shutdown_connections 2023-07-14T19:16:35.863 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:35.860+0000 7fdda3e10700 1 -- 172.21.15.40:0/2928387204 wait complete. 2023-07-14T19:16:35.971 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:16:35 smithi040 ceph-mon[96518]: from='client.? 172.21.15.40:0/1189564659' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2023-07-14T19:16:35.971 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:16:35 smithi191 ceph-mon[98311]: from='client.? 172.21.15.40:0/1189564659' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2023-07-14T19:16:35.992 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:16:35 smithi130 ceph-mon[97832]: from='client.? 172.21.15.40:0/1189564659' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2023-07-14T19:16:36.715 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:36.713+0000 7f00aea65700 1 -- 172.21.15.40:0/3766192443 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f00a80ff7e0 msgr2=0x7f00a81037b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:16:36.715 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:36.713+0000 7f00aea65700 1 --2- 172.21.15.40:0/3766192443 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f00a80ff7e0 0x7f00a81037b0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f00940099c0 tx=0x7f0094009cd0 comp rx=0 tx=0).stop 2023-07-14T19:16:36.715 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:36.713+0000 7f00aea65700 1 -- 172.21.15.40:0/3766192443 shutdown_connections 2023-07-14T19:16:36.716 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:36.713+0000 7f00aea65700 1 --2- 172.21.15.40:0/3766192443 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f00a8103ee0 0x7f00a810ba60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:36.716 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:36.713+0000 7f00aea65700 1 --2- 172.21.15.40:0/3766192443 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f00a80fee40 0x7f00a80ff210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:36.716 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:36.713+0000 7f00aea65700 1 --2- 172.21.15.40:0/3766192443 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f00a80ff7e0 0x7f00a81037b0 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:36.716 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:36.713+0000 7f00aea65700 1 -- 172.21.15.40:0/3766192443 >> 172.21.15.40:0/3766192443 conn(0x7f00a8075ca0 msgr2=0x7f00a80760a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:16:36.716 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:36.713+0000 7f00aea65700 1 -- 172.21.15.40:0/3766192443 shutdown_connections 2023-07-14T19:16:36.716 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:36.713+0000 7f00aea65700 1 -- 172.21.15.40:0/3766192443 wait complete. 2023-07-14T19:16:36.718 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:36.714+0000 7f00aea65700 1 Processor -- start 2023-07-14T19:16:36.718 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:36.714+0000 7f00aea65700 1 -- start start 2023-07-14T19:16:36.718 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:36.714+0000 7f00aea65700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f00a80fee40 0x7f00a8196bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:16:36.718 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:36.714+0000 7f00aea65700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f00a80ff7e0 0x7f00a81970f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:16:36.718 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:36.714+0000 7f00aea65700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f00a8103ee0 0x7f00a81a1c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:16:36.718 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:36.714+0000 7f00aea65700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00a81a2170 con 0x7f00a80fee40 2023-07-14T19:16:36.719 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:36.714+0000 7f00aea65700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00a81a22b0 con 0x7f00a80ff7e0 2023-07-14T19:16:36.719 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:36.714+0000 7f00aea65700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00a81a23f0 con 0x7f00a8103ee0 2023-07-14T19:16:36.719 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:36.714+0000 7f00ac801700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f00a80fee40 0x7f00a8196bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:16:36.719 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:36.714+0000 7f00ac801700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f00a80fee40 0x7f00a8196bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:33358/0 (socket says 172.21.15.40:33358) 2023-07-14T19:16:36.719 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:36.714+0000 7f00ac801700 1 -- 172.21.15.40:0/2784581362 learned_addr learned my addr 172.21.15.40:0/2784581362 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-07-14T19:16:36.719 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:36.714+0000 7f00a7fff700 1 --2- 172.21.15.40:0/2784581362 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f00a80ff7e0 0x7f00a81970f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:16:36.719 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:36.714+0000 7f00ad002700 1 --2- 172.21.15.40:0/2784581362 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f00a8103ee0 0x7f00a81a1c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:16:36.720 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:36.714+0000 7f00ac801700 1 -- 172.21.15.40:0/2784581362 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f00a8103ee0 msgr2=0x7f00a81a1c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:16:36.720 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:36.714+0000 7f00ac801700 1 --2- 172.21.15.40:0/2784581362 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f00a8103ee0 0x7f00a81a1c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:36.720 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:36.714+0000 7f00ac801700 1 -- 172.21.15.40:0/2784581362 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f00a80ff7e0 msgr2=0x7f00a81970f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:16:36.720 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:36.714+0000 7f00ac801700 1 --2- 172.21.15.40:0/2784581362 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f00a80ff7e0 0x7f00a81970f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:36.720 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:36.714+0000 7f00ac801700 1 -- 172.21.15.40:0/2784581362 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00940095f0 con 0x7f00a80fee40 2023-07-14T19:16:36.721 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:36.715+0000 7f00ac801700 1 --2- 172.21.15.40:0/2784581362 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f00a80fee40 0x7f00a8196bb0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f009c00d950 tx=0x7f009c00dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:16:36.721 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:36.715+0000 7f00a5ffb700 1 -- 172.21.15.40:0/2784581362 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f009c00f890 con 0x7f00a80fee40 2023-07-14T19:16:36.721 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:36.715+0000 7f00aea65700 1 -- 172.21.15.40:0/2784581362 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f00a8194ac0 con 0x7f00a80fee40 2023-07-14T19:16:36.721 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:36.715+0000 7f00a5ffb700 1 -- 172.21.15.40:0/2784581362 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f009c010460 con 0x7f00a80fee40 2023-07-14T19:16:36.721 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:36.715+0000 7f00a5ffb700 1 -- 172.21.15.40:0/2784581362 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f009c00e610 con 0x7f00a80fee40 2023-07-14T19:16:36.721 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:36.715+0000 7f00aea65700 1 -- 172.21.15.40:0/2784581362 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f00a8194fe0 con 0x7f00a80fee40 2023-07-14T19:16:36.721 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:36.716+0000 7f00aea65700 1 -- 172.21.15.40:0/2784581362 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f00a804e340 con 0x7f00a80fee40 2023-07-14T19:16:36.722 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:36.717+0000 7f00a5ffb700 1 -- 172.21.15.40:0/2784581362 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7f009c0108d0 con 0x7f00a80fee40 2023-07-14T19:16:36.722 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:36.718+0000 7f00a5ffb700 1 --2- 172.21.15.40:0/2784581362 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f00980a11e0 0x7f00980a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:16:36.722 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:36.718+0000 7f00a7fff700 1 --2- 172.21.15.40:0/2784581362 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f00980a11e0 0x7f00980a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:16:36.723 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:36.718+0000 7f00a5ffb700 1 -- 172.21.15.40:0/2784581362 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f009c0cc6d0 con 0x7f00a80fee40 2023-07-14T19:16:36.723 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:36.718+0000 7f00a7fff700 1 --2- 172.21.15.40:0/2784581362 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f00980a11e0 0x7f00980a3690 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f009400b580 tx=0x7f0094003680 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:16:36.724 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:36.722+0000 7f00a5ffb700 1 -- 172.21.15.40:0/2784581362 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f009c095e70 con 0x7f00a80fee40 2023-07-14T19:16:36.926 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:36.924+0000 7f00aea65700 1 -- 172.21.15.40:0/2784581362 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) v1 -- 0x7f00a8068750 con 0x7f00a80fee40 2023-07-14T19:16:36.927 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:36.925+0000 7f00a5ffb700 1 -- 172.21.15.40:0/2784581362 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f00a8068750 con 0x7f00a80fee40 2023-07-14T19:16:36.927 INFO:teuthology.orchestra.run.smithi040.stdout:214748364835 2023-07-14T19:16:36.929 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:36.927+0000 7f00aea65700 1 -- 172.21.15.40:0/2784581362 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f00980a11e0 msgr2=0x7f00980a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:16:36.929 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:36.927+0000 7f00aea65700 1 --2- 172.21.15.40:0/2784581362 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f00980a11e0 0x7f00980a3690 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f009400b580 tx=0x7f0094003680 comp rx=0 tx=0).stop 2023-07-14T19:16:36.930 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:36.927+0000 7f00aea65700 1 -- 172.21.15.40:0/2784581362 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f00a80fee40 msgr2=0x7f00a8196bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:16:36.930 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:36.927+0000 7f00aea65700 1 --2- 172.21.15.40:0/2784581362 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f00a80fee40 0x7f00a8196bb0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f009c00d950 tx=0x7f009c00dc60 comp rx=0 tx=0).stop 2023-07-14T19:16:36.930 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:36.927+0000 7f00aea65700 1 -- 172.21.15.40:0/2784581362 shutdown_connections 2023-07-14T19:16:36.930 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:36.927+0000 7f00aea65700 1 --2- 172.21.15.40:0/2784581362 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f00980a11e0 0x7f00980a3690 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:36.931 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:36.927+0000 7f00aea65700 1 --2- 172.21.15.40:0/2784581362 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f00a8103ee0 0x7f00a81a1c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:36.931 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:36.927+0000 7f00aea65700 1 --2- 172.21.15.40:0/2784581362 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f00a80fee40 0x7f00a8196bb0 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:36.932 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:36.927+0000 7f00aea65700 1 --2- 172.21.15.40:0/2784581362 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f00a80ff7e0 0x7f00a81970f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:36.932 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:36.927+0000 7f00aea65700 1 -- 172.21.15.40:0/2784581362 >> 172.21.15.40:0/2784581362 conn(0x7f00a8075ca0 msgr2=0x7f00a80fc4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:16:36.932 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:36.927+0000 7f00aea65700 1 -- 172.21.15.40:0/2784581362 shutdown_connections 2023-07-14T19:16:36.933 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:36.927+0000 7f00aea65700 1 -- 172.21.15.40:0/2784581362 wait complete. 2023-07-14T19:16:36.992 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:16:36 smithi130 ceph-mon[97832]: pgmap v273: 97 pgs: 97 active+clean; 638 KiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 255 B/s wr, 9 op/s 2023-07-14T19:16:36.992 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:16:36 smithi130 ceph-mon[97832]: from='client.? 172.21.15.40:0/2928387204' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2023-07-14T19:16:37.073 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:16:36 smithi040 ceph-mon[96518]: pgmap v273: 97 pgs: 97 active+clean; 638 KiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 255 B/s wr, 9 op/s 2023-07-14T19:16:37.074 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:16:36 smithi040 ceph-mon[96518]: from='client.? 172.21.15.40:0/2928387204' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2023-07-14T19:16:37.222 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:16:36 smithi191 ceph-mon[98311]: pgmap v273: 97 pgs: 97 active+clean; 638 KiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 255 B/s wr, 9 op/s 2023-07-14T19:16:37.222 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:16:36 smithi191 ceph-mon[98311]: from='client.? 172.21.15.40:0/2928387204' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2023-07-14T19:16:37.306 INFO:tasks.cephadm.ceph_manager.ceph:need seq 326417514513 got 326417514515 for osd.11 2023-07-14T19:16:37.306 DEBUG:teuthology.parallel:result is None 2023-07-14T19:16:37.353 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738432 got 34359738434 for osd.0 2023-07-14T19:16:37.353 DEBUG:teuthology.parallel:result is None 2023-07-14T19:16:37.442 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/adc52f04-2279-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T19:16:37.511 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:37.509+0000 7f041b1b4700 1 -- 172.21.15.40:0/3883402878 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0414100070 msgr2=0x7f0414100440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:16:37.511 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:37.509+0000 7f041b1b4700 1 --2- 172.21.15.40:0/3883402878 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0414100070 0x7f0414100440 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f0404009990 tx=0x7f0404009ca0 comp rx=0 tx=0).stop 2023-07-14T19:16:37.512 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:37.509+0000 7f041b1b4700 1 -- 172.21.15.40:0/3883402878 shutdown_connections 2023-07-14T19:16:37.512 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:37.509+0000 7f041b1b4700 1 --2- 172.21.15.40:0/3883402878 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0414109620 0x7f041410ba00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:37.512 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:37.509+0000 7f041b1b4700 1 --2- 172.21.15.40:0/3883402878 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0414100070 0x7f0414100440 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:37.512 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:37.509+0000 7f041b1b4700 1 --2- 172.21.15.40:0/3883402878 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0414100980 0x7f04141090e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:37.512 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:37.509+0000 7f041b1b4700 1 -- 172.21.15.40:0/3883402878 >> 172.21.15.40:0/3883402878 conn(0x7f0414074550 msgr2=0x7f0414074950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:16:37.512 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:37.509+0000 7f041b1b4700 1 -- 172.21.15.40:0/3883402878 shutdown_connections 2023-07-14T19:16:37.513 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:37.509+0000 7f041b1b4700 1 -- 172.21.15.40:0/3883402878 wait complete. 2023-07-14T19:16:37.514 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:37.510+0000 7f041b1b4700 1 Processor -- start 2023-07-14T19:16:37.514 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:37.510+0000 7f041b1b4700 1 -- start start 2023-07-14T19:16:37.514 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:37.510+0000 7f041b1b4700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0414100070 0x7f0414194ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:16:37.514 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:37.510+0000 7f041b1b4700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0414100980 0x7f04141951e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:16:37.515 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:37.510+0000 7f041b1b4700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0414109620 0x7f04141985a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:16:37.515 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:37.510+0000 7f041b1b4700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0414198ba0 con 0x7f0414109620 2023-07-14T19:16:37.515 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:37.510+0000 7f041b1b4700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0414198d10 con 0x7f0414100980 2023-07-14T19:16:37.515 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:37.510+0000 7f041b1b4700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0414198ed0 con 0x7f0414100070 2023-07-14T19:16:37.516 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:37.510+0000 7f0419751700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0414109620 0x7f04141985a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:16:37.516 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:37.510+0000 7f0419751700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0414109620 0x7f04141985a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:33376/0 (socket says 172.21.15.40:33376) 2023-07-14T19:16:37.516 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:37.510+0000 7f0419751700 1 -- 172.21.15.40:0/3864325393 learned_addr learned my addr 172.21.15.40:0/3864325393 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-07-14T19:16:37.516 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:37.511+0000 7f0413fff700 1 --2- 172.21.15.40:0/3864325393 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0414100980 0x7f04141951e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:16:37.517 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:37.511+0000 7f0418f50700 1 --2- 172.21.15.40:0/3864325393 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0414100070 0x7f0414194ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:16:37.517 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:37.511+0000 7f0419751700 1 -- 172.21.15.40:0/3864325393 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0414100070 msgr2=0x7f0414194ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:16:37.517 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:37.511+0000 7f0419751700 1 --2- 172.21.15.40:0/3864325393 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0414100070 0x7f0414194ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:37.517 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:37.511+0000 7f0419751700 1 -- 172.21.15.40:0/3864325393 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0414100980 msgr2=0x7f04141951e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:16:37.517 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:37.511+0000 7f0419751700 1 --2- 172.21.15.40:0/3864325393 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0414100980 0x7f04141951e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:37.518 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:37.511+0000 7f0419751700 1 -- 172.21.15.40:0/3864325393 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04040095f0 con 0x7f0414109620 2023-07-14T19:16:37.518 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:37.511+0000 7f0413fff700 1 --2- 172.21.15.40:0/3864325393 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0414100980 0x7f04141951e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T19:16:37.518 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:37.511+0000 7f0419751700 1 --2- 172.21.15.40:0/3864325393 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0414109620 0x7f04141985a0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f040c009fd0 tx=0x7f040c00e8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:16:37.518 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:37.511+0000 7f0411ffb700 1 -- 172.21.15.40:0/3864325393 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f040c009e70 con 0x7f0414109620 2023-07-14T19:16:37.518 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:37.511+0000 7f041b1b4700 1 -- 172.21.15.40:0/3864325393 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f04141991b0 con 0x7f0414109620 2023-07-14T19:16:37.518 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:37.511+0000 7f0411ffb700 1 -- 172.21.15.40:0/3864325393 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f040c004bb0 con 0x7f0414109620 2023-07-14T19:16:37.519 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:37.511+0000 7f0411ffb700 1 -- 172.21.15.40:0/3864325393 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f040c004890 con 0x7f0414109620 2023-07-14T19:16:37.519 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:37.511+0000 7f041b1b4700 1 -- 172.21.15.40:0/3864325393 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f04140ff130 con 0x7f0414109620 2023-07-14T19:16:37.519 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:37.512+0000 7f041b1b4700 1 -- 172.21.15.40:0/3864325393 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f041404e340 con 0x7f0414109620 2023-07-14T19:16:37.520 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:37.513+0000 7f0411ffb700 1 -- 172.21.15.40:0/3864325393 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7f040c003a00 con 0x7f0414109620 2023-07-14T19:16:37.520 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:37.514+0000 7f0411ffb700 1 --2- 172.21.15.40:0/3864325393 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f04000a11e0 0x7f04000a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:16:37.520 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:37.514+0000 7f0418f50700 1 --2- 172.21.15.40:0/3864325393 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f04000a11e0 0x7f04000a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:16:37.520 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:37.514+0000 7f0411ffb700 1 -- 172.21.15.40:0/3864325393 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f040c016070 con 0x7f0414109620 2023-07-14T19:16:37.521 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:37.514+0000 7f0418f50700 1 --2- 172.21.15.40:0/3864325393 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f04000a11e0 0x7f04000a3690 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f0404006010 tx=0x7f040400b260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:16:37.521 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:37.518+0000 7f0411ffb700 1 -- 172.21.15.40:0/3864325393 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f040c098040 con 0x7f0414109620 2023-07-14T19:16:37.632 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:37.630+0000 7f608ccee700 1 -- 172.21.15.40:0/2743527540 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6088103ee0 msgr2=0x7f608810ba60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:16:37.633 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:37.630+0000 7f608ccee700 1 --2- 172.21.15.40:0/2743527540 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6088103ee0 0x7f608810ba60 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f608000b0d0 tx=0x7f608000b3e0 comp rx=0 tx=0).stop 2023-07-14T19:16:37.633 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:37.631+0000 7f608ccee700 1 -- 172.21.15.40:0/2743527540 shutdown_connections 2023-07-14T19:16:37.633 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:37.631+0000 7f608ccee700 1 --2- 172.21.15.40:0/2743527540 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f60880fee40 0x7f60880ff210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:37.633 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:37.631+0000 7f608ccee700 1 --2- 172.21.15.40:0/2743527540 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6088103ee0 0x7f608810ba60 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:37.633 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:37.631+0000 7f608ccee700 1 --2- 172.21.15.40:0/2743527540 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f60880ff7e0 0x7f60881037b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:37.633 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:37.631+0000 7f608ccee700 1 -- 172.21.15.40:0/2743527540 >> 172.21.15.40:0/2743527540 conn(0x7f6088075ca0 msgr2=0x7f60880760a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:16:37.634 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:37.631+0000 7f608ccee700 1 -- 172.21.15.40:0/2743527540 shutdown_connections 2023-07-14T19:16:37.634 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:37.631+0000 7f608ccee700 1 -- 172.21.15.40:0/2743527540 wait complete. 2023-07-14T19:16:37.635 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:37.632+0000 7f608ccee700 1 Processor -- start 2023-07-14T19:16:37.635 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:37.632+0000 7f608ccee700 1 -- start start 2023-07-14T19:16:37.635 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:37.632+0000 7f608ccee700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f60880fee40 0x7f6088198080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:16:37.636 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:37.632+0000 7f608ccee700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f60880ff7e0 0x7f60881985c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:16:37.636 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:37.632+0000 7f608ccee700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6088103ee0 0x7f608819ca30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:16:37.636 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:37.632+0000 7f608ccee700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f608819d040 con 0x7f60880ff7e0 2023-07-14T19:16:37.636 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:37.632+0000 7f608ccee700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f608819d1b0 con 0x7f6088103ee0 2023-07-14T19:16:37.636 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:37.632+0000 7f608ccee700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f608819d370 con 0x7f60880fee40 2023-07-14T19:16:37.636 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:37.632+0000 7f6085d9b700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f60880ff7e0 0x7f60881985c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:16:37.636 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:37.632+0000 7f6085d9b700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f60880ff7e0 0x7f60881985c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:33384/0 (socket says 172.21.15.40:33384) 2023-07-14T19:16:37.637 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:37.632+0000 7f6085d9b700 1 -- 172.21.15.40:0/3223347982 learned_addr learned my addr 172.21.15.40:0/3223347982 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-07-14T19:16:37.637 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:37.632+0000 7f608659c700 1 --2- 172.21.15.40:0/3223347982 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f60880fee40 0x7f6088198080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:16:37.637 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:37.632+0000 7f6086d9d700 1 --2- 172.21.15.40:0/3223347982 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6088103ee0 0x7f608819ca30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:16:37.637 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:37.632+0000 7f6085d9b700 1 -- 172.21.15.40:0/3223347982 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f60880fee40 msgr2=0x7f6088198080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:16:37.637 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:37.632+0000 7f6085d9b700 1 --2- 172.21.15.40:0/3223347982 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f60880fee40 0x7f6088198080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:37.637 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:37.633+0000 7f6085d9b700 1 -- 172.21.15.40:0/3223347982 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6088103ee0 msgr2=0x7f608819ca30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:16:37.637 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:37.633+0000 7f6085d9b700 1 --2- 172.21.15.40:0/3223347982 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6088103ee0 0x7f608819ca30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:37.638 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:37.633+0000 7f6085d9b700 1 -- 172.21.15.40:0/3223347982 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6080009d00 con 0x7f60880ff7e0 2023-07-14T19:16:37.638 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:37.633+0000 7f6086d9d700 1 --2- 172.21.15.40:0/3223347982 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6088103ee0 0x7f608819ca30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T19:16:37.638 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:37.633+0000 7f6085d9b700 1 --2- 172.21.15.40:0/3223347982 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f60880ff7e0 0x7f60881985c0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f607c00d950 tx=0x7f607c00dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:16:37.638 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:37.633+0000 7f60777fe700 1 -- 172.21.15.40:0/3223347982 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f607c004220 con 0x7f60880ff7e0 2023-07-14T19:16:37.638 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:37.633+0000 7f608ccee700 1 -- 172.21.15.40:0/3223347982 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f608819d650 con 0x7f60880ff7e0 2023-07-14T19:16:37.638 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:37.633+0000 7f60777fe700 1 -- 172.21.15.40:0/3223347982 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f607c007620 con 0x7f60880ff7e0 2023-07-14T19:16:37.638 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:37.633+0000 7f60777fe700 1 -- 172.21.15.40:0/3223347982 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f607c005690 con 0x7f60880ff7e0 2023-07-14T19:16:37.638 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:37.633+0000 7f608ccee700 1 -- 172.21.15.40:0/3223347982 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f60881a4eb0 con 0x7f60880ff7e0 2023-07-14T19:16:37.639 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:37.634+0000 7f608ccee700 1 -- 172.21.15.40:0/3223347982 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f608804e340 con 0x7f60880ff7e0 2023-07-14T19:16:37.639 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:37.635+0000 7f60777fe700 1 -- 172.21.15.40:0/3223347982 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7f607c004750 con 0x7f60880ff7e0 2023-07-14T19:16:37.640 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:37.636+0000 7f60777fe700 1 --2- 172.21.15.40:0/3223347982 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f60700a11e0 0x7f60700a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:16:37.640 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:37.636+0000 7f608659c700 1 --2- 172.21.15.40:0/3223347982 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f60700a11e0 0x7f60700a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:16:37.640 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:37.636+0000 7f60777fe700 1 -- 172.21.15.40:0/3223347982 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f607c0ccb50 con 0x7f60880ff7e0 2023-07-14T19:16:37.640 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:37.636+0000 7f608659c700 1 --2- 172.21.15.40:0/3223347982 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f60700a11e0 0x7f60700a3690 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f6078009c40 tx=0x7f6078009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:16:37.641 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:37.639+0000 7f60777fe700 1 -- 172.21.15.40:0/3223347982 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f607c0cce60 con 0x7f60880ff7e0 2023-07-14T19:16:37.685 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/adc52f04-2279-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T19:16:37.732 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:37.730+0000 7f041b1b4700 1 -- 172.21.15.40:0/3864325393 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 9} v 0) v1 -- 0x7f04140ff350 con 0x7f0414109620 2023-07-14T19:16:37.734 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:37.733+0000 7f0411ffb700 1 -- 172.21.15.40:0/3864325393 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 9}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f040c004000 con 0x7f0414109620 2023-07-14T19:16:37.735 INFO:teuthology.orchestra.run.smithi040.stdout:274877906971 2023-07-14T19:16:37.737 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:37.735+0000 7f041b1b4700 1 -- 172.21.15.40:0/3864325393 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f04000a11e0 msgr2=0x7f04000a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:16:37.737 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:37.735+0000 7f041b1b4700 1 --2- 172.21.15.40:0/3864325393 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f04000a11e0 0x7f04000a3690 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f0404006010 tx=0x7f040400b260 comp rx=0 tx=0).stop 2023-07-14T19:16:37.738 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:37.735+0000 7f041b1b4700 1 -- 172.21.15.40:0/3864325393 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0414109620 msgr2=0x7f04141985a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:16:37.738 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:37.735+0000 7f041b1b4700 1 --2- 172.21.15.40:0/3864325393 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0414109620 0x7f04141985a0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f040c009fd0 tx=0x7f040c00e8e0 comp rx=0 tx=0).stop 2023-07-14T19:16:37.738 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:37.735+0000 7f041b1b4700 1 -- 172.21.15.40:0/3864325393 shutdown_connections 2023-07-14T19:16:37.738 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:37.735+0000 7f041b1b4700 1 --2- 172.21.15.40:0/3864325393 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f04000a11e0 0x7f04000a3690 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:37.739 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:37.735+0000 7f041b1b4700 1 --2- 172.21.15.40:0/3864325393 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0414100070 0x7f0414194ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:37.739 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:37.735+0000 7f041b1b4700 1 --2- 172.21.15.40:0/3864325393 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0414109620 0x7f04141985a0 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:37.739 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:37.735+0000 7f041b1b4700 1 --2- 172.21.15.40:0/3864325393 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0414100980 0x7f04141951e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:37.740 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:37.735+0000 7f041b1b4700 1 -- 172.21.15.40:0/3864325393 >> 172.21.15.40:0/3864325393 conn(0x7f0414074550 msgr2=0x7f04140fbf40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:16:37.740 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:37.735+0000 7f041b1b4700 1 -- 172.21.15.40:0/3864325393 shutdown_connections 2023-07-14T19:16:37.740 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:37.735+0000 7f041b1b4700 1 -- 172.21.15.40:0/3864325393 wait complete. 2023-07-14T19:16:37.855 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:37.853+0000 7f608ccee700 1 -- 172.21.15.40:0/3223347982 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7f60881a5140 con 0x7f60880ff7e0 2023-07-14T19:16:37.855 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:37.853+0000 7f60777fe700 1 -- 172.21.15.40:0/3223347982 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f607c016090 con 0x7f60880ff7e0 2023-07-14T19:16:37.856 INFO:teuthology.orchestra.run.smithi040.stdout:137438953521 2023-07-14T19:16:37.857 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:37.856+0000 7f608ccee700 1 -- 172.21.15.40:0/3223347982 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f60700a11e0 msgr2=0x7f60700a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:16:37.857 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:37.856+0000 7f608ccee700 1 --2- 172.21.15.40:0/3223347982 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f60700a11e0 0x7f60700a3690 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f6078009c40 tx=0x7f6078009260 comp rx=0 tx=0).stop 2023-07-14T19:16:37.858 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:37.856+0000 7f608ccee700 1 -- 172.21.15.40:0/3223347982 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f60880ff7e0 msgr2=0x7f60881985c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:16:37.858 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:37.856+0000 7f608ccee700 1 --2- 172.21.15.40:0/3223347982 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f60880ff7e0 0x7f60881985c0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f607c00d950 tx=0x7f607c00dc60 comp rx=0 tx=0).stop 2023-07-14T19:16:37.858 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:37.856+0000 7f608ccee700 1 -- 172.21.15.40:0/3223347982 shutdown_connections 2023-07-14T19:16:37.858 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:37.856+0000 7f608ccee700 1 --2- 172.21.15.40:0/3223347982 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f60700a11e0 0x7f60700a3690 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:37.858 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:37.856+0000 7f608ccee700 1 --2- 172.21.15.40:0/3223347982 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f60880fee40 0x7f6088198080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:37.858 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:37.856+0000 7f608ccee700 1 --2- 172.21.15.40:0/3223347982 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6088103ee0 0x7f608819ca30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:37.858 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:37.856+0000 7f608ccee700 1 --2- 172.21.15.40:0/3223347982 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f60880ff7e0 0x7f60881985c0 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:37.859 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:37.856+0000 7f608ccee700 1 -- 172.21.15.40:0/3223347982 >> 172.21.15.40:0/3223347982 conn(0x7f6088075ca0 msgr2=0x7f6088068990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:16:37.859 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:37.856+0000 7f608ccee700 1 -- 172.21.15.40:0/3223347982 shutdown_connections 2023-07-14T19:16:37.859 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:37.856+0000 7f608ccee700 1 -- 172.21.15.40:0/3223347982 wait complete. 2023-07-14T19:16:37.992 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:16:37 smithi130 ceph-mon[97832]: from='client.? 172.21.15.40:0/2784581362' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2023-07-14T19:16:38.146 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:38.112+0000 7f7eba1bd700 1 -- 172.21.15.40:0/3812147110 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7eb4103ee0 msgr2=0x7f7eb410ba60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:16:38.147 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:38.112+0000 7f7eba1bd700 1 --2- 172.21.15.40:0/3812147110 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7eb4103ee0 0x7f7eb410ba60 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f7eac0099c0 tx=0x7f7eac009cd0 comp rx=0 tx=0).stop 2023-07-14T19:16:38.147 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:38.112+0000 7f7eba1bd700 1 -- 172.21.15.40:0/3812147110 shutdown_connections 2023-07-14T19:16:38.147 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:38.112+0000 7f7eba1bd700 1 --2- 172.21.15.40:0/3812147110 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7eb4103ee0 0x7f7eb410ba60 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:38.147 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:38.112+0000 7f7eba1bd700 1 --2- 172.21.15.40:0/3812147110 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7eb40fee40 0x7f7eb40ff210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:38.147 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:38.112+0000 7f7eba1bd700 1 --2- 172.21.15.40:0/3812147110 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7eb40ff7e0 0x7f7eb41037b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:38.147 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:38.112+0000 7f7eba1bd700 1 -- 172.21.15.40:0/3812147110 >> 172.21.15.40:0/3812147110 conn(0x7f7eb4075ca0 msgr2=0x7f7eb40760a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:16:38.147 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:38.112+0000 7f7eba1bd700 1 -- 172.21.15.40:0/3812147110 shutdown_connections 2023-07-14T19:16:38.148 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:38.112+0000 7f7eba1bd700 1 -- 172.21.15.40:0/3812147110 wait complete. 2023-07-14T19:16:38.148 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:38.112+0000 7f7eba1bd700 1 Processor -- start 2023-07-14T19:16:38.148 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:38.113+0000 7f7eba1bd700 1 -- start start 2023-07-14T19:16:38.148 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:38.113+0000 7f7eba1bd700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7eb40fee40 0x7f7eb419a970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:16:38.148 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:38.113+0000 7f7eba1bd700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7eb40ff7e0 0x7f7eb419aeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:16:38.148 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:38.113+0000 7f7eba1bd700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7eb4103ee0 0x7f7eb41949f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:16:38.149 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:38.113+0000 7f7eba1bd700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7eb4195060 con 0x7f7eb40ff7e0 2023-07-14T19:16:38.149 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:38.113+0000 7f7eba1bd700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7eb41951d0 con 0x7f7eb4103ee0 2023-07-14T19:16:38.149 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:38.113+0000 7f7eba1bd700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7eb4195390 con 0x7f7eb40fee40 2023-07-14T19:16:38.149 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:38.113+0000 7f7eb2ffd700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7eb40ff7e0 0x7f7eb419aeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:16:38.149 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:38.113+0000 7f7eb2ffd700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7eb40ff7e0 0x7f7eb419aeb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:33398/0 (socket says 172.21.15.40:33398) 2023-07-14T19:16:38.149 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:38.113+0000 7f7eb2ffd700 1 -- 172.21.15.40:0/85037800 learned_addr learned my addr 172.21.15.40:0/85037800 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-07-14T19:16:38.150 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:38.113+0000 7f7eb37fe700 1 --2- 172.21.15.40:0/85037800 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7eb40fee40 0x7f7eb419a970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:16:38.150 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:38.113+0000 7f7eb2ffd700 1 -- 172.21.15.40:0/85037800 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7eb40fee40 msgr2=0x7f7eb419a970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:16:38.150 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:38.113+0000 7f7eb2ffd700 1 --2- 172.21.15.40:0/85037800 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7eb40fee40 0x7f7eb419a970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:38.150 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:38.113+0000 7f7eb2ffd700 1 -- 172.21.15.40:0/85037800 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7eb4103ee0 msgr2=0x7f7eb41949f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:16:38.150 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:38.113+0000 7f7eb2ffd700 1 --2- 172.21.15.40:0/85037800 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7eb4103ee0 0x7f7eb41949f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:38.150 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:38.113+0000 7f7eb2ffd700 1 -- 172.21.15.40:0/85037800 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7eac0095f0 con 0x7f7eb40ff7e0 2023-07-14T19:16:38.150 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:38.113+0000 7f7eb37fe700 1 --2- 172.21.15.40:0/85037800 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7eb40fee40 0x7f7eb419a970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-14T19:16:38.151 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:38.113+0000 7f7eb2ffd700 1 --2- 172.21.15.40:0/85037800 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7eb40ff7e0 0x7f7eb419aeb0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f7ea800c9e0 tx=0x7f7ea800cda0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:16:38.151 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:38.114+0000 7f7eb0ff9700 1 -- 172.21.15.40:0/85037800 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ea8007aa0 con 0x7f7eb40ff7e0 2023-07-14T19:16:38.151 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:38.114+0000 7f7eb0ff9700 1 -- 172.21.15.40:0/85037800 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f7ea8007c00 con 0x7f7eb40ff7e0 2023-07-14T19:16:38.151 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:38.114+0000 7f7eba1bd700 1 -- 172.21.15.40:0/85037800 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7eb4195670 con 0x7f7eb40ff7e0 2023-07-14T19:16:38.151 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:38.114+0000 7f7eb0ff9700 1 -- 172.21.15.40:0/85037800 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ea8018660 con 0x7f7eb40ff7e0 2023-07-14T19:16:38.151 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:38.114+0000 7f7eba1bd700 1 -- 172.21.15.40:0/85037800 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7eb41a1960 con 0x7f7eb40ff7e0 2023-07-14T19:16:38.151 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:38.115+0000 7f7eba1bd700 1 -- 172.21.15.40:0/85037800 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7eb404e340 con 0x7f7eb40ff7e0 2023-07-14T19:16:38.152 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:38.116+0000 7f7eb0ff9700 1 -- 172.21.15.40:0/85037800 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7f7ea800f450 con 0x7f7eb40ff7e0 2023-07-14T19:16:38.152 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:38.116+0000 7f7eb0ff9700 1 --2- 172.21.15.40:0/85037800 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f7e9c0a11e0 0x7f7e9c0a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:16:38.152 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:38.116+0000 7f7eb37fe700 1 --2- 172.21.15.40:0/85037800 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f7e9c0a11e0 0x7f7e9c0a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:16:38.152 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:38.116+0000 7f7eb0ff9700 1 -- 172.21.15.40:0/85037800 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f7ea80cca10 con 0x7f7eb40ff7e0 2023-07-14T19:16:38.152 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:38.117+0000 7f7eb37fe700 1 --2- 172.21.15.40:0/85037800 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f7e9c0a11e0 0x7f7e9c0a3690 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f7ea40060b0 tx=0x7f7ea40076a0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:16:38.152 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:38.120+0000 7f7eb0ff9700 1 -- 172.21.15.40:0/85037800 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f7ea80961b0 con 0x7f7eb40ff7e0 2023-07-14T19:16:38.221 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:16:37 smithi040 ceph-mon[96518]: from='client.? 172.21.15.40:0/2784581362' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2023-07-14T19:16:38.222 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:16:37 smithi191 ceph-mon[98311]: from='client.? 172.21.15.40:0/2784581362' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2023-07-14T19:16:38.257 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/adc52f04-2279-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T19:16:38.338 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:38.336+0000 7f7eba1bd700 1 -- 172.21.15.40:0/85037800 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f7eb41a1d40 con 0x7f7eb40ff7e0 2023-07-14T19:16:38.338 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:38.336+0000 7f7eb0ff9700 1 -- 172.21.15.40:0/85037800 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f7ea80997d0 con 0x7f7eb40ff7e0 2023-07-14T19:16:38.339 INFO:teuthology.orchestra.run.smithi040.stdout:55834574910 2023-07-14T19:16:38.340 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:38.338+0000 7f7eba1bd700 1 -- 172.21.15.40:0/85037800 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f7e9c0a11e0 msgr2=0x7f7e9c0a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:16:38.340 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:38.338+0000 7f7eba1bd700 1 --2- 172.21.15.40:0/85037800 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f7e9c0a11e0 0x7f7e9c0a3690 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f7ea40060b0 tx=0x7f7ea40076a0 comp rx=0 tx=0).stop 2023-07-14T19:16:38.341 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:38.338+0000 7f7eba1bd700 1 -- 172.21.15.40:0/85037800 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7eb40ff7e0 msgr2=0x7f7eb419aeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:16:38.341 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:38.338+0000 7f7eba1bd700 1 --2- 172.21.15.40:0/85037800 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7eb40ff7e0 0x7f7eb419aeb0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f7ea800c9e0 tx=0x7f7ea800cda0 comp rx=0 tx=0).stop 2023-07-14T19:16:38.341 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:38.338+0000 7f7eba1bd700 1 -- 172.21.15.40:0/85037800 shutdown_connections 2023-07-14T19:16:38.341 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:38.338+0000 7f7eba1bd700 1 --2- 172.21.15.40:0/85037800 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f7e9c0a11e0 0x7f7e9c0a3690 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:38.341 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:38.338+0000 7f7eba1bd700 1 --2- 172.21.15.40:0/85037800 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7eb40fee40 0x7f7eb419a970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:38.341 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:38.338+0000 7f7eba1bd700 1 --2- 172.21.15.40:0/85037800 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7eb4103ee0 0x7f7eb41949f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:38.342 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:38.338+0000 7f7eba1bd700 1 --2- 172.21.15.40:0/85037800 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7eb40ff7e0 0x7f7eb419aeb0 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:38.342 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:38.338+0000 7f7eba1bd700 1 -- 172.21.15.40:0/85037800 >> 172.21.15.40:0/85037800 conn(0x7f7eb4075ca0 msgr2=0x7f7eb4102790 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:16:38.342 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:38.339+0000 7f7eba1bd700 1 -- 172.21.15.40:0/85037800 shutdown_connections 2023-07-14T19:16:38.342 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:38.339+0000 7f7eba1bd700 1 -- 172.21.15.40:0/85037800 wait complete. 2023-07-14T19:16:38.676 INFO:tasks.cephadm.ceph_manager.ceph:need seq 214748364834 got 214748364835 for osd.7 2023-07-14T19:16:38.677 DEBUG:teuthology.parallel:result is None 2023-07-14T19:16:38.971 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:16:38 smithi040 ceph-mon[96518]: pgmap v274: 97 pgs: 97 active+clean; 638 KiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 255 B/s wr, 10 op/s 2023-07-14T19:16:38.971 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:16:38 smithi040 ceph-mon[96518]: from='client.? 172.21.15.40:0/3864325393' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2023-07-14T19:16:38.971 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:16:38 smithi040 ceph-mon[96518]: from='client.? 172.21.15.40:0/3223347982' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2023-07-14T19:16:38.971 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:16:38 smithi040 ceph-mon[96518]: from='client.? 172.21.15.40:0/85037800' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2023-07-14T19:16:38.992 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:16:38 smithi130 ceph-mon[97832]: pgmap v274: 97 pgs: 97 active+clean; 638 KiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 255 B/s wr, 10 op/s 2023-07-14T19:16:38.993 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:16:38 smithi130 ceph-mon[97832]: from='client.? 172.21.15.40:0/3864325393' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2023-07-14T19:16:38.993 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:16:38 smithi130 ceph-mon[97832]: from='client.? 172.21.15.40:0/3223347982' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2023-07-14T19:16:38.993 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:16:38 smithi130 ceph-mon[97832]: from='client.? 172.21.15.40:0/85037800' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2023-07-14T19:16:39.222 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:16:38 smithi191 ceph-mon[98311]: pgmap v274: 97 pgs: 97 active+clean; 638 KiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 255 B/s wr, 10 op/s 2023-07-14T19:16:39.222 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:16:38 smithi191 ceph-mon[98311]: from='client.? 172.21.15.40:0/3864325393' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2023-07-14T19:16:39.222 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:16:38 smithi191 ceph-mon[98311]: from='client.? 172.21.15.40:0/3223347982' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2023-07-14T19:16:39.222 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:16:38 smithi191 ceph-mon[98311]: from='client.? 172.21.15.40:0/85037800' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2023-07-14T19:16:39.351 INFO:tasks.cephadm.ceph_manager.ceph:need seq 274877906969 got 274877906971 for osd.9 2023-07-14T19:16:39.351 DEBUG:teuthology.parallel:result is None 2023-07-14T19:16:39.593 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834574908 got 55834574910 for osd.1 2023-07-14T19:16:39.593 DEBUG:teuthology.parallel:result is None 2023-07-14T19:16:39.706 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:39.538+0000 7f3b8c724700 1 -- 172.21.15.40:0/3519949704 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3b8410a2d0 msgr2=0x7f3b8410c6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:16:39.706 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:39.538+0000 7f3b8c724700 1 --2- 172.21.15.40:0/3519949704 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3b8410a2d0 0x7f3b8410c6f0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f3b7c00b0a0 tx=0x7f3b7c00b3b0 comp rx=0 tx=0).stop 2023-07-14T19:16:39.706 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:39.538+0000 7f3b8c724700 1 -- 172.21.15.40:0/3519949704 shutdown_connections 2023-07-14T19:16:39.707 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:39.538+0000 7f3b8c724700 1 --2- 172.21.15.40:0/3519949704 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3b8410a2d0 0x7f3b8410c6f0 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:39.707 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:39.538+0000 7f3b8c724700 1 --2- 172.21.15.40:0/3519949704 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3b84100ef0 0x7f3b841012c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:39.707 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:39.538+0000 7f3b8c724700 1 --2- 172.21.15.40:0/3519949704 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3b84101800 0x7f3b84109d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:39.707 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:39.538+0000 7f3b8c724700 1 -- 172.21.15.40:0/3519949704 >> 172.21.15.40:0/3519949704 conn(0x7f3b84075d80 msgr2=0x7f3b84076180 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:16:39.707 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:39.538+0000 7f3b8c724700 1 -- 172.21.15.40:0/3519949704 shutdown_connections 2023-07-14T19:16:39.707 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:39.538+0000 7f3b8c724700 1 -- 172.21.15.40:0/3519949704 wait complete. 2023-07-14T19:16:39.707 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:39.539+0000 7f3b8c724700 1 Processor -- start 2023-07-14T19:16:39.708 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:39.539+0000 7f3b8c724700 1 -- start start 2023-07-14T19:16:39.708 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:39.539+0000 7f3b8c724700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3b84100ef0 0x7f3b840feb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:16:39.708 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:39.539+0000 7f3b8c724700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3b84101800 0x7f3b840ff090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:16:39.708 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:39.539+0000 7f3b8c724700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3b8410a2d0 0x7f3b840ff610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:16:39.708 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:39.539+0000 7f3b8c724700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b841a4300 con 0x7f3b84101800 2023-07-14T19:16:39.708 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:39.539+0000 7f3b8c724700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b841a4470 con 0x7f3b8410a2d0 2023-07-14T19:16:39.708 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:39.539+0000 7f3b8c724700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b841a4630 con 0x7f3b84100ef0 2023-07-14T19:16:39.709 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:39.539+0000 7f3b89cbf700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3b84101800 0x7f3b840ff090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:16:39.709 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:39.539+0000 7f3b89cbf700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3b84101800 0x7f3b840ff090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:33424/0 (socket says 172.21.15.40:33424) 2023-07-14T19:16:39.709 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:39.539+0000 7f3b89cbf700 1 -- 172.21.15.40:0/4057827757 learned_addr learned my addr 172.21.15.40:0/4057827757 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-07-14T19:16:39.709 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:39.540+0000 7f3b8acc1700 1 --2- 172.21.15.40:0/4057827757 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3b8410a2d0 0x7f3b840ff610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:16:39.709 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:39.540+0000 7f3b8a4c0700 1 --2- 172.21.15.40:0/4057827757 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3b84100ef0 0x7f3b840feb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:16:39.709 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:39.540+0000 7f3b89cbf700 1 -- 172.21.15.40:0/4057827757 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3b84100ef0 msgr2=0x7f3b840feb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:16:39.709 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:39.540+0000 7f3b89cbf700 1 --2- 172.21.15.40:0/4057827757 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3b84100ef0 0x7f3b840feb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:39.710 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:39.540+0000 7f3b89cbf700 1 -- 172.21.15.40:0/4057827757 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3b8410a2d0 msgr2=0x7f3b840ff610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:16:39.710 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:39.540+0000 7f3b89cbf700 1 --2- 172.21.15.40:0/4057827757 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3b8410a2d0 0x7f3b840ff610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:39.710 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:39.540+0000 7f3b89cbf700 1 -- 172.21.15.40:0/4057827757 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b7c009d00 con 0x7f3b84101800 2023-07-14T19:16:39.710 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:39.540+0000 7f3b8a4c0700 1 --2- 172.21.15.40:0/4057827757 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3b84100ef0 0x7f3b840feb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T19:16:39.710 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:39.540+0000 7f3b89cbf700 1 --2- 172.21.15.40:0/4057827757 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3b84101800 0x7f3b840ff090 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f3b7400d980 tx=0x7f3b7400dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:16:39.711 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:39.540+0000 7f3b7b7fe700 1 -- 172.21.15.40:0/4057827757 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b74004220 con 0x7f3b84101800 2023-07-14T19:16:39.711 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:39.540+0000 7f3b8c724700 1 -- 172.21.15.40:0/4057827757 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3b841a4910 con 0x7f3b84101800 2023-07-14T19:16:39.711 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:39.540+0000 7f3b7b7fe700 1 -- 172.21.15.40:0/4057827757 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3b74007640 con 0x7f3b84101800 2023-07-14T19:16:39.711 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:39.540+0000 7f3b7b7fe700 1 -- 172.21.15.40:0/4057827757 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b74005610 con 0x7f3b84101800 2023-07-14T19:16:39.711 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:39.540+0000 7f3b8c724700 1 -- 172.21.15.40:0/4057827757 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3b841a4e60 con 0x7f3b84101800 2023-07-14T19:16:39.711 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:39.542+0000 7f3b8c724700 1 -- 172.21.15.40:0/4057827757 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3b8404e340 con 0x7f3b84101800 2023-07-14T19:16:39.711 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:39.544+0000 7f3b7b7fe700 1 -- 172.21.15.40:0/4057827757 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7f3b74004750 con 0x7f3b84101800 2023-07-14T19:16:39.712 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:39.545+0000 7f3b7b7fe700 1 --2- 172.21.15.40:0/4057827757 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f3b700a1260 0x7f3b700a3710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:16:39.712 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:39.545+0000 7f3b8a4c0700 1 --2- 172.21.15.40:0/4057827757 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f3b700a1260 0x7f3b700a3710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:16:39.712 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:39.545+0000 7f3b7b7fe700 1 -- 172.21.15.40:0/4057827757 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f3b740cccb0 con 0x7f3b84101800 2023-07-14T19:16:39.712 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:39.545+0000 7f3b8a4c0700 1 --2- 172.21.15.40:0/4057827757 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f3b700a1260 0x7f3b700a3710 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f3b80005950 tx=0x7f3b800058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:16:39.712 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:39.547+0000 7f3b7b7fe700 1 -- 172.21.15.40:0/4057827757 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f3b74096350 con 0x7f3b84101800 2023-07-14T19:16:39.731 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:39.507+0000 7effef89a700 1 -- 172.21.15.40:0/680368638 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7effe8100900 msgr2=0x7effe81048c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:16:39.731 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:39.507+0000 7effef89a700 1 --2- 172.21.15.40:0/680368638 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7effe8100900 0x7effe81048c0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7effd80099c0 tx=0x7effd8009cd0 comp rx=0 tx=0).stop 2023-07-14T19:16:39.731 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:39.507+0000 7effef89a700 1 -- 172.21.15.40:0/680368638 shutdown_connections 2023-07-14T19:16:39.731 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:39.507+0000 7effef89a700 1 --2- 172.21.15.40:0/680368638 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7effe8104ff0 0x7effe810cb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:39.732 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:39.507+0000 7effef89a700 1 --2- 172.21.15.40:0/680368638 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7effe80fff60 0x7effe8100330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:39.732 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:39.507+0000 7effef89a700 1 --2- 172.21.15.40:0/680368638 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7effe8100900 0x7effe81048c0 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:39.732 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:39.507+0000 7effef89a700 1 -- 172.21.15.40:0/680368638 >> 172.21.15.40:0/680368638 conn(0x7effe8077ca0 msgr2=0x7effe80780a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:16:39.732 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:39.507+0000 7effef89a700 1 -- 172.21.15.40:0/680368638 shutdown_connections 2023-07-14T19:16:39.732 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:39.507+0000 7effef89a700 1 -- 172.21.15.40:0/680368638 wait complete. 2023-07-14T19:16:39.732 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:39.508+0000 7effef89a700 1 Processor -- start 2023-07-14T19:16:39.732 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:39.508+0000 7effef89a700 1 -- start start 2023-07-14T19:16:39.733 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:39.508+0000 7effef89a700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7effe80fff60 0x7effe8072940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:16:39.733 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:39.508+0000 7effef89a700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7effe8100900 0x7effe8072e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:16:39.733 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:39.508+0000 7effef89a700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7effe8104ff0 0x7effe8073400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:16:39.733 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:39.508+0000 7effef89a700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effe81a4180 con 0x7effe80fff60 2023-07-14T19:16:39.733 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:39.509+0000 7effef89a700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effe81a42f0 con 0x7effe8104ff0 2023-07-14T19:16:39.733 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:39.509+0000 7effef89a700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effe81a44b0 con 0x7effe8100900 2023-07-14T19:16:39.734 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:39.509+0000 7effed636700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7effe80fff60 0x7effe8072940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:16:39.734 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:39.509+0000 7effed636700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7effe80fff60 0x7effe8072940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:33410/0 (socket says 172.21.15.40:33410) 2023-07-14T19:16:39.734 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:39.509+0000 7effed636700 1 -- 172.21.15.40:0/320746698 learned_addr learned my addr 172.21.15.40:0/320746698 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-07-14T19:16:39.734 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:39.509+0000 7effede37700 1 --2- 172.21.15.40:0/320746698 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7effe8104ff0 0x7effe8073400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:16:39.734 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:39.509+0000 7effece35700 1 --2- 172.21.15.40:0/320746698 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7effe8100900 0x7effe8072e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:16:39.734 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:39.509+0000 7effed636700 1 -- 172.21.15.40:0/320746698 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7effe8100900 msgr2=0x7effe8072e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:16:39.734 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:39.509+0000 7effed636700 1 --2- 172.21.15.40:0/320746698 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7effe8100900 0x7effe8072e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:39.734 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:39.509+0000 7effed636700 1 -- 172.21.15.40:0/320746698 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7effe8104ff0 msgr2=0x7effe8073400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:16:39.735 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:39.509+0000 7effed636700 1 --2- 172.21.15.40:0/320746698 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7effe8104ff0 0x7effe8073400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:39.735 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:39.509+0000 7effed636700 1 -- 172.21.15.40:0/320746698 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7effd80095f0 con 0x7effe80fff60 2023-07-14T19:16:39.735 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:39.509+0000 7effed636700 1 --2- 172.21.15.40:0/320746698 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7effe80fff60 0x7effe8072940 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7effe400d950 tx=0x7effe400dd10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:16:39.735 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:39.509+0000 7effde7fc700 1 -- 172.21.15.40:0/320746698 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effe4009b40 con 0x7effe80fff60 2023-07-14T19:16:39.735 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:39.509+0000 7effef89a700 1 -- 172.21.15.40:0/320746698 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7effe81a4790 con 0x7effe80fff60 2023-07-14T19:16:39.735 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:39.509+0000 7effde7fc700 1 -- 172.21.15.40:0/320746698 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7effe4009ca0 con 0x7effe80fff60 2023-07-14T19:16:39.735 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:39.509+0000 7effde7fc700 1 -- 172.21.15.40:0/320746698 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effe400f620 con 0x7effe80fff60 2023-07-14T19:16:39.736 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:39.509+0000 7effef89a700 1 -- 172.21.15.40:0/320746698 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7effe81a4c60 con 0x7effe80fff60 2023-07-14T19:16:39.736 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:39.510+0000 7effef89a700 1 -- 172.21.15.40:0/320746698 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7effe804e340 con 0x7effe80fff60 2023-07-14T19:16:39.736 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:39.511+0000 7effde7fc700 1 -- 172.21.15.40:0/320746698 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7effe4010460 con 0x7effe80fff60 2023-07-14T19:16:39.736 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:39.512+0000 7effde7fc700 1 --2- 172.21.15.40:0/320746698 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7effd40a11e0 0x7effd40a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:16:39.736 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:39.512+0000 7effde7fc700 1 -- 172.21.15.40:0/320746698 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7979+0+0 (secure 0 0 0) 0x7effe40ccc70 con 0x7effe80fff60 2023-07-14T19:16:39.736 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:39.512+0000 7effece35700 1 --2- 172.21.15.40:0/320746698 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7effd40a11e0 0x7effd40a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:16:39.736 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:39.513+0000 7effece35700 1 --2- 172.21.15.40:0/320746698 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7effd40a11e0 0x7effd40a3690 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7effd8000c00 tx=0x7effd8005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:16:39.737 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:39.516+0000 7effde7fc700 1 -- 172.21.15.40:0/320746698 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7effe4096410 con 0x7effe80fff60 2023-07-14T19:16:39.749 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:39.747+0000 7effef89a700 1 -- 172.21.15.40:0/320746698 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) v1 -- 0x7effe8073e70 con 0x7effe80fff60 2023-07-14T19:16:39.749 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:39.747+0000 7effde7fc700 1 -- 172.21.15.40:0/320746698 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7effe4016020 con 0x7effe80fff60 2023-07-14T19:16:39.749 INFO:teuthology.orchestra.run.smithi040.stdout:193273528361 2023-07-14T19:16:39.751 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:39.749+0000 7effef89a700 1 -- 172.21.15.40:0/320746698 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7effd40a11e0 msgr2=0x7effd40a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:16:39.751 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:39.749+0000 7effef89a700 1 --2- 172.21.15.40:0/320746698 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7effd40a11e0 0x7effd40a3690 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7effd8000c00 tx=0x7effd8005fb0 comp rx=0 tx=0).stop 2023-07-14T19:16:39.751 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:39.749+0000 7effef89a700 1 -- 172.21.15.40:0/320746698 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7effe80fff60 msgr2=0x7effe8072940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:16:39.752 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:39.749+0000 7effef89a700 1 --2- 172.21.15.40:0/320746698 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7effe80fff60 0x7effe8072940 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7effe400d950 tx=0x7effe400dd10 comp rx=0 tx=0).stop 2023-07-14T19:16:39.752 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:39.749+0000 7effef89a700 1 -- 172.21.15.40:0/320746698 shutdown_connections 2023-07-14T19:16:39.752 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:39.749+0000 7effef89a700 1 --2- 172.21.15.40:0/320746698 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7effd40a11e0 0x7effd40a3690 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:39.752 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:39.749+0000 7effef89a700 1 --2- 172.21.15.40:0/320746698 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7effe8104ff0 0x7effe8073400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:39.752 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:39.749+0000 7effef89a700 1 --2- 172.21.15.40:0/320746698 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7effe80fff60 0x7effe8072940 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:39.752 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:39.749+0000 7effef89a700 1 --2- 172.21.15.40:0/320746698 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7effe8100900 0x7effe8072e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:39.752 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:39.749+0000 7effef89a700 1 -- 172.21.15.40:0/320746698 >> 172.21.15.40:0/320746698 conn(0x7effe8077ca0 msgr2=0x7effe80fd8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:16:39.753 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:39.750+0000 7effef89a700 1 -- 172.21.15.40:0/320746698 shutdown_connections 2023-07-14T19:16:39.753 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:39.750+0000 7effef89a700 1 -- 172.21.15.40:0/320746698 wait complete. 2023-07-14T19:16:39.766 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:39.764+0000 7f3b8c724700 1 -- 172.21.15.40:0/4057827757 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7f3b841a50a0 con 0x7f3b84101800 2023-07-14T19:16:39.766 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:39.765+0000 7f3b7b7fe700 1 -- 172.21.15.40:0/4057827757 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f3b74016090 con 0x7f3b84101800 2023-07-14T19:16:39.767 INFO:teuthology.orchestra.run.smithi040.stdout:77309411387 2023-07-14T19:16:39.769 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:39.767+0000 7f3b8c724700 1 -- 172.21.15.40:0/4057827757 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f3b700a1260 msgr2=0x7f3b700a3710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:16:39.769 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:39.767+0000 7f3b8c724700 1 --2- 172.21.15.40:0/4057827757 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f3b700a1260 0x7f3b700a3710 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f3b80005950 tx=0x7f3b800058e0 comp rx=0 tx=0).stop 2023-07-14T19:16:39.769 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:39.767+0000 7f3b8c724700 1 -- 172.21.15.40:0/4057827757 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3b84101800 msgr2=0x7f3b840ff090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:16:39.769 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:39.767+0000 7f3b8c724700 1 --2- 172.21.15.40:0/4057827757 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3b84101800 0x7f3b840ff090 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f3b7400d980 tx=0x7f3b7400dc90 comp rx=0 tx=0).stop 2023-07-14T19:16:39.769 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:39.767+0000 7f3b8c724700 1 -- 172.21.15.40:0/4057827757 shutdown_connections 2023-07-14T19:16:39.769 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:39.767+0000 7f3b8c724700 1 --2- 172.21.15.40:0/4057827757 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f3b700a1260 0x7f3b700a3710 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:39.770 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:39.767+0000 7f3b8c724700 1 --2- 172.21.15.40:0/4057827757 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3b84100ef0 0x7f3b840feb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:39.770 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:39.767+0000 7f3b8c724700 1 --2- 172.21.15.40:0/4057827757 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3b8410a2d0 0x7f3b840ff610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:39.770 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:39.767+0000 7f3b8c724700 1 --2- 172.21.15.40:0/4057827757 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3b84101800 0x7f3b840ff090 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:39.770 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:39.767+0000 7f3b8c724700 1 -- 172.21.15.40:0/4057827757 >> 172.21.15.40:0/4057827757 conn(0x7f3b84075d80 msgr2=0x7f3b84103db0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:16:39.770 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:39.767+0000 7f3b8c724700 1 -- 172.21.15.40:0/4057827757 shutdown_connections 2023-07-14T19:16:39.770 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:39.767+0000 7f3b8c724700 1 -- 172.21.15.40:0/4057827757 wait complete. 2023-07-14T19:16:39.980 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/adc52f04-2279-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T19:16:40.007 INFO:tasks.cephadm.ceph_manager.ceph:need seq 137438953520 got 137438953521 for osd.4 2023-07-14T19:16:40.008 DEBUG:teuthology.parallel:result is None 2023-07-14T19:16:40.992 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:16:40 smithi130 ceph-mon[97832]: pgmap v275: 97 pgs: 97 active+clean; 638 KiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2023-07-14T19:16:40.992 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:16:40 smithi130 ceph-mon[97832]: from='client.? 172.21.15.40:0/320746698' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2023-07-14T19:16:40.993 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:16:40 smithi130 ceph-mon[97832]: from='client.? 172.21.15.40:0/4057827757' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2023-07-14T19:16:41.030 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:16:40 smithi040 ceph-mon[96518]: pgmap v275: 97 pgs: 97 active+clean; 638 KiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2023-07-14T19:16:41.030 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:16:40 smithi040 ceph-mon[96518]: from='client.? 172.21.15.40:0/320746698' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2023-07-14T19:16:41.030 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:16:40 smithi040 ceph-mon[96518]: from='client.? 172.21.15.40:0/4057827757' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2023-07-14T19:16:41.031 INFO:tasks.cephadm.ceph_manager.ceph:need seq 77309411384 got 77309411387 for osd.2 2023-07-14T19:16:41.031 DEBUG:teuthology.parallel:result is None 2023-07-14T19:16:41.193 INFO:tasks.cephadm.ceph_manager.ceph:need seq 193273528358 got 193273528361 for osd.6 2023-07-14T19:16:41.193 DEBUG:teuthology.parallel:result is None 2023-07-14T19:16:41.222 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:16:40 smithi191 ceph-mon[98311]: pgmap v275: 97 pgs: 97 active+clean; 638 KiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2023-07-14T19:16:41.222 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:16:40 smithi191 ceph-mon[98311]: from='client.? 172.21.15.40:0/320746698' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2023-07-14T19:16:41.222 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:16:40 smithi191 ceph-mon[98311]: from='client.? 172.21.15.40:0/4057827757' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2023-07-14T19:16:42.308 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:42.305+0000 7f8c61f69700 1 -- 172.21.15.40:0/105517154 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8c5c0fee40 msgr2=0x7f8c5c0ff210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:16:42.308 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:42.305+0000 7f8c61f69700 1 --2- 172.21.15.40:0/105517154 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8c5c0fee40 0x7f8c5c0ff210 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f8c4c0099c0 tx=0x7f8c4c009cd0 comp rx=0 tx=0).stop 2023-07-14T19:16:42.309 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:42.305+0000 7f8c61f69700 1 -- 172.21.15.40:0/105517154 shutdown_connections 2023-07-14T19:16:42.309 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:42.305+0000 7f8c61f69700 1 --2- 172.21.15.40:0/105517154 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8c5c103ee0 0x7f8c5c10ba60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:42.309 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:42.305+0000 7f8c61f69700 1 --2- 172.21.15.40:0/105517154 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8c5c0fee40 0x7f8c5c0ff210 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:42.309 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:42.305+0000 7f8c61f69700 1 --2- 172.21.15.40:0/105517154 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8c5c0ff7e0 0x7f8c5c1037b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:42.310 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:42.305+0000 7f8c61f69700 1 -- 172.21.15.40:0/105517154 >> 172.21.15.40:0/105517154 conn(0x7f8c5c075ca0 msgr2=0x7f8c5c0760a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:16:42.310 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:42.305+0000 7f8c61f69700 1 -- 172.21.15.40:0/105517154 shutdown_connections 2023-07-14T19:16:42.310 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:42.305+0000 7f8c61f69700 1 -- 172.21.15.40:0/105517154 wait complete. 2023-07-14T19:16:42.311 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:42.306+0000 7f8c61f69700 1 Processor -- start 2023-07-14T19:16:42.311 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:42.306+0000 7f8c61f69700 1 -- start start 2023-07-14T19:16:42.313 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:42.306+0000 7f8c61f69700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8c5c0fee40 0x7f8c5c196c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:16:42.313 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:42.306+0000 7f8c61f69700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8c5c0ff7e0 0x7f8c5c197170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:16:42.313 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:42.307+0000 7f8c61f69700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8c5c103ee0 0x7f8c5c1a1cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:16:42.314 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:42.307+0000 7f8c61f69700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c5c197ab0 con 0x7f8c5c0fee40 2023-07-14T19:16:42.314 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:42.307+0000 7f8c61f69700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c5c1a21f0 con 0x7f8c5c0ff7e0 2023-07-14T19:16:42.314 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:42.307+0000 7f8c61f69700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c5c1a23b0 con 0x7f8c5c103ee0 2023-07-14T19:16:42.314 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:42.307+0000 7f8c5b7fe700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8c5c0fee40 0x7f8c5c196c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:16:42.314 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:42.307+0000 7f8c5b7fe700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8c5c0fee40 0x7f8c5c196c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:33434/0 (socket says 172.21.15.40:33434) 2023-07-14T19:16:42.315 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:42.307+0000 7f8c5affd700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8c5c0ff7e0 0x7f8c5c197170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:16:42.315 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:42.307+0000 7f8c5b7fe700 1 -- 172.21.15.40:0/692356787 learned_addr learned my addr 172.21.15.40:0/692356787 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-07-14T19:16:42.315 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:42.307+0000 7f8c5bfff700 1 --2- 172.21.15.40:0/692356787 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8c5c103ee0 0x7f8c5c1a1cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:16:42.315 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:42.307+0000 7f8c5b7fe700 1 -- 172.21.15.40:0/692356787 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8c5c103ee0 msgr2=0x7f8c5c1a1cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:16:42.315 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:42.307+0000 7f8c5b7fe700 1 --2- 172.21.15.40:0/692356787 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8c5c103ee0 0x7f8c5c1a1cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:42.315 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:42.307+0000 7f8c5b7fe700 1 -- 172.21.15.40:0/692356787 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8c5c0ff7e0 msgr2=0x7f8c5c197170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:16:42.316 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:42.307+0000 7f8c5b7fe700 1 --2- 172.21.15.40:0/692356787 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8c5c0ff7e0 0x7f8c5c197170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:42.316 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:42.307+0000 7f8c5b7fe700 1 -- 172.21.15.40:0/692356787 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c4c0095f0 con 0x7f8c5c0fee40 2023-07-14T19:16:42.316 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:42.307+0000 7f8c5b7fe700 1 --2- 172.21.15.40:0/692356787 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8c5c0fee40 0x7f8c5c196c30 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f8c4c009990 tx=0x7f8c4c009580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:16:42.316 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:42.307+0000 7f8c58ff9700 1 -- 172.21.15.40:0/692356787 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c4c01d070 con 0x7f8c5c0fee40 2023-07-14T19:16:42.316 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:42.307+0000 7f8c58ff9700 1 -- 172.21.15.40:0/692356787 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f8c4c003f90 con 0x7f8c5c0fee40 2023-07-14T19:16:42.316 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:42.307+0000 7f8c61f69700 1 -- 172.21.15.40:0/692356787 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8c5c1a2630 con 0x7f8c5c0fee40 2023-07-14T19:16:42.316 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:42.307+0000 7f8c58ff9700 1 -- 172.21.15.40:0/692356787 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c4c022ce0 con 0x7f8c5c0fee40 2023-07-14T19:16:42.317 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:42.307+0000 7f8c61f69700 1 -- 172.21.15.40:0/692356787 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8c5c1a2a40 con 0x7f8c5c0fee40 2023-07-14T19:16:42.317 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:42.308+0000 7f8c61f69700 1 -- 172.21.15.40:0/692356787 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8c5c04e340 con 0x7f8c5c0fee40 2023-07-14T19:16:42.317 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:42.310+0000 7f8c58ff9700 1 -- 172.21.15.40:0/692356787 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7f8c4c022470 con 0x7f8c5c0fee40 2023-07-14T19:16:42.317 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:42.311+0000 7f8c58ff9700 1 --2- 172.21.15.40:0/692356787 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f8c440a11e0 0x7f8c440a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:16:42.317 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:42.311+0000 7f8c58ff9700 1 -- 172.21.15.40:0/692356787 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f8c4c0cdea0 con 0x7f8c5c0fee40 2023-07-14T19:16:42.317 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:42.311+0000 7f8c5affd700 1 --2- 172.21.15.40:0/692356787 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f8c440a11e0 0x7f8c440a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:16:42.317 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:42.311+0000 7f8c5affd700 1 --2- 172.21.15.40:0/692356787 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f8c440a11e0 0x7f8c440a3690 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f8c5c0689d0 tx=0x7f8c5000b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:16:42.318 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:42.314+0000 7f8c58ff9700 1 -- 172.21.15.40:0/692356787 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f8c4c0976d0 con 0x7f8c5c0fee40 2023-07-14T19:16:42.529 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:42.527+0000 7f8c61f69700 1 -- 172.21.15.40:0/692356787 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 8} v 0) v1 -- 0x7f8c5c068710 con 0x7f8c5c0fee40 2023-07-14T19:16:42.530 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:42.527+0000 7f8c58ff9700 1 -- 172.21.15.40:0/692356787 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 8}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f8c4c09acf0 con 0x7f8c5c0fee40 2023-07-14T19:16:42.530 INFO:teuthology.orchestra.run.smithi040.stdout:244813135905 2023-07-14T19:16:42.531 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:42.529+0000 7f8c61f69700 1 -- 172.21.15.40:0/692356787 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f8c440a11e0 msgr2=0x7f8c440a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:16:42.532 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:42.529+0000 7f8c61f69700 1 --2- 172.21.15.40:0/692356787 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f8c440a11e0 0x7f8c440a3690 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f8c5c0689d0 tx=0x7f8c5000b410 comp rx=0 tx=0).stop 2023-07-14T19:16:42.532 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:42.529+0000 7f8c61f69700 1 -- 172.21.15.40:0/692356787 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8c5c0fee40 msgr2=0x7f8c5c196c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:16:42.532 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:42.529+0000 7f8c61f69700 1 --2- 172.21.15.40:0/692356787 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8c5c0fee40 0x7f8c5c196c30 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f8c4c009990 tx=0x7f8c4c009580 comp rx=0 tx=0).stop 2023-07-14T19:16:42.532 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:42.529+0000 7f8c61f69700 1 -- 172.21.15.40:0/692356787 shutdown_connections 2023-07-14T19:16:42.532 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:42.530+0000 7f8c61f69700 1 --2- 172.21.15.40:0/692356787 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f8c440a11e0 0x7f8c440a3690 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:42.533 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:42.530+0000 7f8c61f69700 1 --2- 172.21.15.40:0/692356787 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8c5c103ee0 0x7f8c5c1a1cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:42.533 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:42.530+0000 7f8c61f69700 1 --2- 172.21.15.40:0/692356787 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8c5c0fee40 0x7f8c5c196c30 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:42.533 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:42.530+0000 7f8c61f69700 1 --2- 172.21.15.40:0/692356787 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8c5c0ff7e0 0x7f8c5c197170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:42.533 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:42.530+0000 7f8c61f69700 1 -- 172.21.15.40:0/692356787 >> 172.21.15.40:0/692356787 conn(0x7f8c5c075ca0 msgr2=0x7f8c5c0fc510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:16:42.533 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:42.530+0000 7f8c61f69700 1 -- 172.21.15.40:0/692356787 shutdown_connections 2023-07-14T19:16:42.533 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:42.530+0000 7f8c61f69700 1 -- 172.21.15.40:0/692356787 wait complete. 2023-07-14T19:16:42.733 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:42.731+0000 7fea86280700 1 -- 172.21.15.40:0/2026409 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fea800fee40 msgr2=0x7fea800ff210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:16:42.734 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:42.731+0000 7fea86280700 1 --2- 172.21.15.40:0/2026409 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fea800fee40 0x7fea800ff210 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7fea700099c0 tx=0x7fea70009cd0 comp rx=0 tx=0).stop 2023-07-14T19:16:42.734 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:42.731+0000 7fea86280700 1 -- 172.21.15.40:0/2026409 shutdown_connections 2023-07-14T19:16:42.734 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:42.731+0000 7fea86280700 1 --2- 172.21.15.40:0/2026409 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fea80103ee0 0x7fea8010ba60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:42.734 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:42.731+0000 7fea86280700 1 --2- 172.21.15.40:0/2026409 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fea800fee40 0x7fea800ff210 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:42.734 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:42.731+0000 7fea86280700 1 --2- 172.21.15.40:0/2026409 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fea800ff7e0 0x7fea801037b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:42.734 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:42.731+0000 7fea86280700 1 -- 172.21.15.40:0/2026409 >> 172.21.15.40:0/2026409 conn(0x7fea80075ca0 msgr2=0x7fea800760a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:16:42.735 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:42.732+0000 7fea86280700 1 -- 172.21.15.40:0/2026409 shutdown_connections 2023-07-14T19:16:42.735 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:42.732+0000 7fea86280700 1 -- 172.21.15.40:0/2026409 wait complete. 2023-07-14T19:16:42.736 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:42.732+0000 7fea86280700 1 Processor -- start 2023-07-14T19:16:42.736 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:42.732+0000 7fea86280700 1 -- start start 2023-07-14T19:16:42.737 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:42.733+0000 7fea86280700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fea800fee40 0x7fea8019aa50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:16:42.737 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:42.733+0000 7fea86280700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fea800ff7e0 0x7fea8019af90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:16:42.737 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:42.733+0000 7fea86280700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fea80103ee0 0x7fea80194ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:16:42.737 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:42.733+0000 7fea86280700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea80195140 con 0x7fea80103ee0 2023-07-14T19:16:42.737 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:42.733+0000 7fea86280700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea801952b0 con 0x7fea800fee40 2023-07-14T19:16:42.737 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:42.733+0000 7fea86280700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea80195470 con 0x7fea800ff7e0 2023-07-14T19:16:42.737 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:42.733+0000 7fea8481d700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fea80103ee0 0x7fea80194ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:16:42.738 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:42.733+0000 7fea8481d700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fea80103ee0 0x7fea80194ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:33452/0 (socket says 172.21.15.40:33452) 2023-07-14T19:16:42.738 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:42.733+0000 7fea8481d700 1 -- 172.21.15.40:0/3489437636 learned_addr learned my addr 172.21.15.40:0/3489437636 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-07-14T19:16:42.738 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:42.733+0000 7fea7ffff700 1 --2- 172.21.15.40:0/3489437636 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fea800fee40 0x7fea8019aa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:16:42.738 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:42.733+0000 7fea7f7fe700 1 --2- 172.21.15.40:0/3489437636 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fea800ff7e0 0x7fea8019af90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:16:42.738 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:42.733+0000 7fea8481d700 1 -- 172.21.15.40:0/3489437636 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fea800ff7e0 msgr2=0x7fea8019af90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:16:42.738 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:42.733+0000 7fea8481d700 1 --2- 172.21.15.40:0/3489437636 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fea800ff7e0 0x7fea8019af90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:42.738 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:42.733+0000 7fea8481d700 1 -- 172.21.15.40:0/3489437636 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fea800fee40 msgr2=0x7fea8019aa50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:16:42.739 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:42.733+0000 7fea8481d700 1 --2- 172.21.15.40:0/3489437636 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fea800fee40 0x7fea8019aa50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:42.739 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:42.733+0000 7fea8481d700 1 -- 172.21.15.40:0/3489437636 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea700095f0 con 0x7fea80103ee0 2023-07-14T19:16:42.739 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:42.734+0000 7fea8481d700 1 --2- 172.21.15.40:0/3489437636 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fea80103ee0 0x7fea80194ad0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fea78007f10 tx=0x7fea7800d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:16:42.739 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:42.734+0000 7fea7d7fa700 1 -- 172.21.15.40:0/3489437636 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea78017070 con 0x7fea80103ee0 2023-07-14T19:16:42.739 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:42.734+0000 7fea86280700 1 -- 172.21.15.40:0/3489437636 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fea80195750 con 0x7fea80103ee0 2023-07-14T19:16:42.739 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:42.734+0000 7fea7d7fa700 1 -- 172.21.15.40:0/3489437636 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fea7800f040 con 0x7fea80103ee0 2023-07-14T19:16:42.739 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:42.734+0000 7fea7d7fa700 1 -- 172.21.15.40:0/3489437636 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea78005740 con 0x7fea80103ee0 2023-07-14T19:16:42.739 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:42.734+0000 7fea86280700 1 -- 172.21.15.40:0/3489437636 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fea800682d0 con 0x7fea80103ee0 2023-07-14T19:16:42.740 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:42.735+0000 7fea86280700 1 -- 172.21.15.40:0/3489437636 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fea6c005320 con 0x7fea80103ee0 2023-07-14T19:16:42.740 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:42.736+0000 7fea7d7fa700 1 -- 172.21.15.40:0/3489437636 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7fea780058a0 con 0x7fea80103ee0 2023-07-14T19:16:42.740 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:42.737+0000 7fea7d7fa700 1 --2- 172.21.15.40:0/3489437636 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7fea680a11e0 0x7fea680a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:16:42.741 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:42.737+0000 7fea7ffff700 1 --2- 172.21.15.40:0/3489437636 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7fea680a11e0 0x7fea680a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:16:42.741 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:42.737+0000 7fea7d7fa700 1 -- 172.21.15.40:0/3489437636 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7979+0+0 (secure 0 0 0) 0x7fea780cbf20 con 0x7fea80103ee0 2023-07-14T19:16:42.741 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:42.737+0000 7fea7ffff700 1 --2- 172.21.15.40:0/3489437636 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7fea680a11e0 0x7fea680a3690 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fea70000c00 tx=0x7fea70003680 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:16:42.745 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:42.744+0000 7fea7d7fa700 1 -- 172.21.15.40:0/3489437636 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fea780956c0 con 0x7fea80103ee0 2023-07-14T19:16:42.832 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:16:42 smithi040 ceph-mon[96518]: pgmap v276: 97 pgs: 97 active+clean; 638 KiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 9 op/s 2023-07-14T19:16:42.833 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:16:42 smithi040 ceph-mon[96518]: from='client.? 172.21.15.40:0/692356787' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2023-07-14T19:16:42.950 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:42.948+0000 7fea86280700 1 -- 172.21.15.40:0/3489437636 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 10} v 0) v1 -- 0x7fea6c005cc0 con 0x7fea80103ee0 2023-07-14T19:16:42.950 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:42.948+0000 7fea7d7fa700 1 -- 172.21.15.40:0/3489437636 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 10}]=0 v0) v1 ==== 75+0+13 (secure 0 0 0) 0x7fea78098ce0 con 0x7fea80103ee0 2023-07-14T19:16:42.950 INFO:teuthology.orchestra.run.smithi040.stdout:296352743448 2023-07-14T19:16:42.952 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:42.950+0000 7fea86280700 1 -- 172.21.15.40:0/3489437636 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7fea680a11e0 msgr2=0x7fea680a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:16:42.952 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:42.950+0000 7fea86280700 1 --2- 172.21.15.40:0/3489437636 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7fea680a11e0 0x7fea680a3690 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fea70000c00 tx=0x7fea70003680 comp rx=0 tx=0).stop 2023-07-14T19:16:42.952 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:42.950+0000 7fea86280700 1 -- 172.21.15.40:0/3489437636 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fea80103ee0 msgr2=0x7fea80194ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:16:42.952 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:42.950+0000 7fea86280700 1 --2- 172.21.15.40:0/3489437636 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fea80103ee0 0x7fea80194ad0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fea78007f10 tx=0x7fea7800d3b0 comp rx=0 tx=0).stop 2023-07-14T19:16:42.953 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:42.950+0000 7fea86280700 1 -- 172.21.15.40:0/3489437636 shutdown_connections 2023-07-14T19:16:42.953 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:42.950+0000 7fea86280700 1 --2- 172.21.15.40:0/3489437636 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7fea680a11e0 0x7fea680a3690 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:42.953 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:42.950+0000 7fea86280700 1 --2- 172.21.15.40:0/3489437636 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fea80103ee0 0x7fea80194ad0 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:42.953 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:42.950+0000 7fea86280700 1 --2- 172.21.15.40:0/3489437636 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fea800fee40 0x7fea8019aa50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:42.953 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:42.950+0000 7fea86280700 1 --2- 172.21.15.40:0/3489437636 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fea800ff7e0 0x7fea8019af90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:42.953 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:42.950+0000 7fea86280700 1 -- 172.21.15.40:0/3489437636 >> 172.21.15.40:0/3489437636 conn(0x7fea80075ca0 msgr2=0x7fea80102790 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:16:42.953 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:42.950+0000 7fea86280700 1 -- 172.21.15.40:0/3489437636 shutdown_connections 2023-07-14T19:16:42.954 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:42.950+0000 7fea86280700 1 -- 172.21.15.40:0/3489437636 wait complete. 2023-07-14T19:16:42.992 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:16:42 smithi130 ceph-mon[97832]: pgmap v276: 97 pgs: 97 active+clean; 638 KiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 9 op/s 2023-07-14T19:16:42.992 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:16:42 smithi130 ceph-mon[97832]: from='client.? 172.21.15.40:0/692356787' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2023-07-14T19:16:43.222 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:16:42 smithi191 ceph-mon[98311]: pgmap v276: 97 pgs: 97 active+clean; 638 KiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 9 op/s 2023-07-14T19:16:43.222 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:16:42 smithi191 ceph-mon[98311]: from='client.? 172.21.15.40:0/692356787' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2023-07-14T19:16:43.280 INFO:tasks.cephadm.ceph_manager.ceph:need seq 244813135902 got 244813135905 for osd.8 2023-07-14T19:16:43.281 DEBUG:teuthology.parallel:result is None 2023-07-14T19:16:43.326 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:43.324+0000 7fce163ad700 1 -- 172.21.15.40:0/3048301288 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fce100fffb0 msgr2=0x7fce10100380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:16:43.326 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:43.324+0000 7fce163ad700 1 --2- 172.21.15.40:0/3048301288 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fce100fffb0 0x7fce10100380 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7fce000099c0 tx=0x7fce00009cd0 comp rx=0 tx=0).stop 2023-07-14T19:16:43.326 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:43.324+0000 7fce163ad700 1 -- 172.21.15.40:0/3048301288 shutdown_connections 2023-07-14T19:16:43.326 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:43.324+0000 7fce163ad700 1 --2- 172.21.15.40:0/3048301288 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fce10105000 0x7fce1010cb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:43.327 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:43.324+0000 7fce163ad700 1 --2- 172.21.15.40:0/3048301288 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fce100fffb0 0x7fce10100380 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:43.327 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:43.324+0000 7fce163ad700 1 --2- 172.21.15.40:0/3048301288 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fce10100950 0x7fce101048d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:43.327 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:43.324+0000 7fce163ad700 1 -- 172.21.15.40:0/3048301288 >> 172.21.15.40:0/3048301288 conn(0x7fce10077cd0 msgr2=0x7fce100780d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:16:43.327 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:43.324+0000 7fce163ad700 1 -- 172.21.15.40:0/3048301288 shutdown_connections 2023-07-14T19:16:43.327 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:43.324+0000 7fce163ad700 1 -- 172.21.15.40:0/3048301288 wait complete. 2023-07-14T19:16:43.329 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:43.325+0000 7fce163ad700 1 Processor -- start 2023-07-14T19:16:43.329 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:43.325+0000 7fce163ad700 1 -- start start 2023-07-14T19:16:43.329 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:43.325+0000 7fce163ad700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fce100fffb0 0x7fce10195d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:16:43.329 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:43.325+0000 7fce163ad700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fce10100950 0x7fce10196280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:16:43.329 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:43.325+0000 7fce163ad700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fce10105000 0x7fce1019a6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:16:43.329 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:43.325+0000 7fce163ad700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce1019ad00 con 0x7fce10100950 2023-07-14T19:16:43.330 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:43.325+0000 7fce163ad700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce1019ae70 con 0x7fce10105000 2023-07-14T19:16:43.330 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:43.325+0000 7fce163ad700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce1019b030 con 0x7fce100fffb0 2023-07-14T19:16:43.330 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:43.325+0000 7fce0f7fe700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fce10100950 0x7fce10196280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:16:43.330 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:43.325+0000 7fce0f7fe700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fce10100950 0x7fce10196280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:33464/0 (socket says 172.21.15.40:33464) 2023-07-14T19:16:43.330 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:43.325+0000 7fce0f7fe700 1 -- 172.21.15.40:0/3291437336 learned_addr learned my addr 172.21.15.40:0/3291437336 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-07-14T19:16:43.330 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:43.325+0000 7fce0ffff700 1 --2- 172.21.15.40:0/3291437336 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fce100fffb0 0x7fce10195d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:16:43.330 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:43.326+0000 7fce1494a700 1 --2- 172.21.15.40:0/3291437336 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fce10105000 0x7fce1019a6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:16:43.331 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:43.326+0000 7fce0f7fe700 1 -- 172.21.15.40:0/3291437336 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fce100fffb0 msgr2=0x7fce10195d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:16:43.331 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:43.326+0000 7fce0f7fe700 1 --2- 172.21.15.40:0/3291437336 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fce100fffb0 0x7fce10195d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:43.331 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:43.326+0000 7fce0f7fe700 1 -- 172.21.15.40:0/3291437336 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fce10105000 msgr2=0x7fce1019a6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:16:43.331 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:43.326+0000 7fce0f7fe700 1 --2- 172.21.15.40:0/3291437336 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fce10105000 0x7fce1019a6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:43.331 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:43.326+0000 7fce0f7fe700 1 -- 172.21.15.40:0/3291437336 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce000095f0 con 0x7fce10100950 2023-07-14T19:16:43.331 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:43.326+0000 7fce0f7fe700 1 --2- 172.21.15.40:0/3291437336 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fce10100950 0x7fce10196280 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7fce0400d980 tx=0x7fce0400dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:16:43.331 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:43.326+0000 7fce0d7fa700 1 -- 172.21.15.40:0/3291437336 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce04009930 con 0x7fce10100950 2023-07-14T19:16:43.331 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:43.326+0000 7fce163ad700 1 -- 172.21.15.40:0/3291437336 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fce1019b310 con 0x7fce10100950 2023-07-14T19:16:43.332 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:43.326+0000 7fce0d7fa700 1 -- 172.21.15.40:0/3291437336 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fce04010460 con 0x7fce10100950 2023-07-14T19:16:43.332 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:43.326+0000 7fce0d7fa700 1 -- 172.21.15.40:0/3291437336 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce0400f620 con 0x7fce10100950 2023-07-14T19:16:43.332 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:43.326+0000 7fce163ad700 1 -- 172.21.15.40:0/3291437336 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fce101a2b70 con 0x7fce10100950 2023-07-14T19:16:43.333 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:43.328+0000 7fce163ad700 1 -- 172.21.15.40:0/3291437336 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fce1004e340 con 0x7fce10100950 2023-07-14T19:16:43.333 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:43.328+0000 7fce0d7fa700 1 -- 172.21.15.40:0/3291437336 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7fce040108d0 con 0x7fce10100950 2023-07-14T19:16:43.333 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:43.329+0000 7fce0d7fa700 1 --2- 172.21.15.40:0/3291437336 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7fcdf80a11e0 0x7fcdf80a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:16:43.333 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:43.329+0000 7fce0d7fa700 1 -- 172.21.15.40:0/3291437336 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7979+0+0 (secure 0 0 0) 0x7fce040cc3a0 con 0x7fce10100950 2023-07-14T19:16:43.333 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:43.329+0000 7fce0ffff700 1 --2- 172.21.15.40:0/3291437336 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7fcdf80a11e0 0x7fcdf80a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:16:43.333 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:43.330+0000 7fce0ffff700 1 --2- 172.21.15.40:0/3291437336 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7fcdf80a11e0 0x7fcdf80a3690 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fce00000c00 tx=0x7fce00003680 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:16:43.335 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:43.334+0000 7fce0d7fa700 1 -- 172.21.15.40:0/3291437336 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fce04095b40 con 0x7fce10100950 2023-07-14T19:16:43.393 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:43.391+0000 7f37d58db700 1 -- 172.21.15.40:0/2193983054 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f37d00feab0 msgr2=0x7f37d00fee80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:16:43.393 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:43.391+0000 7f37d58db700 1 --2- 172.21.15.40:0/2193983054 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f37d00feab0 0x7f37d00fee80 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f37c0009990 tx=0x7f37c0009ca0 comp rx=0 tx=0).stop 2023-07-14T19:16:43.394 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:43.391+0000 7f37d58db700 1 -- 172.21.15.40:0/2193983054 shutdown_connections 2023-07-14T19:16:43.394 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:43.391+0000 7f37d58db700 1 --2- 172.21.15.40:0/2193983054 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f37d010c660 0x7f37d010ea40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:43.394 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:43.391+0000 7f37d58db700 1 --2- 172.21.15.40:0/2193983054 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f37d00feab0 0x7f37d00fee80 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:43.394 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:43.391+0000 7f37d58db700 1 --2- 172.21.15.40:0/2193983054 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f37d00ff3c0 0x7f37d010c120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:43.394 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:43.391+0000 7f37d58db700 1 -- 172.21.15.40:0/2193983054 >> 172.21.15.40:0/2193983054 conn(0x7f37d0075ca0 msgr2=0x7f37d00760a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:16:43.394 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:43.391+0000 7f37d58db700 1 -- 172.21.15.40:0/2193983054 shutdown_connections 2023-07-14T19:16:43.394 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:43.391+0000 7f37d58db700 1 -- 172.21.15.40:0/2193983054 wait complete. 2023-07-14T19:16:43.396 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:43.392+0000 7f37d58db700 1 Processor -- start 2023-07-14T19:16:43.396 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:43.392+0000 7f37d58db700 1 -- start start 2023-07-14T19:16:43.396 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:43.392+0000 7f37d58db700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f37d00feab0 0x7f37d0197f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:16:43.396 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:43.392+0000 7f37d58db700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f37d00ff3c0 0x7f37d0198440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:16:43.396 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:43.392+0000 7f37d58db700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f37d010c660 0x7f37d019c8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:16:43.397 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:43.392+0000 7f37d58db700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37d019cec0 con 0x7f37d00feab0 2023-07-14T19:16:43.397 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:43.392+0000 7f37d58db700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37d019d030 con 0x7f37d00ff3c0 2023-07-14T19:16:43.397 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:43.392+0000 7f37d58db700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37d019d1f0 con 0x7f37d010c660 2023-07-14T19:16:43.397 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:43.392+0000 7f37ceffd700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f37d00feab0 0x7f37d0197f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:16:43.397 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:43.392+0000 7f37ceffd700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f37d00feab0 0x7f37d0197f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:33476/0 (socket says 172.21.15.40:33476) 2023-07-14T19:16:43.397 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:43.392+0000 7f37ceffd700 1 -- 172.21.15.40:0/4073566768 learned_addr learned my addr 172.21.15.40:0/4073566768 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-07-14T19:16:43.397 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:43.393+0000 7f37cf7fe700 1 --2- 172.21.15.40:0/4073566768 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f37d010c660 0x7f37d019c8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:16:43.398 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:43.393+0000 7f37ce7fc700 1 --2- 172.21.15.40:0/4073566768 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f37d00ff3c0 0x7f37d0198440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:16:43.398 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:43.393+0000 7f37ceffd700 1 -- 172.21.15.40:0/4073566768 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f37d010c660 msgr2=0x7f37d019c8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:16:43.398 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:43.393+0000 7f37ceffd700 1 --2- 172.21.15.40:0/4073566768 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f37d010c660 0x7f37d019c8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:43.398 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:43.393+0000 7f37ceffd700 1 -- 172.21.15.40:0/4073566768 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f37d00ff3c0 msgr2=0x7f37d0198440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:16:43.398 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:43.393+0000 7f37ceffd700 1 --2- 172.21.15.40:0/4073566768 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f37d00ff3c0 0x7f37d0198440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:43.398 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:43.393+0000 7f37ceffd700 1 -- 172.21.15.40:0/4073566768 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37c00095f0 con 0x7f37d00feab0 2023-07-14T19:16:43.398 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:43.393+0000 7f37ceffd700 1 --2- 172.21.15.40:0/4073566768 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f37d00feab0 0x7f37d0197f00 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f37c00095c0 tx=0x7f37c000f710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:16:43.399 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:43.393+0000 7f37d48d9700 1 -- 172.21.15.40:0/4073566768 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37c001d070 con 0x7f37d00feab0 2023-07-14T19:16:43.399 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:43.393+0000 7f37d58db700 1 -- 172.21.15.40:0/4073566768 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f37d019d470 con 0x7f37d00feab0 2023-07-14T19:16:43.399 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:43.393+0000 7f37d48d9700 1 -- 172.21.15.40:0/4073566768 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f37c000fb50 con 0x7f37d00feab0 2023-07-14T19:16:43.399 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:43.393+0000 7f37d48d9700 1 -- 172.21.15.40:0/4073566768 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37c00173f0 con 0x7f37d00feab0 2023-07-14T19:16:43.399 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:43.393+0000 7f37d58db700 1 -- 172.21.15.40:0/4073566768 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f37d01a4d30 con 0x7f37d00feab0 2023-07-14T19:16:43.399 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:43.394+0000 7f37d58db700 1 -- 172.21.15.40:0/4073566768 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f37d004e340 con 0x7f37d00feab0 2023-07-14T19:16:43.400 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:43.395+0000 7f37d48d9700 1 -- 172.21.15.40:0/4073566768 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7f37c000fcc0 con 0x7f37d00feab0 2023-07-14T19:16:43.400 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:43.396+0000 7f37d48d9700 1 --2- 172.21.15.40:0/4073566768 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f37b80a11e0 0x7f37b80a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:16:43.400 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:43.396+0000 7f37d48d9700 1 -- 172.21.15.40:0/4073566768 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f37c00cdbe0 con 0x7f37d00feab0 2023-07-14T19:16:43.400 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:43.396+0000 7f37ce7fc700 1 --2- 172.21.15.40:0/4073566768 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f37b80a11e0 0x7f37b80a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:16:43.401 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:43.396+0000 7f37ce7fc700 1 --2- 172.21.15.40:0/4073566768 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f37b80a11e0 0x7f37b80a3690 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f37c4005ea0 tx=0x7f37c4005e30 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:16:43.401 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:43.399+0000 7f37d48d9700 1 -- 172.21.15.40:0/4073566768 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f37c0097410 con 0x7f37d00feab0 2023-07-14T19:16:43.551 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:43.550+0000 7fce163ad700 1 -- 172.21.15.40:0/3291437336 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7fce101a2d00 con 0x7fce10100950 2023-07-14T19:16:43.551 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:43.551+0000 7fce0d7fa700 1 -- 172.21.15.40:0/3291437336 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7fce04099160 con 0x7fce10100950 2023-07-14T19:16:43.552 INFO:teuthology.orchestra.run.smithi040.stdout:158913789998 2023-07-14T19:16:43.554 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:43.553+0000 7fce163ad700 1 -- 172.21.15.40:0/3291437336 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7fcdf80a11e0 msgr2=0x7fcdf80a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:16:43.554 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:43.553+0000 7fce163ad700 1 --2- 172.21.15.40:0/3291437336 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7fcdf80a11e0 0x7fcdf80a3690 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fce00000c00 tx=0x7fce00003680 comp rx=0 tx=0).stop 2023-07-14T19:16:43.554 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:43.553+0000 7fce163ad700 1 -- 172.21.15.40:0/3291437336 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fce10100950 msgr2=0x7fce10196280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:16:43.554 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:43.553+0000 7fce163ad700 1 --2- 172.21.15.40:0/3291437336 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fce10100950 0x7fce10196280 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7fce0400d980 tx=0x7fce0400dc90 comp rx=0 tx=0).stop 2023-07-14T19:16:43.554 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:43.553+0000 7fce163ad700 1 -- 172.21.15.40:0/3291437336 shutdown_connections 2023-07-14T19:16:43.554 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:43.553+0000 7fce163ad700 1 --2- 172.21.15.40:0/3291437336 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7fcdf80a11e0 0x7fcdf80a3690 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:43.555 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:43.553+0000 7fce163ad700 1 --2- 172.21.15.40:0/3291437336 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fce100fffb0 0x7fce10195d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:43.555 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:43.553+0000 7fce163ad700 1 --2- 172.21.15.40:0/3291437336 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fce10105000 0x7fce1019a6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:43.555 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:43.553+0000 7fce163ad700 1 --2- 172.21.15.40:0/3291437336 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fce10100950 0x7fce10196280 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:43.555 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:43.553+0000 7fce163ad700 1 -- 172.21.15.40:0/3291437336 >> 172.21.15.40:0/3291437336 conn(0x7fce10077cd0 msgr2=0x7fce100fe870 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:16:43.555 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:43.553+0000 7fce163ad700 1 -- 172.21.15.40:0/3291437336 shutdown_connections 2023-07-14T19:16:43.555 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:43.553+0000 7fce163ad700 1 -- 172.21.15.40:0/3291437336 wait complete. 2023-07-14T19:16:43.601 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:43.600+0000 7f37d58db700 1 -- 172.21.15.40:0/4073566768 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7f37d01a4ec0 con 0x7f37d00feab0 2023-07-14T19:16:43.601 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:43.600+0000 7f37d48d9700 1 -- 172.21.15.40:0/4073566768 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f37c009aa30 con 0x7f37d00feab0 2023-07-14T19:16:43.602 INFO:teuthology.orchestra.run.smithi040.stdout:107374182455 2023-07-14T19:16:43.604 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:43.603+0000 7f37d58db700 1 -- 172.21.15.40:0/4073566768 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f37b80a11e0 msgr2=0x7f37b80a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:16:43.604 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:43.603+0000 7f37d58db700 1 --2- 172.21.15.40:0/4073566768 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f37b80a11e0 0x7f37b80a3690 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f37c4005ea0 tx=0x7f37c4005e30 comp rx=0 tx=0).stop 2023-07-14T19:16:43.604 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:43.603+0000 7f37d58db700 1 -- 172.21.15.40:0/4073566768 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f37d00feab0 msgr2=0x7f37d0197f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:16:43.604 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:43.603+0000 7f37d58db700 1 --2- 172.21.15.40:0/4073566768 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f37d00feab0 0x7f37d0197f00 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f37c00095c0 tx=0x7f37c000f710 comp rx=0 tx=0).stop 2023-07-14T19:16:43.605 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:43.603+0000 7f37d58db700 1 -- 172.21.15.40:0/4073566768 shutdown_connections 2023-07-14T19:16:43.605 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:43.603+0000 7f37d58db700 1 --2- 172.21.15.40:0/4073566768 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f37b80a11e0 0x7f37b80a3690 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:43.605 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:43.603+0000 7f37d58db700 1 --2- 172.21.15.40:0/4073566768 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f37d010c660 0x7f37d019c8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:43.605 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:43.603+0000 7f37d58db700 1 --2- 172.21.15.40:0/4073566768 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f37d00feab0 0x7f37d0197f00 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:43.605 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:43.603+0000 7f37d58db700 1 --2- 172.21.15.40:0/4073566768 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f37d00ff3c0 0x7f37d0198440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:43.605 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:43.603+0000 7f37d58db700 1 -- 172.21.15.40:0/4073566768 >> 172.21.15.40:0/4073566768 conn(0x7f37d0075ca0 msgr2=0x7f37d00fc4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:16:43.605 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:43.603+0000 7f37d58db700 1 -- 172.21.15.40:0/4073566768 shutdown_connections 2023-07-14T19:16:43.606 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:43.603+0000 7f37d58db700 1 -- 172.21.15.40:0/4073566768 wait complete. 2023-07-14T19:16:43.790 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:16:43 smithi040 ceph-mon[96518]: from='client.? 172.21.15.40:0/3489437636' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2023-07-14T19:16:43.791 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:16:43 smithi040 ceph-mon[96518]: from='client.? 172.21.15.40:0/3291437336' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2023-07-14T19:16:43.791 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:16:43 smithi040 ceph-mon[96518]: from='client.? 172.21.15.40:0/4073566768' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2023-07-14T19:16:43.892 INFO:tasks.cephadm.ceph_manager.ceph:need seq 296352743445 got 296352743448 for osd.10 2023-07-14T19:16:43.893 DEBUG:teuthology.parallel:result is None 2023-07-14T19:16:44.222 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:16:43 smithi191 ceph-mon[98311]: from='client.? 172.21.15.40:0/3489437636' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2023-07-14T19:16:44.222 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:16:43 smithi191 ceph-mon[98311]: from='client.? 172.21.15.40:0/3291437336' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2023-07-14T19:16:44.222 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:16:43 smithi191 ceph-mon[98311]: from='client.? 172.21.15.40:0/4073566768' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2023-07-14T19:16:44.242 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:16:43 smithi130 ceph-mon[97832]: from='client.? 172.21.15.40:0/3489437636' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2023-07-14T19:16:44.242 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:16:43 smithi130 ceph-mon[97832]: from='client.? 172.21.15.40:0/3291437336' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2023-07-14T19:16:44.243 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:16:43 smithi130 ceph-mon[97832]: from='client.? 172.21.15.40:0/4073566768' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2023-07-14T19:16:44.410 INFO:tasks.cephadm.ceph_manager.ceph:need seq 107374182452 got 107374182455 for osd.3 2023-07-14T19:16:44.411 DEBUG:teuthology.parallel:result is None 2023-07-14T19:16:44.484 INFO:tasks.cephadm.ceph_manager.ceph:need seq 158913789996 got 158913789998 for osd.5 2023-07-14T19:16:44.485 DEBUG:teuthology.parallel:result is None 2023-07-14T19:16:44.485 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2023-07-14T19:16:44.486 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell --fsid adc52f04-2279-11ee-9b33-001a4aab830c -- ceph pg dump --format=json 2023-07-14T19:16:45.049 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/adc52f04-2279-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T19:16:45.221 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:16:44 smithi040 ceph-mon[96518]: pgmap v277: 97 pgs: 97 active+clean; 638 KiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2023-07-14T19:16:45.221 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:16:44 smithi191 ceph-mon[98311]: pgmap v277: 97 pgs: 97 active+clean; 638 KiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2023-07-14T19:16:45.242 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:16:44 smithi130 ceph-mon[97832]: pgmap v277: 97 pgs: 97 active+clean; 638 KiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2023-07-14T19:16:46.409 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:46.406+0000 7f9bc62a7700 1 -- 172.21.15.40:0/275931145 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9bc0103ee0 msgr2=0x7f9bc010ba60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:16:46.409 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:46.406+0000 7f9bc62a7700 1 --2- 172.21.15.40:0/275931145 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9bc0103ee0 0x7f9bc010ba60 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f9bb800b0d0 tx=0x7f9bb800b3e0 comp rx=0 tx=0).stop 2023-07-14T19:16:46.409 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:46.406+0000 7f9bc62a7700 1 -- 172.21.15.40:0/275931145 shutdown_connections 2023-07-14T19:16:46.410 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:46.406+0000 7f9bc62a7700 1 --2- 172.21.15.40:0/275931145 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9bc00fee40 0x7f9bc00ff210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:46.410 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:46.406+0000 7f9bc62a7700 1 --2- 172.21.15.40:0/275931145 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9bc0103ee0 0x7f9bc010ba60 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:46.410 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:46.406+0000 7f9bc62a7700 1 --2- 172.21.15.40:0/275931145 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9bc00ff7e0 0x7f9bc01037b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:46.410 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:46.406+0000 7f9bc62a7700 1 -- 172.21.15.40:0/275931145 >> 172.21.15.40:0/275931145 conn(0x7f9bc0075ca0 msgr2=0x7f9bc00760a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:16:46.410 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:46.407+0000 7f9bc62a7700 1 -- 172.21.15.40:0/275931145 shutdown_connections 2023-07-14T19:16:46.410 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:46.407+0000 7f9bc62a7700 1 -- 172.21.15.40:0/275931145 wait complete. 2023-07-14T19:16:46.410 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:46.407+0000 7f9bc62a7700 1 Processor -- start 2023-07-14T19:16:46.411 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:46.407+0000 7f9bc62a7700 1 -- start start 2023-07-14T19:16:46.411 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:46.408+0000 7f9bc62a7700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9bc00fee40 0x7f9bc0195f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:16:46.411 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:46.408+0000 7f9bc62a7700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9bc00ff7e0 0x7f9bc0196450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:16:46.411 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:46.408+0000 7f9bc62a7700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9bc0103ee0 0x7f9bc019a8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:16:46.411 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:46.408+0000 7f9bc62a7700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9bc019aed0 con 0x7f9bc00ff7e0 2023-07-14T19:16:46.411 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:46.408+0000 7f9bc62a7700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9bc019b040 con 0x7f9bc0103ee0 2023-07-14T19:16:46.411 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:46.408+0000 7f9bc62a7700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9bc019b200 con 0x7f9bc00fee40 2023-07-14T19:16:46.412 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:46.408+0000 7f9bbf7fe700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9bc00ff7e0 0x7f9bc0196450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:16:46.412 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:46.408+0000 7f9bbf7fe700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9bc00ff7e0 0x7f9bc0196450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:33490/0 (socket says 172.21.15.40:33490) 2023-07-14T19:16:46.412 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:46.408+0000 7f9bbf7fe700 1 -- 172.21.15.40:0/355013863 learned_addr learned my addr 172.21.15.40:0/355013863 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-07-14T19:16:46.413 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:46.408+0000 7f9bc4844700 1 --2- 172.21.15.40:0/355013863 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9bc0103ee0 0x7f9bc019a8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:16:46.413 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:46.408+0000 7f9bbffff700 1 --2- 172.21.15.40:0/355013863 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9bc00fee40 0x7f9bc0195f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:16:46.413 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:46.408+0000 7f9bbf7fe700 1 -- 172.21.15.40:0/355013863 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9bc00fee40 msgr2=0x7f9bc0195f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:16:46.413 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:46.408+0000 7f9bbf7fe700 1 --2- 172.21.15.40:0/355013863 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9bc00fee40 0x7f9bc0195f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:46.413 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:46.408+0000 7f9bbf7fe700 1 -- 172.21.15.40:0/355013863 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9bc0103ee0 msgr2=0x7f9bc019a8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:16:46.414 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:46.408+0000 7f9bbf7fe700 1 --2- 172.21.15.40:0/355013863 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9bc0103ee0 0x7f9bc019a8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:46.414 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:46.408+0000 7f9bbf7fe700 1 -- 172.21.15.40:0/355013863 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9bb8009d00 con 0x7f9bc00ff7e0 2023-07-14T19:16:46.414 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:46.408+0000 7f9bbf7fe700 1 --2- 172.21.15.40:0/355013863 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9bc00ff7e0 0x7f9bc0196450 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f9bb400d980 tx=0x7f9bb400dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:16:46.414 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:46.408+0000 7f9bbd7fa700 1 -- 172.21.15.40:0/355013863 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bb4009930 con 0x7f9bc00ff7e0 2023-07-14T19:16:46.414 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:46.408+0000 7f9bc62a7700 1 -- 172.21.15.40:0/355013863 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9bc019b4e0 con 0x7f9bc00ff7e0 2023-07-14T19:16:46.414 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:46.408+0000 7f9bbd7fa700 1 -- 172.21.15.40:0/355013863 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f9bb4010460 con 0x7f9bc00ff7e0 2023-07-14T19:16:46.414 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:46.408+0000 7f9bbd7fa700 1 -- 172.21.15.40:0/355013863 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bb400f620 con 0x7f9bc00ff7e0 2023-07-14T19:16:46.415 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:46.408+0000 7f9bc62a7700 1 -- 172.21.15.40:0/355013863 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9bc01a2d40 con 0x7f9bc00ff7e0 2023-07-14T19:16:46.415 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:46.409+0000 7f9bc62a7700 1 -- 172.21.15.40:0/355013863 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9bc004e340 con 0x7f9bc00ff7e0 2023-07-14T19:16:46.416 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:46.411+0000 7f9bbd7fa700 1 -- 172.21.15.40:0/355013863 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7f9bb40108d0 con 0x7f9bc00ff7e0 2023-07-14T19:16:46.416 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:46.412+0000 7f9bbd7fa700 1 --2- 172.21.15.40:0/355013863 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f9ba80a11e0 0x7f9ba80a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:16:46.416 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:46.412+0000 7f9bbffff700 1 --2- 172.21.15.40:0/355013863 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f9ba80a11e0 0x7f9ba80a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:16:46.416 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:46.412+0000 7f9bbd7fa700 1 -- 172.21.15.40:0/355013863 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f9bb40cc430 con 0x7f9bc00ff7e0 2023-07-14T19:16:46.416 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:46.412+0000 7f9bbffff700 1 --2- 172.21.15.40:0/355013863 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f9ba80a11e0 0x7f9ba80a3690 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f9bb0005950 tx=0x7f9bb00058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:16:46.416 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:46.415+0000 7f9bbd7fa700 1 -- 172.21.15.40:0/355013863 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f9bb4095ad0 con 0x7f9bc00ff7e0 2023-07-14T19:16:46.608 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:46.607+0000 7f9bc62a7700 1 -- 172.21.15.40:0/355013863 --> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f9bc0068810 con 0x7f9ba80a11e0 2023-07-14T19:16:46.620 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:46.619+0000 7f9bbd7fa700 1 -- 172.21.15.40:0/355013863 <== mgr.14152 v2:172.21.15.40:6800/1693027118 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+341154 (secure 0 0 0) 0x7f9bc0068810 con 0x7f9ba80a11e0 2023-07-14T19:16:46.624 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-07-14T19:16:46.625 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:46.622+0000 7f9bc62a7700 1 -- 172.21.15.40:0/355013863 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f9ba80a11e0 msgr2=0x7f9ba80a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:16:46.626 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:46.622+0000 7f9bc62a7700 1 --2- 172.21.15.40:0/355013863 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f9ba80a11e0 0x7f9ba80a3690 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f9bb0005950 tx=0x7f9bb00058e0 comp rx=0 tx=0).stop 2023-07-14T19:16:46.626 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:46.622+0000 7f9bc62a7700 1 -- 172.21.15.40:0/355013863 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9bc00ff7e0 msgr2=0x7f9bc0196450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:16:46.626 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:46.622+0000 7f9bc62a7700 1 --2- 172.21.15.40:0/355013863 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9bc00ff7e0 0x7f9bc0196450 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f9bb400d980 tx=0x7f9bb400dc90 comp rx=0 tx=0).stop 2023-07-14T19:16:46.626 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:46.623+0000 7f9bc62a7700 1 -- 172.21.15.40:0/355013863 shutdown_connections 2023-07-14T19:16:46.626 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:46.623+0000 7f9bc62a7700 1 --2- 172.21.15.40:0/355013863 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f9ba80a11e0 0x7f9ba80a3690 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:46.626 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:46.623+0000 7f9bc62a7700 1 --2- 172.21.15.40:0/355013863 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9bc00fee40 0x7f9bc0195f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:46.626 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:46.623+0000 7f9bc62a7700 1 --2- 172.21.15.40:0/355013863 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9bc0103ee0 0x7f9bc019a8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:46.627 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:46.623+0000 7f9bc62a7700 1 --2- 172.21.15.40:0/355013863 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9bc00ff7e0 0x7f9bc0196450 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:46.627 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:46.623+0000 7f9bc62a7700 1 -- 172.21.15.40:0/355013863 >> 172.21.15.40:0/355013863 conn(0x7f9bc0075ca0 msgr2=0x7f9bc0068100 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:16:46.627 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:46.623+0000 7f9bc62a7700 1 -- 172.21.15.40:0/355013863 shutdown_connections 2023-07-14T19:16:46.627 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:46.623+0000 7f9bc62a7700 1 -- 172.21.15.40:0/355013863 wait complete. 2023-07-14T19:16:46.627 INFO:teuthology.orchestra.run.smithi040.stderr:dumped all 2023-07-14T19:16:46.971 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:16:46 smithi040 ceph-mon[96518]: pgmap v278: 97 pgs: 97 active+clean; 638 KiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2023-07-14T19:16:47.009 INFO:teuthology.orchestra.run.smithi040.stdout:{"pg_ready":true,"pg_map":{"version":278,"stamp":"2023-07-14T19:16:44.820657+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":653790,"num_objects":102,"num_object_clones":0,"num_object_copies":306,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":102,"num_whiteouts":0,"num_read":534,"num_read_kb":386,"num_write":410,"num_write_kb":3064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":338,"ondisk_log_size":338,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":335060,"kb_used_data":10868,"kb_used_omap":20,"kb_used_meta":324075,"kb_avail":1124606764,"statfs":{"total":1151940427776,"available":1151597326336,"internally_reserved":0,"allocated":11128832,"data_stored":6951202,"data_compressed":107142,"data_compressed_allocated":135168,"data_compressed_original":1769472,"omap_allocated":21372,"internal_metadata":331852932},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":113,"num_read_kb":55,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003322"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-07-14T19:15:59.938963+0000","last_change":"2023-07-14T19:15:58.250082+0000","last_active":"2023-07-14T19:15:59.938963+0000","last_peered":"2023-07-14T19:15:59.938963+0000","last_clean":"2023-07-14T19:15:59.938963+0000","last_became_active":"2023-07-14T19:15:58.249783+0000","last_became_peered":"2023-07-14T19:15:58.249783+0000","last_unstale":"2023-07-14T19:15:59.938963+0000","last_undegraded":"2023-07-14T19:15:59.938963+0000","last_fullsized":"2023-07-14T19:15:59.938963+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_clean_scrub_stamp":"2023-07-14T19:15:57.219223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T03:38:35.618742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-07-14T19:15:59.573145+0000","last_change":"2023-07-14T19:15:58.240082+0000","last_active":"2023-07-14T19:15:59.573145+0000","last_peered":"2023-07-14T19:15:59.573145+0000","last_clean":"2023-07-14T19:15:59.573145+0000","last_became_active":"2023-07-14T19:15:58.239788+0000","last_became_peered":"2023-07-14T19:15:58.239788+0000","last_unstale":"2023-07-14T19:15:59.573145+0000","last_undegraded":"2023-07-14T19:15:59.573145+0000","last_fullsized":"2023-07-14T19:15:59.573145+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_clean_scrub_stamp":"2023-07-14T19:15:57.219223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T05:20:02.889590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-07-14T19:15:59.573092+0000","last_change":"2023-07-14T19:15:58.248010+0000","last_active":"2023-07-14T19:15:59.573092+0000","last_peered":"2023-07-14T19:15:59.573092+0000","last_clean":"2023-07-14T19:15:59.573092+0000","last_became_active":"2023-07-14T19:15:58.247755+0000","last_became_peered":"2023-07-14T19:15:58.247755+0000","last_unstale":"2023-07-14T19:15:59.573092+0000","last_undegraded":"2023-07-14T19:15:59.573092+0000","last_fullsized":"2023-07-14T19:15:59.573092+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_clean_scrub_stamp":"2023-07-14T19:15:57.219223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T19:47:43.289654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-07-14T19:15:59.229952+0000","last_change":"2023-07-14T19:15:58.246946+0000","last_active":"2023-07-14T19:15:59.229952+0000","last_peered":"2023-07-14T19:15:59.229952+0000","last_clean":"2023-07-14T19:15:59.229952+0000","last_became_active":"2023-07-14T19:15:58.246598+0000","last_became_peered":"2023-07-14T19:15:58.246598+0000","last_unstale":"2023-07-14T19:15:59.229952+0000","last_undegraded":"2023-07-14T19:15:59.229952+0000","last_fullsized":"2023-07-14T19:15:59.229952+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_clean_scrub_stamp":"2023-07-14T19:15:57.219223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T06:05:42.069121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-07-14T19:15:59.232238+0000","last_change":"2023-07-14T19:15:58.252632+0000","last_active":"2023-07-14T19:15:59.232238+0000","last_peered":"2023-07-14T19:15:59.232238+0000","last_clean":"2023-07-14T19:15:59.232238+0000","last_became_active":"2023-07-14T19:15:58.252319+0000","last_became_peered":"2023-07-14T19:15:58.252319+0000","last_unstale":"2023-07-14T19:15:59.232238+0000","last_undegraded":"2023-07-14T19:15:59.232238+0000","last_fullsized":"2023-07-14T19:15:59.232238+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_clean_scrub_stamp":"2023-07-14T19:15:57.219223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T06:07:06.243539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-07-14T19:15:59.722095+0000","last_change":"2023-07-14T19:15:58.248305+0000","last_active":"2023-07-14T19:15:59.722095+0000","last_peered":"2023-07-14T19:15:59.722095+0000","last_clean":"2023-07-14T19:15:59.722095+0000","last_became_active":"2023-07-14T19:15:58.247347+0000","last_became_peered":"2023-07-14T19:15:58.247347+0000","last_unstale":"2023-07-14T19:15:59.722095+0000","last_undegraded":"2023-07-14T19:15:59.722095+0000","last_fullsized":"2023-07-14T19:15:59.722095+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_clean_scrub_stamp":"2023-07-14T19:15:57.219223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T05:16:26.859500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-07-14T19:15:59.721610+0000","last_change":"2023-07-14T19:15:58.240871+0000","last_active":"2023-07-14T19:15:59.721610+0000","last_peered":"2023-07-14T19:15:59.721610+0000","last_clean":"2023-07-14T19:15:59.721610+0000","last_became_active":"2023-07-14T19:15:58.240574+0000","last_became_peered":"2023-07-14T19:15:58.240574+0000","last_unstale":"2023-07-14T19:15:59.721610+0000","last_undegraded":"2023-07-14T19:15:59.721610+0000","last_fullsized":"2023-07-14T19:15:59.721610+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_clean_scrub_stamp":"2023-07-14T19:15:57.219223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T00:07:11.508517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-07-14T19:15:59.721987+0000","last_change":"2023-07-14T19:15:58.241979+0000","last_active":"2023-07-14T19:15:59.721987+0000","last_peered":"2023-07-14T19:15:59.721987+0000","last_clean":"2023-07-14T19:15:59.721987+0000","last_became_active":"2023-07-14T19:15:58.241111+0000","last_became_peered":"2023-07-14T19:15:58.241111+0000","last_unstale":"2023-07-14T19:15:59.721987+0000","last_undegraded":"2023-07-14T19:15:59.721987+0000","last_fullsized":"2023-07-14T19:15:59.721987+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_clean_scrub_stamp":"2023-07-14T19:15:57.219223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T07:04:12.177825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-07-14T19:15:59.721751+0000","last_change":"2023-07-14T19:15:58.248866+0000","last_active":"2023-07-14T19:15:59.721751+0000","last_peered":"2023-07-14T19:15:59.721751+0000","last_clean":"2023-07-14T19:15:59.721751+0000","last_became_active":"2023-07-14T19:15:58.248456+0000","last_became_peered":"2023-07-14T19:15:58.248456+0000","last_unstale":"2023-07-14T19:15:59.721751+0000","last_undegraded":"2023-07-14T19:15:59.721751+0000","last_fullsized":"2023-07-14T19:15:59.721751+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_clean_scrub_stamp":"2023-07-14T19:15:57.219223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:28:00.287289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-07-14T19:15:59.722575+0000","last_change":"2023-07-14T19:15:58.248296+0000","last_active":"2023-07-14T19:15:59.722575+0000","last_peered":"2023-07-14T19:15:59.722575+0000","last_clean":"2023-07-14T19:15:59.722575+0000","last_became_active":"2023-07-14T19:15:58.247810+0000","last_became_peered":"2023-07-14T19:15:58.247810+0000","last_unstale":"2023-07-14T19:15:59.722575+0000","last_undegraded":"2023-07-14T19:15:59.722575+0000","last_fullsized":"2023-07-14T19:15:59.722575+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_clean_scrub_stamp":"2023-07-14T19:15:57.219223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T03:06:56.929583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-07-14T19:15:59.722644+0000","last_change":"2023-07-14T19:15:58.252057+0000","last_active":"2023-07-14T19:15:59.722644+0000","last_peered":"2023-07-14T19:15:59.722644+0000","last_clean":"2023-07-14T19:15:59.722644+0000","last_became_active":"2023-07-14T19:15:58.251425+0000","last_became_peered":"2023-07-14T19:15:58.251425+0000","last_unstale":"2023-07-14T19:15:59.722644+0000","last_undegraded":"2023-07-14T19:15:59.722644+0000","last_fullsized":"2023-07-14T19:15:59.722644+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_clean_scrub_stamp":"2023-07-14T19:15:57.219223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T00:23:30.017734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-07-14T19:15:59.229817+0000","last_change":"2023-07-14T19:15:58.241009+0000","last_active":"2023-07-14T19:15:59.229817+0000","last_peered":"2023-07-14T19:15:59.229817+0000","last_clean":"2023-07-14T19:15:59.229817+0000","last_became_active":"2023-07-14T19:15:58.240622+0000","last_became_peered":"2023-07-14T19:15:58.240622+0000","last_unstale":"2023-07-14T19:15:59.229817+0000","last_undegraded":"2023-07-14T19:15:59.229817+0000","last_fullsized":"2023-07-14T19:15:59.229817+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_clean_scrub_stamp":"2023-07-14T19:15:57.219223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T05:28:57.780030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-07-14T19:15:59.722108+0000","last_change":"2023-07-14T19:15:58.243377+0000","last_active":"2023-07-14T19:15:59.722108+0000","last_peered":"2023-07-14T19:15:59.722108+0000","last_clean":"2023-07-14T19:15:59.722108+0000","last_became_active":"2023-07-14T19:15:58.242225+0000","last_became_peered":"2023-07-14T19:15:58.242225+0000","last_unstale":"2023-07-14T19:15:59.722108+0000","last_undegraded":"2023-07-14T19:15:59.722108+0000","last_fullsized":"2023-07-14T19:15:59.722108+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_clean_scrub_stamp":"2023-07-14T19:15:57.219223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:21:59.269854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-07-14T19:15:59.573307+0000","last_change":"2023-07-14T19:15:58.244556+0000","last_active":"2023-07-14T19:15:59.573307+0000","last_peered":"2023-07-14T19:15:59.573307+0000","last_clean":"2023-07-14T19:15:59.573307+0000","last_became_active":"2023-07-14T19:15:58.244228+0000","last_became_peered":"2023-07-14T19:15:58.244228+0000","last_unstale":"2023-07-14T19:15:59.573307+0000","last_undegraded":"2023-07-14T19:15:59.573307+0000","last_fullsized":"2023-07-14T19:15:59.573307+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_clean_scrub_stamp":"2023-07-14T19:15:57.219223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:11:22.589361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-07-14T19:15:59.572875+0000","last_change":"2023-07-14T19:15:58.250176+0000","last_active":"2023-07-14T19:15:59.572875+0000","last_peered":"2023-07-14T19:15:59.572875+0000","last_clean":"2023-07-14T19:15:59.572875+0000","last_became_active":"2023-07-14T19:15:58.249708+0000","last_became_peered":"2023-07-14T19:15:58.249708+0000","last_unstale":"2023-07-14T19:15:59.572875+0000","last_undegraded":"2023-07-14T19:15:59.572875+0000","last_fullsized":"2023-07-14T19:15:59.572875+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_clean_scrub_stamp":"2023-07-14T19:15:57.219223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T22:33:15.826269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-07-14T19:15:59.721419+0000","last_change":"2023-07-14T19:15:58.246859+0000","last_active":"2023-07-14T19:15:59.721419+0000","last_peered":"2023-07-14T19:15:59.721419+0000","last_clean":"2023-07-14T19:15:59.721419+0000","last_became_active":"2023-07-14T19:15:58.246541+0000","last_became_peered":"2023-07-14T19:15:58.246541+0000","last_unstale":"2023-07-14T19:15:59.721419+0000","last_undegraded":"2023-07-14T19:15:59.721419+0000","last_fullsized":"2023-07-14T19:15:59.721419+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_clean_scrub_stamp":"2023-07-14T19:15:57.219223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T03:07:22.056544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-07-14T19:15:59.939090+0000","last_change":"2023-07-14T19:15:58.250112+0000","last_active":"2023-07-14T19:15:59.939090+0000","last_peered":"2023-07-14T19:15:59.939090+0000","last_clean":"2023-07-14T19:15:59.939090+0000","last_became_active":"2023-07-14T19:15:58.249824+0000","last_became_peered":"2023-07-14T19:15:58.249824+0000","last_unstale":"2023-07-14T19:15:59.939090+0000","last_undegraded":"2023-07-14T19:15:59.939090+0000","last_fullsized":"2023-07-14T19:15:59.939090+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_clean_scrub_stamp":"2023-07-14T19:15:57.219223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T22:34:26.596011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-07-14T19:15:59.723229+0000","last_change":"2023-07-14T19:15:58.243011+0000","last_active":"2023-07-14T19:15:59.723229+0000","last_peered":"2023-07-14T19:15:59.723229+0000","last_clean":"2023-07-14T19:15:59.723229+0000","last_became_active":"2023-07-14T19:15:58.242693+0000","last_became_peered":"2023-07-14T19:15:58.242693+0000","last_unstale":"2023-07-14T19:15:59.723229+0000","last_undegraded":"2023-07-14T19:15:59.723229+0000","last_fullsized":"2023-07-14T19:15:59.723229+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_clean_scrub_stamp":"2023-07-14T19:15:57.219223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:04:37.187798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-07-14T19:15:59.721093+0000","last_change":"2023-07-14T19:15:58.251298+0000","last_active":"2023-07-14T19:15:59.721093+0000","last_peered":"2023-07-14T19:15:59.721093+0000","last_clean":"2023-07-14T19:15:59.721093+0000","last_became_active":"2023-07-14T19:15:58.250984+0000","last_became_peered":"2023-07-14T19:15:58.250984+0000","last_unstale":"2023-07-14T19:15:59.721093+0000","last_undegraded":"2023-07-14T19:15:59.721093+0000","last_fullsized":"2023-07-14T19:15:59.721093+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_clean_scrub_stamp":"2023-07-14T19:15:57.219223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T20:29:32.350678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"81'13","reported_seq":67,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-07-14T19:16:42.686648+0000","last_change":"2023-07-14T19:15:57.235142+0000","last_active":"2023-07-14T19:16:42.686648+0000","last_peered":"2023-07-14T19:16:42.686648+0000","last_clean":"2023-07-14T19:16:42.686648+0000","last_became_active":"2023-07-14T19:15:57.234803+0000","last_became_peered":"2023-07-14T19:15:57.234803+0000","last_unstale":"2023-07-14T19:16:42.686648+0000","last_undegraded":"2023-07-14T19:16:42.686648+0000","last_fullsized":"2023-07-14T19:16:42.686648+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:15:56.210166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:15:56.210166+0000","last_clean_scrub_stamp":"2023-07-14T19:15:56.210166+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:27:04.191229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":40,"num_read_kb":40,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-07-14T19:15:59.722681+0000","last_change":"2023-07-14T19:15:58.247688+0000","last_active":"2023-07-14T19:15:59.722681+0000","last_peered":"2023-07-14T19:15:59.722681+0000","last_clean":"2023-07-14T19:15:59.722681+0000","last_became_active":"2023-07-14T19:15:58.247400+0000","last_became_peered":"2023-07-14T19:15:58.247400+0000","last_unstale":"2023-07-14T19:15:59.722681+0000","last_undegraded":"2023-07-14T19:15:59.722681+0000","last_fullsized":"2023-07-14T19:15:59.722681+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_clean_scrub_stamp":"2023-07-14T19:15:57.219223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T21:28:57.315894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"81'2","reported_seq":16,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-07-14T19:16:00.704237+0000","last_change":"2023-07-14T19:15:57.240026+0000","last_active":"2023-07-14T19:16:00.704237+0000","last_peered":"2023-07-14T19:16:00.704237+0000","last_clean":"2023-07-14T19:16:00.704237+0000","last_became_active":"2023-07-14T19:15:57.239687+0000","last_became_peered":"2023-07-14T19:15:57.239687+0000","last_unstale":"2023-07-14T19:16:00.704237+0000","last_undegraded":"2023-07-14T19:16:00.704237+0000","last_fullsized":"2023-07-14T19:16:00.704237+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:15:56.210166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:15:56.210166+0000","last_clean_scrub_stamp":"2023-07-14T19:15:56.210166+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:18:22.780240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-07-14T19:15:59.573195+0000","last_change":"2023-07-14T19:15:58.247039+0000","last_active":"2023-07-14T19:15:59.573195+0000","last_peered":"2023-07-14T19:15:59.573195+0000","last_clean":"2023-07-14T19:15:59.573195+0000","last_became_active":"2023-07-14T19:15:58.246685+0000","last_became_peered":"2023-07-14T19:15:58.246685+0000","last_unstale":"2023-07-14T19:15:59.573195+0000","last_undegraded":"2023-07-14T19:15:59.573195+0000","last_fullsized":"2023-07-14T19:15:59.573195+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_clean_scrub_stamp":"2023-07-14T19:15:57.219223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T03:20:32.272596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"81'12","reported_seq":27,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-07-14T19:16:05.632654+0000","last_change":"2023-07-14T19:15:57.244612+0000","last_active":"2023-07-14T19:16:05.632654+0000","last_peered":"2023-07-14T19:16:05.632654+0000","last_clean":"2023-07-14T19:16:05.632654+0000","last_became_active":"2023-07-14T19:15:57.243761+0000","last_became_peered":"2023-07-14T19:15:57.243761+0000","last_unstale":"2023-07-14T19:16:05.632654+0000","last_undegraded":"2023-07-14T19:16:05.632654+0000","last_fullsized":"2023-07-14T19:16:05.632654+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:15:56.210166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:15:56.210166+0000","last_clean_scrub_stamp":"2023-07-14T19:15:56.210166+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T21:44:07.276569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-07-14T19:15:59.721874+0000","last_change":"2023-07-14T19:15:58.239903+0000","last_active":"2023-07-14T19:15:59.721874+0000","last_peered":"2023-07-14T19:15:59.721874+0000","last_clean":"2023-07-14T19:15:59.721874+0000","last_became_active":"2023-07-14T19:15:58.239627+0000","last_became_peered":"2023-07-14T19:15:58.239627+0000","last_unstale":"2023-07-14T19:15:59.721874+0000","last_undegraded":"2023-07-14T19:15:59.721874+0000","last_fullsized":"2023-07-14T19:15:59.721874+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_clean_scrub_stamp":"2023-07-14T19:15:57.219223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:56:30.086686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"81'5","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-07-14T19:16:04.619350+0000","last_change":"2023-07-14T19:15:57.241773+0000","last_active":"2023-07-14T19:16:04.619350+0000","last_peered":"2023-07-14T19:16:04.619350+0000","last_clean":"2023-07-14T19:16:04.619350+0000","last_became_active":"2023-07-14T19:15:57.241451+0000","last_became_peered":"2023-07-14T19:15:57.241451+0000","last_unstale":"2023-07-14T19:16:04.619350+0000","last_undegraded":"2023-07-14T19:16:04.619350+0000","last_fullsized":"2023-07-14T19:16:04.619350+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:15:56.210166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:15:56.210166+0000","last_clean_scrub_stamp":"2023-07-14T19:15:56.210166+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T00:53:37.977462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-07-14T19:15:59.231297+0000","last_change":"2023-07-14T19:15:58.250564+0000","last_active":"2023-07-14T19:15:59.231297+0000","last_peered":"2023-07-14T19:15:59.231297+0000","last_clean":"2023-07-14T19:15:59.231297+0000","last_became_active":"2023-07-14T19:15:58.250277+0000","last_became_peered":"2023-07-14T19:15:58.250277+0000","last_unstale":"2023-07-14T19:15:59.231297+0000","last_undegraded":"2023-07-14T19:15:59.231297+0000","last_fullsized":"2023-07-14T19:15:59.231297+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_clean_scrub_stamp":"2023-07-14T19:15:57.219223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:37:56.001097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"81'5","reported_seq":59,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-07-14T19:16:41.686609+0000","last_change":"2023-07-14T19:15:57.245586+0000","last_active":"2023-07-14T19:16:41.686609+0000","last_peered":"2023-07-14T19:16:41.686609+0000","last_clean":"2023-07-14T19:16:41.686609+0000","last_became_active":"2023-07-14T19:15:57.244767+0000","last_became_peered":"2023-07-14T19:15:57.244767+0000","last_unstale":"2023-07-14T19:16:41.686609+0000","last_undegraded":"2023-07-14T19:16:41.686609+0000","last_fullsized":"2023-07-14T19:16:41.686609+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:15:56.210166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:15:56.210166+0000","last_clean_scrub_stamp":"2023-07-14T19:15:56.210166+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T06:43:35.975084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":574,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":40,"num_read_kb":2,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-07-14T19:15:59.232307+0000","last_change":"2023-07-14T19:15:58.240509+0000","last_active":"2023-07-14T19:15:59.232307+0000","last_peered":"2023-07-14T19:15:59.232307+0000","last_clean":"2023-07-14T19:15:59.232307+0000","last_became_active":"2023-07-14T19:15:58.240198+0000","last_became_peered":"2023-07-14T19:15:58.240198+0000","last_unstale":"2023-07-14T19:15:59.232307+0000","last_undegraded":"2023-07-14T19:15:59.232307+0000","last_fullsized":"2023-07-14T19:15:59.232307+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_clean_scrub_stamp":"2023-07-14T19:15:57.219223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T19:29:25.962372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"81'7","reported_seq":21,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-07-14T19:16:03.612450+0000","last_change":"2023-07-14T19:15:57.239582+0000","last_active":"2023-07-14T19:16:03.612450+0000","last_peered":"2023-07-14T19:16:03.612450+0000","last_clean":"2023-07-14T19:16:03.612450+0000","last_became_active":"2023-07-14T19:15:57.239244+0000","last_became_peered":"2023-07-14T19:15:57.239244+0000","last_unstale":"2023-07-14T19:16:03.612450+0000","last_undegraded":"2023-07-14T19:16:03.612450+0000","last_fullsized":"2023-07-14T19:16:03.612450+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:15:56.210166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:15:56.210166+0000","last_clean_scrub_stamp":"2023-07-14T19:15:56.210166+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T03:16:58.719650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-07-14T19:15:59.231791+0000","last_change":"2023-07-14T19:15:58.249443+0000","last_active":"2023-07-14T19:15:59.231791+0000","last_peered":"2023-07-14T19:15:59.231791+0000","last_clean":"2023-07-14T19:15:59.231791+0000","last_became_active":"2023-07-14T19:15:58.249131+0000","last_became_peered":"2023-07-14T19:15:58.249131+0000","last_unstale":"2023-07-14T19:15:59.231791+0000","last_undegraded":"2023-07-14T19:15:59.231791+0000","last_fullsized":"2023-07-14T19:15:59.231791+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_clean_scrub_stamp":"2023-07-14T19:15:57.219223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T03:45:01.049663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"81'3","reported_seq":17,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-07-14T19:16:02.597743+0000","last_change":"2023-07-14T19:15:57.243725+0000","last_active":"2023-07-14T19:16:02.597743+0000","last_peered":"2023-07-14T19:16:02.597743+0000","last_clean":"2023-07-14T19:16:02.597743+0000","last_became_active":"2023-07-14T19:15:57.243431+0000","last_became_peered":"2023-07-14T19:15:57.243431+0000","last_unstale":"2023-07-14T19:16:02.597743+0000","last_undegraded":"2023-07-14T19:16:02.597743+0000","last_fullsized":"2023-07-14T19:16:02.597743+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:15:56.210166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:15:56.210166+0000","last_clean_scrub_stamp":"2023-07-14T19:15:56.210166+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:41:49.174581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-07-14T19:15:59.721836+0000","last_change":"2023-07-14T19:15:58.239460+0000","last_active":"2023-07-14T19:15:59.721836+0000","last_peered":"2023-07-14T19:15:59.721836+0000","last_clean":"2023-07-14T19:15:59.721836+0000","last_became_active":"2023-07-14T19:15:58.239086+0000","last_became_peered":"2023-07-14T19:15:58.239086+0000","last_unstale":"2023-07-14T19:15:59.721836+0000","last_undegraded":"2023-07-14T19:15:59.721836+0000","last_fullsized":"2023-07-14T19:15:59.721836+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_clean_scrub_stamp":"2023-07-14T19:15:57.219223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T23:25:07.452583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"81'3","reported_seq":18,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-07-14T19:16:04.617213+0000","last_change":"2023-07-14T19:15:57.240173+0000","last_active":"2023-07-14T19:16:04.617213+0000","last_peered":"2023-07-14T19:16:04.617213+0000","last_clean":"2023-07-14T19:16:04.617213+0000","last_became_active":"2023-07-14T19:15:57.239822+0000","last_became_peered":"2023-07-14T19:15:57.239822+0000","last_unstale":"2023-07-14T19:16:04.617213+0000","last_undegraded":"2023-07-14T19:16:04.617213+0000","last_fullsized":"2023-07-14T19:16:04.617213+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:15:56.210166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:15:56.210166+0000","last_clean_scrub_stamp":"2023-07-14T19:15:56.210166+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T20:22:32.066346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-07-14T19:15:59.722607+0000","last_change":"2023-07-14T19:15:58.241533+0000","last_active":"2023-07-14T19:15:59.722607+0000","last_peered":"2023-07-14T19:15:59.722607+0000","last_clean":"2023-07-14T19:15:59.722607+0000","last_became_active":"2023-07-14T19:15:58.241216+0000","last_became_peered":"2023-07-14T19:15:58.241216+0000","last_unstale":"2023-07-14T19:15:59.722607+0000","last_undegraded":"2023-07-14T19:15:59.722607+0000","last_fullsized":"2023-07-14T19:15:59.722607+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_clean_scrub_stamp":"2023-07-14T19:15:57.219223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:40:27.802949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"81'4","reported_seq":88,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-07-14T19:16:39.683031+0000","last_change":"2023-07-14T19:15:57.236808+0000","last_active":"2023-07-14T19:16:39.683031+0000","last_peered":"2023-07-14T19:16:39.683031+0000","last_clean":"2023-07-14T19:16:39.683031+0000","last_became_active":"2023-07-14T19:15:57.236480+0000","last_became_peered":"2023-07-14T19:15:57.236480+0000","last_unstale":"2023-07-14T19:16:39.683031+0000","last_undegraded":"2023-07-14T19:16:39.683031+0000","last_fullsized":"2023-07-14T19:16:39.683031+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:15:56.210166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:15:56.210166+0000","last_clean_scrub_stamp":"2023-07-14T19:15:56.210166+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T20:50:18.427213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":70,"num_read_kb":70,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-07-14T19:15:59.721741+0000","last_change":"2023-07-14T19:15:58.243236+0000","last_active":"2023-07-14T19:15:59.721741+0000","last_peered":"2023-07-14T19:15:59.721741+0000","last_clean":"2023-07-14T19:15:59.721741+0000","last_became_active":"2023-07-14T19:15:58.242951+0000","last_became_peered":"2023-07-14T19:15:58.242951+0000","last_unstale":"2023-07-14T19:15:59.721741+0000","last_undegraded":"2023-07-14T19:15:59.721741+0000","last_fullsized":"2023-07-14T19:15:59.721741+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_clean_scrub_stamp":"2023-07-14T19:15:57.219223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T05:20:59.047342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"81'4","reported_seq":18,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-07-14T19:16:04.617616+0000","last_change":"2023-07-14T19:15:57.242251+0000","last_active":"2023-07-14T19:16:04.617616+0000","last_peered":"2023-07-14T19:16:04.617616+0000","last_clean":"2023-07-14T19:16:04.617616+0000","last_became_active":"2023-07-14T19:15:57.241907+0000","last_became_peered":"2023-07-14T19:15:57.241907+0000","last_unstale":"2023-07-14T19:16:04.617616+0000","last_undegraded":"2023-07-14T19:16:04.617616+0000","last_fullsized":"2023-07-14T19:16:04.617616+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:15:56.210166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:15:56.210166+0000","last_clean_scrub_stamp":"2023-07-14T19:15:56.210166+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:52:53.425429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-07-14T19:15:59.573106+0000","last_change":"2023-07-14T19:15:58.248412+0000","last_active":"2023-07-14T19:15:59.573106+0000","last_peered":"2023-07-14T19:15:59.573106+0000","last_clean":"2023-07-14T19:15:59.573106+0000","last_became_active":"2023-07-14T19:15:58.247619+0000","last_became_peered":"2023-07-14T19:15:58.247619+0000","last_unstale":"2023-07-14T19:15:59.573106+0000","last_undegraded":"2023-07-14T19:15:59.573106+0000","last_fullsized":"2023-07-14T19:15:59.573106+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_clean_scrub_stamp":"2023-07-14T19:15:57.219223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T00:15:52.855579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"81'5","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-07-14T19:16:03.611516+0000","last_change":"2023-07-14T19:15:57.240436+0000","last_active":"2023-07-14T19:16:03.611516+0000","last_peered":"2023-07-14T19:16:03.611516+0000","last_clean":"2023-07-14T19:16:03.611516+0000","last_became_active":"2023-07-14T19:15:57.240148+0000","last_became_peered":"2023-07-14T19:15:57.240148+0000","last_unstale":"2023-07-14T19:16:03.611516+0000","last_undegraded":"2023-07-14T19:16:03.611516+0000","last_fullsized":"2023-07-14T19:16:03.611516+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:15:56.210166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:15:56.210166+0000","last_clean_scrub_stamp":"2023-07-14T19:15:56.210166+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T03:15:40.160793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-07-14T19:15:59.722062+0000","last_change":"2023-07-14T19:15:58.246127+0000","last_active":"2023-07-14T19:15:59.722062+0000","last_peered":"2023-07-14T19:15:59.722062+0000","last_clean":"2023-07-14T19:15:59.722062+0000","last_became_active":"2023-07-14T19:15:58.245778+0000","last_became_peered":"2023-07-14T19:15:58.245778+0000","last_unstale":"2023-07-14T19:15:59.722062+0000","last_undegraded":"2023-07-14T19:15:59.722062+0000","last_fullsized":"2023-07-14T19:15:59.722062+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_clean_scrub_stamp":"2023-07-14T19:15:57.219223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T00:21:57.683936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"81'28","reported_seq":117,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-07-14T19:16:39.682868+0000","last_change":"2023-07-14T19:15:57.243660+0000","last_active":"2023-07-14T19:16:39.682868+0000","last_peered":"2023-07-14T19:16:39.682868+0000","last_clean":"2023-07-14T19:16:39.682868+0000","last_became_active":"2023-07-14T19:15:57.243338+0000","last_became_peered":"2023-07-14T19:15:57.243338+0000","last_unstale":"2023-07-14T19:16:39.682868+0000","last_undegraded":"2023-07-14T19:16:39.682868+0000","last_fullsized":"2023-07-14T19:16:39.682868+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:15:56.210166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:15:56.210166+0000","last_clean_scrub_stamp":"2023-07-14T19:15:56.210166+0000","objects_scrubbed":0,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T06:12:23.930218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56946,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":75,"num_read_kb":93,"num_write":30,"num_write_kb":64,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-07-14T19:15:59.232525+0000","last_change":"2023-07-14T19:15:58.244162+0000","last_active":"2023-07-14T19:15:59.232525+0000","last_peered":"2023-07-14T19:15:59.232525+0000","last_clean":"2023-07-14T19:15:59.232525+0000","last_became_active":"2023-07-14T19:15:58.243912+0000","last_became_peered":"2023-07-14T19:15:58.243912+0000","last_unstale":"2023-07-14T19:15:59.232525+0000","last_undegraded":"2023-07-14T19:15:59.232525+0000","last_fullsized":"2023-07-14T19:15:59.232525+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_clean_scrub_stamp":"2023-07-14T19:15:57.219223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T23:05:24.992671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"81'9","reported_seq":25,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-07-14T19:16:05.632720+0000","last_change":"2023-07-14T19:15:57.246975+0000","last_active":"2023-07-14T19:16:05.632720+0000","last_peered":"2023-07-14T19:16:05.632720+0000","last_clean":"2023-07-14T19:16:05.632720+0000","last_became_active":"2023-07-14T19:15:57.246481+0000","last_became_peered":"2023-07-14T19:15:57.246481+0000","last_unstale":"2023-07-14T19:16:05.632720+0000","last_undegraded":"2023-07-14T19:16:05.632720+0000","last_fullsized":"2023-07-14T19:16:05.632720+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:15:56.210166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:15:56.210166+0000","last_clean_scrub_stamp":"2023-07-14T19:15:56.210166+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T06:50:24.165830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-07-14T19:15:59.721632+0000","last_change":"2023-07-14T19:15:58.241870+0000","last_active":"2023-07-14T19:15:59.721632+0000","last_peered":"2023-07-14T19:15:59.721632+0000","last_clean":"2023-07-14T19:15:59.721632+0000","last_became_active":"2023-07-14T19:15:58.241503+0000","last_became_peered":"2023-07-14T19:15:58.241503+0000","last_unstale":"2023-07-14T19:15:59.721632+0000","last_undegraded":"2023-07-14T19:15:59.721632+0000","last_fullsized":"2023-07-14T19:15:59.721632+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_clean_scrub_stamp":"2023-07-14T19:15:57.219223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T23:05:14.779450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"81'5","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-07-14T19:16:04.617233+0000","last_change":"2023-07-14T19:15:57.236715+0000","last_active":"2023-07-14T19:16:04.617233+0000","last_peered":"2023-07-14T19:16:04.617233+0000","last_clean":"2023-07-14T19:16:04.617233+0000","last_became_active":"2023-07-14T19:15:57.236391+0000","last_became_peered":"2023-07-14T19:15:57.236391+0000","last_unstale":"2023-07-14T19:16:04.617233+0000","last_undegraded":"2023-07-14T19:16:04.617233+0000","last_fullsized":"2023-07-14T19:16:04.617233+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:15:56.210166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:15:56.210166+0000","last_clean_scrub_stamp":"2023-07-14T19:15:56.210166+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:06:23.938675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-07-14T19:15:59.721870+0000","last_change":"2023-07-14T19:15:58.245110+0000","last_active":"2023-07-14T19:15:59.721870+0000","last_peered":"2023-07-14T19:15:59.721870+0000","last_clean":"2023-07-14T19:15:59.721870+0000","last_became_active":"2023-07-14T19:15:58.244779+0000","last_became_peered":"2023-07-14T19:15:58.244779+0000","last_unstale":"2023-07-14T19:15:59.721870+0000","last_undegraded":"2023-07-14T19:15:59.721870+0000","last_fullsized":"2023-07-14T19:15:59.721870+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_clean_scrub_stamp":"2023-07-14T19:15:57.219223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T00:24:23.826897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"81'5","reported_seq":58,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-07-14T19:16:40.684665+0000","last_change":"2023-07-14T19:15:57.236795+0000","last_active":"2023-07-14T19:16:40.684665+0000","last_peered":"2023-07-14T19:16:40.684665+0000","last_clean":"2023-07-14T19:16:40.684665+0000","last_became_active":"2023-07-14T19:15:57.236442+0000","last_became_peered":"2023-07-14T19:15:57.236442+0000","last_unstale":"2023-07-14T19:16:40.684665+0000","last_undegraded":"2023-07-14T19:16:40.684665+0000","last_fullsized":"2023-07-14T19:16:40.684665+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:15:56.210166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:15:56.210166+0000","last_clean_scrub_stamp":"2023-07-14T19:15:56.210166+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:05:41.764871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":540,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":40,"num_read_kb":3,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-07-14T19:15:59.721448+0000","last_change":"2023-07-14T19:15:58.247790+0000","last_active":"2023-07-14T19:15:59.721448+0000","last_peered":"2023-07-14T19:15:59.721448+0000","last_clean":"2023-07-14T19:15:59.721448+0000","last_became_active":"2023-07-14T19:15:58.247461+0000","last_became_peered":"2023-07-14T19:15:58.247461+0000","last_unstale":"2023-07-14T19:15:59.721448+0000","last_undegraded":"2023-07-14T19:15:59.721448+0000","last_fullsized":"2023-07-14T19:15:59.721448+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_clean_scrub_stamp":"2023-07-14T19:15:57.219223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:29:24.306834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"81'17","reported_seq":32,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-07-14T19:16:04.618370+0000","last_change":"2023-07-14T19:15:57.244534+0000","last_active":"2023-07-14T19:16:04.618370+0000","last_peered":"2023-07-14T19:16:04.618370+0000","last_clean":"2023-07-14T19:16:04.618370+0000","last_became_active":"2023-07-14T19:15:57.243107+0000","last_became_peered":"2023-07-14T19:15:57.243107+0000","last_unstale":"2023-07-14T19:16:04.618370+0000","last_undegraded":"2023-07-14T19:16:04.618370+0000","last_fullsized":"2023-07-14T19:16:04.618370+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:15:56.210166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:15:56.210166+0000","last_clean_scrub_stamp":"2023-07-14T19:15:56.210166+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T21:48:40.199873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-07-14T19:15:59.231092+0000","last_change":"2023-07-14T19:15:58.250511+0000","last_active":"2023-07-14T19:15:59.231092+0000","last_peered":"2023-07-14T19:15:59.231092+0000","last_clean":"2023-07-14T19:15:59.231092+0000","last_became_active":"2023-07-14T19:15:58.250159+0000","last_became_peered":"2023-07-14T19:15:58.250159+0000","last_unstale":"2023-07-14T19:15:59.231092+0000","last_undegraded":"2023-07-14T19:15:59.231092+0000","last_fullsized":"2023-07-14T19:15:59.231092+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_clean_scrub_stamp":"2023-07-14T19:15:57.219223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T21:32:01.440745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"81'2","reported_seq":17,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-07-14T19:16:05.632535+0000","last_change":"2023-07-14T19:15:57.238719+0000","last_active":"2023-07-14T19:16:05.632535+0000","last_peered":"2023-07-14T19:16:05.632535+0000","last_clean":"2023-07-14T19:16:05.632535+0000","last_became_active":"2023-07-14T19:15:57.238404+0000","last_became_peered":"2023-07-14T19:15:57.238404+0000","last_unstale":"2023-07-14T19:16:05.632535+0000","last_undegraded":"2023-07-14T19:16:05.632535+0000","last_fullsized":"2023-07-14T19:16:05.632535+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:15:56.210166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:15:56.210166+0000","last_clean_scrub_stamp":"2023-07-14T19:15:56.210166+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T03:14:38.948894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-07-14T19:15:59.938888+0000","last_change":"2023-07-14T19:15:58.245052+0000","last_active":"2023-07-14T19:15:59.938888+0000","last_peered":"2023-07-14T19:15:59.938888+0000","last_clean":"2023-07-14T19:15:59.938888+0000","last_became_active":"2023-07-14T19:15:58.244733+0000","last_became_peered":"2023-07-14T19:15:58.244733+0000","last_unstale":"2023-07-14T19:15:59.938888+0000","last_undegraded":"2023-07-14T19:15:59.938888+0000","last_fullsized":"2023-07-14T19:15:59.938888+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_clean_scrub_stamp":"2023-07-14T19:15:57.219223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T03:43:49.689439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"81'1","reported_seq":15,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-07-14T19:15:59.722595+0000","last_change":"2023-07-14T19:15:57.245402+0000","last_active":"2023-07-14T19:15:59.722595+0000","last_peered":"2023-07-14T19:15:59.722595+0000","last_clean":"2023-07-14T19:15:59.722595+0000","last_became_active":"2023-07-14T19:15:57.245092+0000","last_became_peered":"2023-07-14T19:15:57.245092+0000","last_unstale":"2023-07-14T19:15:59.722595+0000","last_undegraded":"2023-07-14T19:15:59.722595+0000","last_fullsized":"2023-07-14T19:15:59.722595+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:15:56.210166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:15:56.210166+0000","last_clean_scrub_stamp":"2023-07-14T19:15:56.210166+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:17:28.260009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"81'1","reported_seq":13,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-07-14T19:16:00.307970+0000","last_change":"2023-07-14T19:15:58.247611+0000","last_active":"2023-07-14T19:16:00.307970+0000","last_peered":"2023-07-14T19:16:00.307970+0000","last_clean":"2023-07-14T19:16:00.307970+0000","last_became_active":"2023-07-14T19:15:58.247313+0000","last_became_peered":"2023-07-14T19:15:58.247313+0000","last_unstale":"2023-07-14T19:16:00.307970+0000","last_undegraded":"2023-07-14T19:16:00.307970+0000","last_fullsized":"2023-07-14T19:16:00.307970+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_clean_scrub_stamp":"2023-07-14T19:15:57.219223+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T06:40:19.317380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":123,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"81'5","reported_seq":56,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-07-14T19:16:39.683026+0000","last_change":"2023-07-14T19:15:57.242149+0000","last_active":"2023-07-14T19:16:39.683026+0000","last_peered":"2023-07-14T19:16:39.683026+0000","last_clean":"2023-07-14T19:16:39.683026+0000","last_became_active":"2023-07-14T19:15:57.241847+0000","last_became_peered":"2023-07-14T19:15:57.241847+0000","last_unstale":"2023-07-14T19:16:39.683026+0000","last_undegraded":"2023-07-14T19:16:39.683026+0000","last_fullsized":"2023-07-14T19:16:39.683026+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:15:56.210166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:15:56.210166+0000","last_clean_scrub_stamp":"2023-07-14T19:15:56.210166+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:14:47.640280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":540,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":37,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-07-14T19:15:59.722352+0000","last_change":"2023-07-14T19:15:58.244573+0000","last_active":"2023-07-14T19:15:59.722352+0000","last_peered":"2023-07-14T19:15:59.722352+0000","last_clean":"2023-07-14T19:15:59.722352+0000","last_became_active":"2023-07-14T19:15:58.244165+0000","last_became_peered":"2023-07-14T19:15:58.244165+0000","last_unstale":"2023-07-14T19:15:59.722352+0000","last_undegraded":"2023-07-14T19:15:59.722352+0000","last_fullsized":"2023-07-14T19:15:59.722352+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_clean_scrub_stamp":"2023-07-14T19:15:57.219223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T05:36:19.851400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-07-14T19:15:59.231138+0000","last_change":"2023-07-14T19:15:58.246282+0000","last_active":"2023-07-14T19:15:59.231138+0000","last_peered":"2023-07-14T19:15:59.231138+0000","last_clean":"2023-07-14T19:15:59.231138+0000","last_became_active":"2023-07-14T19:15:58.245984+0000","last_became_peered":"2023-07-14T19:15:58.245984+0000","last_unstale":"2023-07-14T19:15:59.231138+0000","last_undegraded":"2023-07-14T19:15:59.231138+0000","last_fullsized":"2023-07-14T19:15:59.231138+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_clean_scrub_stamp":"2023-07-14T19:15:57.219223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T00:42:41.109802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"81'5","reported_seq":55,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-07-14T19:16:38.681343+0000","last_change":"2023-07-14T19:15:57.244731+0000","last_active":"2023-07-14T19:16:38.681343+0000","last_peered":"2023-07-14T19:16:38.681343+0000","last_clean":"2023-07-14T19:16:38.681343+0000","last_became_active":"2023-07-14T19:15:57.244384+0000","last_became_peered":"2023-07-14T19:15:57.244384+0000","last_unstale":"2023-07-14T19:16:38.681343+0000","last_undegraded":"2023-07-14T19:16:38.681343+0000","last_fullsized":"2023-07-14T19:16:38.681343+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:15:56.210166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:15:56.210166+0000","last_clean_scrub_stamp":"2023-07-14T19:15:56.210166+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T22:20:50.336578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":36,"num_read_kb":36,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-07-14T19:15:59.722302+0000","last_change":"2023-07-14T19:15:58.241457+0000","last_active":"2023-07-14T19:15:59.722302+0000","last_peered":"2023-07-14T19:15:59.722302+0000","last_clean":"2023-07-14T19:15:59.722302+0000","last_became_active":"2023-07-14T19:15:58.241113+0000","last_became_peered":"2023-07-14T19:15:58.241113+0000","last_unstale":"2023-07-14T19:15:59.722302+0000","last_undegraded":"2023-07-14T19:15:59.722302+0000","last_fullsized":"2023-07-14T19:15:59.722302+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_clean_scrub_stamp":"2023-07-14T19:15:57.219223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T21:28:59.714151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"81'6","reported_seq":23,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-07-14T19:16:05.632583+0000","last_change":"2023-07-14T19:15:57.235057+0000","last_active":"2023-07-14T19:16:05.632583+0000","last_peered":"2023-07-14T19:16:05.632583+0000","last_clean":"2023-07-14T19:16:05.632583+0000","last_became_active":"2023-07-14T19:15:57.234751+0000","last_became_peered":"2023-07-14T19:15:57.234751+0000","last_unstale":"2023-07-14T19:16:05.632583+0000","last_undegraded":"2023-07-14T19:16:05.632583+0000","last_fullsized":"2023-07-14T19:16:05.632583+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:15:56.210166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:15:56.210166+0000","last_clean_scrub_stamp":"2023-07-14T19:15:56.210166+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:12:00.277001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":336,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-07-14T19:15:59.721531+0000","last_change":"2023-07-14T19:14:58.969281+0000","last_active":"2023-07-14T19:15:59.721531+0000","last_peered":"2023-07-14T19:15:59.721531+0000","last_clean":"2023-07-14T19:15:59.721531+0000","last_became_active":"2023-07-14T19:14:58.968767+0000","last_became_peered":"2023-07-14T19:14:58.968767+0000","last_unstale":"2023-07-14T19:15:59.721531+0000","last_undegraded":"2023-07-14T19:15:59.721531+0000","last_fullsized":"2023-07-14T19:15:59.721531+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:12:03.589019+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:12:03.589019+0000","last_clean_scrub_stamp":"2023-07-14T19:12:03.589019+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T05:39:04.087637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-07-14T19:15:59.722476+0000","last_change":"2023-07-14T19:15:58.250737+0000","last_active":"2023-07-14T19:15:59.722476+0000","last_peered":"2023-07-14T19:15:59.722476+0000","last_clean":"2023-07-14T19:15:59.722476+0000","last_became_active":"2023-07-14T19:15:58.250379+0000","last_became_peered":"2023-07-14T19:15:58.250379+0000","last_unstale":"2023-07-14T19:15:59.722476+0000","last_undegraded":"2023-07-14T19:15:59.722476+0000","last_fullsized":"2023-07-14T19:15:59.722476+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_clean_scrub_stamp":"2023-07-14T19:15:57.219223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T19:57:17.147937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"81'4","reported_seq":18,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-07-14T19:16:02.592793+0000","last_change":"2023-07-14T19:15:57.236799+0000","last_active":"2023-07-14T19:16:02.592793+0000","last_peered":"2023-07-14T19:16:02.592793+0000","last_clean":"2023-07-14T19:16:02.592793+0000","last_became_active":"2023-07-14T19:15:57.236503+0000","last_became_peered":"2023-07-14T19:15:57.236503+0000","last_unstale":"2023-07-14T19:16:02.592793+0000","last_undegraded":"2023-07-14T19:16:02.592793+0000","last_fullsized":"2023-07-14T19:16:02.592793+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:15:56.210166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:15:56.210166+0000","last_clean_scrub_stamp":"2023-07-14T19:15:56.210166+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T00:26:24.967329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"81'4","reported_seq":19,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-07-14T19:16:04.618843+0000","last_change":"2023-07-14T19:15:57.245788+0000","last_active":"2023-07-14T19:16:04.618843+0000","last_peered":"2023-07-14T19:16:04.618843+0000","last_clean":"2023-07-14T19:16:04.618843+0000","last_became_active":"2023-07-14T19:15:57.244749+0000","last_became_peered":"2023-07-14T19:15:57.244749+0000","last_unstale":"2023-07-14T19:16:04.618843+0000","last_undegraded":"2023-07-14T19:16:04.618843+0000","last_fullsized":"2023-07-14T19:16:04.618843+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:15:56.210166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:15:56.210166+0000","last_clean_scrub_stamp":"2023-07-14T19:15:56.210166+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T00:56:58.078952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-07-14T19:15:59.722962+0000","last_change":"2023-07-14T19:15:58.250973+0000","last_active":"2023-07-14T19:15:59.722962+0000","last_peered":"2023-07-14T19:15:59.722962+0000","last_clean":"2023-07-14T19:15:59.722962+0000","last_became_active":"2023-07-14T19:15:58.250634+0000","last_became_peered":"2023-07-14T19:15:58.250634+0000","last_unstale":"2023-07-14T19:15:59.722962+0000","last_undegraded":"2023-07-14T19:15:59.722962+0000","last_fullsized":"2023-07-14T19:15:59.722962+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_clean_scrub_stamp":"2023-07-14T19:15:57.219223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T06:14:32.307565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"81'9","reported_seq":23,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-07-14T19:16:08.026032+0000","last_change":"2023-07-14T19:15:57.234996+0000","last_active":"2023-07-14T19:16:08.026032+0000","last_peered":"2023-07-14T19:16:08.026032+0000","last_clean":"2023-07-14T19:16:08.026032+0000","last_became_active":"2023-07-14T19:15:57.234697+0000","last_became_peered":"2023-07-14T19:15:57.234697+0000","last_unstale":"2023-07-14T19:16:08.026032+0000","last_undegraded":"2023-07-14T19:16:08.026032+0000","last_fullsized":"2023-07-14T19:16:08.026032+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:15:56.210166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:15:56.210166+0000","last_clean_scrub_stamp":"2023-07-14T19:15:56.210166+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T21:46:18.232628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-07-14T19:15:59.722957+0000","last_change":"2023-07-14T19:15:58.241753+0000","last_active":"2023-07-14T19:15:59.722957+0000","last_peered":"2023-07-14T19:15:59.722957+0000","last_clean":"2023-07-14T19:15:59.722957+0000","last_became_active":"2023-07-14T19:15:58.241485+0000","last_became_peered":"2023-07-14T19:15:58.241485+0000","last_unstale":"2023-07-14T19:15:59.722957+0000","last_undegraded":"2023-07-14T19:15:59.722957+0000","last_fullsized":"2023-07-14T19:15:59.722957+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_clean_scrub_stamp":"2023-07-14T19:15:57.219223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T03:02:48.399750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"81'7","reported_seq":22,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-07-14T19:16:04.618776+0000","last_change":"2023-07-14T19:15:57.240019+0000","last_active":"2023-07-14T19:16:04.618776+0000","last_peered":"2023-07-14T19:16:04.618776+0000","last_clean":"2023-07-14T19:16:04.618776+0000","last_became_active":"2023-07-14T19:15:57.239521+0000","last_became_peered":"2023-07-14T19:15:57.239521+0000","last_unstale":"2023-07-14T19:16:04.618776+0000","last_undegraded":"2023-07-14T19:16:04.618776+0000","last_fullsized":"2023-07-14T19:16:04.618776+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:15:56.210166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:15:56.210166+0000","last_clean_scrub_stamp":"2023-07-14T19:15:56.210166+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T23:32:40.283168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-07-14T19:15:59.573249+0000","last_change":"2023-07-14T19:15:58.248187+0000","last_active":"2023-07-14T19:15:59.573249+0000","last_peered":"2023-07-14T19:15:59.573249+0000","last_clean":"2023-07-14T19:15:59.573249+0000","last_became_active":"2023-07-14T19:15:58.247854+0000","last_became_peered":"2023-07-14T19:15:58.247854+0000","last_unstale":"2023-07-14T19:15:59.573249+0000","last_undegraded":"2023-07-14T19:15:59.573249+0000","last_fullsized":"2023-07-14T19:15:59.573249+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_clean_scrub_stamp":"2023-07-14T19:15:57.219223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T21:01:16.786236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"81'2","reported_seq":16,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-07-14T19:16:01.572678+0000","last_change":"2023-07-14T19:15:57.245012+0000","last_active":"2023-07-14T19:16:01.572678+0000","last_peered":"2023-07-14T19:16:01.572678+0000","last_clean":"2023-07-14T19:16:01.572678+0000","last_became_active":"2023-07-14T19:15:57.244776+0000","last_became_peered":"2023-07-14T19:15:57.244776+0000","last_unstale":"2023-07-14T19:16:01.572678+0000","last_undegraded":"2023-07-14T19:16:01.572678+0000","last_fullsized":"2023-07-14T19:16:01.572678+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:15:56.210166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:15:56.210166+0000","last_clean_scrub_stamp":"2023-07-14T19:15:56.210166+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T20:16:41.721374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-07-14T19:15:59.232170+0000","last_change":"2023-07-14T19:15:58.247704+0000","last_active":"2023-07-14T19:15:59.232170+0000","last_peered":"2023-07-14T19:15:59.232170+0000","last_clean":"2023-07-14T19:15:59.232170+0000","last_became_active":"2023-07-14T19:15:58.247364+0000","last_became_peered":"2023-07-14T19:15:58.247364+0000","last_unstale":"2023-07-14T19:15:59.232170+0000","last_undegraded":"2023-07-14T19:15:59.232170+0000","last_fullsized":"2023-07-14T19:15:59.232170+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_clean_scrub_stamp":"2023-07-14T19:15:57.219223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T19:44:25.096100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"81'7","reported_seq":21,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-07-14T19:16:04.619759+0000","last_change":"2023-07-14T19:15:57.234991+0000","last_active":"2023-07-14T19:16:04.619759+0000","last_peered":"2023-07-14T19:16:04.619759+0000","last_clean":"2023-07-14T19:16:04.619759+0000","last_became_active":"2023-07-14T19:15:57.234680+0000","last_became_peered":"2023-07-14T19:15:57.234680+0000","last_unstale":"2023-07-14T19:16:04.619759+0000","last_undegraded":"2023-07-14T19:16:04.619759+0000","last_fullsized":"2023-07-14T19:16:04.619759+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:15:56.210166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:15:56.210166+0000","last_clean_scrub_stamp":"2023-07-14T19:15:56.210166+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T03:09:07.479398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-07-14T19:15:59.938707+0000","last_change":"2023-07-14T19:15:58.247224+0000","last_active":"2023-07-14T19:15:59.938707+0000","last_peered":"2023-07-14T19:15:59.938707+0000","last_clean":"2023-07-14T19:15:59.938707+0000","last_became_active":"2023-07-14T19:15:58.246889+0000","last_became_peered":"2023-07-14T19:15:58.246889+0000","last_unstale":"2023-07-14T19:15:59.938707+0000","last_undegraded":"2023-07-14T19:15:59.938707+0000","last_fullsized":"2023-07-14T19:15:59.938707+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_clean_scrub_stamp":"2023-07-14T19:15:57.219223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:25:39.860559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"81'3","reported_seq":17,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-07-14T19:16:05.868855+0000","last_change":"2023-07-14T19:15:57.246491+0000","last_active":"2023-07-14T19:16:05.868855+0000","last_peered":"2023-07-14T19:16:05.868855+0000","last_clean":"2023-07-14T19:16:05.868855+0000","last_became_active":"2023-07-14T19:15:57.246178+0000","last_became_peered":"2023-07-14T19:15:57.246178+0000","last_unstale":"2023-07-14T19:16:05.868855+0000","last_undegraded":"2023-07-14T19:16:05.868855+0000","last_fullsized":"2023-07-14T19:16:05.868855+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:15:56.210166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:15:56.210166+0000","last_clean_scrub_stamp":"2023-07-14T19:15:56.210166+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T03:15:13.275825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-07-14T19:15:59.723335+0000","last_change":"2023-07-14T19:15:58.247005+0000","last_active":"2023-07-14T19:15:59.723335+0000","last_peered":"2023-07-14T19:15:59.723335+0000","last_clean":"2023-07-14T19:15:59.723335+0000","last_became_active":"2023-07-14T19:15:58.246156+0000","last_became_peered":"2023-07-14T19:15:58.246156+0000","last_unstale":"2023-07-14T19:15:59.723335+0000","last_undegraded":"2023-07-14T19:15:59.723335+0000","last_fullsized":"2023-07-14T19:15:59.723335+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_clean_scrub_stamp":"2023-07-14T19:15:57.219223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T20:11:25.410164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"81'4","reported_seq":19,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-07-14T19:16:01.572721+0000","last_change":"2023-07-14T19:15:57.245551+0000","last_active":"2023-07-14T19:16:01.572721+0000","last_peered":"2023-07-14T19:16:01.572721+0000","last_clean":"2023-07-14T19:16:01.572721+0000","last_became_active":"2023-07-14T19:15:57.245179+0000","last_became_peered":"2023-07-14T19:15:57.245179+0000","last_unstale":"2023-07-14T19:16:01.572721+0000","last_undegraded":"2023-07-14T19:16:01.572721+0000","last_fullsized":"2023-07-14T19:16:01.572721+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:15:56.210166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:15:56.210166+0000","last_clean_scrub_stamp":"2023-07-14T19:15:56.210166+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T00:35:19.482494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-07-14T19:15:59.722416+0000","last_change":"2023-07-14T19:15:58.250454+0000","last_active":"2023-07-14T19:15:59.722416+0000","last_peered":"2023-07-14T19:15:59.722416+0000","last_clean":"2023-07-14T19:15:59.722416+0000","last_became_active":"2023-07-14T19:15:58.250145+0000","last_became_peered":"2023-07-14T19:15:58.250145+0000","last_unstale":"2023-07-14T19:15:59.722416+0000","last_undegraded":"2023-07-14T19:15:59.722416+0000","last_fullsized":"2023-07-14T19:15:59.722416+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_clean_scrub_stamp":"2023-07-14T19:15:57.219223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T23:46:27.565214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"81'5","reported_seq":59,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-07-14T19:16:41.699844+0000","last_change":"2023-07-14T19:15:57.247545+0000","last_active":"2023-07-14T19:16:41.699844+0000","last_peered":"2023-07-14T19:16:41.699844+0000","last_clean":"2023-07-14T19:16:41.699844+0000","last_became_active":"2023-07-14T19:15:57.247201+0000","last_became_peered":"2023-07-14T19:15:57.247201+0000","last_unstale":"2023-07-14T19:16:41.699844+0000","last_undegraded":"2023-07-14T19:16:41.699844+0000","last_fullsized":"2023-07-14T19:16:41.699844+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:15:56.210166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:15:56.210166+0000","last_clean_scrub_stamp":"2023-07-14T19:15:56.210166+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T23:52:58.892773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1122,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":40,"num_read_kb":2,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-07-14T19:15:59.722674+0000","last_change":"2023-07-14T19:15:58.252144+0000","last_active":"2023-07-14T19:15:59.722674+0000","last_peered":"2023-07-14T19:15:59.722674+0000","last_clean":"2023-07-14T19:15:59.722674+0000","last_became_active":"2023-07-14T19:15:58.251844+0000","last_became_peered":"2023-07-14T19:15:58.251844+0000","last_unstale":"2023-07-14T19:15:59.722674+0000","last_undegraded":"2023-07-14T19:15:59.722674+0000","last_fullsized":"2023-07-14T19:15:59.722674+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_clean_scrub_stamp":"2023-07-14T19:15:57.219223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:19:09.690589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"81'3","reported_seq":18,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-07-14T19:16:04.617360+0000","last_change":"2023-07-14T19:15:57.239560+0000","last_active":"2023-07-14T19:16:04.617360+0000","last_peered":"2023-07-14T19:16:04.617360+0000","last_clean":"2023-07-14T19:16:04.617360+0000","last_became_active":"2023-07-14T19:15:57.239250+0000","last_became_peered":"2023-07-14T19:15:57.239250+0000","last_unstale":"2023-07-14T19:16:04.617360+0000","last_undegraded":"2023-07-14T19:16:04.617360+0000","last_fullsized":"2023-07-14T19:16:04.617360+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:15:56.210166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:15:56.210166+0000","last_clean_scrub_stamp":"2023-07-14T19:15:56.210166+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T21:04:10.055051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"81'1","reported_seq":13,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-07-14T19:16:01.578417+0000","last_change":"2023-07-14T19:15:58.242144+0000","last_active":"2023-07-14T19:16:01.578417+0000","last_peered":"2023-07-14T19:16:01.578417+0000","last_clean":"2023-07-14T19:16:01.578417+0000","last_became_active":"2023-07-14T19:15:58.241597+0000","last_became_peered":"2023-07-14T19:15:58.241597+0000","last_unstale":"2023-07-14T19:16:01.578417+0000","last_undegraded":"2023-07-14T19:16:01.578417+0000","last_fullsized":"2023-07-14T19:16:01.578417+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_clean_scrub_stamp":"2023-07-14T19:15:57.219223+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T22:43:21.380478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":123,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"81'5","reported_seq":19,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-07-14T19:16:04.616641+0000","last_change":"2023-07-14T19:15:57.247059+0000","last_active":"2023-07-14T19:16:04.616641+0000","last_peered":"2023-07-14T19:16:04.616641+0000","last_clean":"2023-07-14T19:16:04.616641+0000","last_became_active":"2023-07-14T19:15:57.246736+0000","last_became_peered":"2023-07-14T19:15:57.246736+0000","last_unstale":"2023-07-14T19:16:04.616641+0000","last_undegraded":"2023-07-14T19:16:04.616641+0000","last_fullsized":"2023-07-14T19:16:04.616641+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:15:56.210166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:15:56.210166+0000","last_clean_scrub_stamp":"2023-07-14T19:15:56.210166+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T20:57:38.014630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-07-14T19:15:59.722005+0000","last_change":"2023-07-14T19:15:58.244180+0000","last_active":"2023-07-14T19:15:59.722005+0000","last_peered":"2023-07-14T19:15:59.722005+0000","last_clean":"2023-07-14T19:15:59.722005+0000","last_became_active":"2023-07-14T19:15:58.243885+0000","last_became_peered":"2023-07-14T19:15:58.243885+0000","last_unstale":"2023-07-14T19:15:59.722005+0000","last_undegraded":"2023-07-14T19:15:59.722005+0000","last_fullsized":"2023-07-14T19:15:59.722005+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_clean_scrub_stamp":"2023-07-14T19:15:57.219223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:56:30.077276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-07-14T19:15:59.938781+0000","last_change":"2023-07-14T19:15:58.246944+0000","last_active":"2023-07-14T19:15:59.938781+0000","last_peered":"2023-07-14T19:15:59.938781+0000","last_clean":"2023-07-14T19:15:59.938781+0000","last_became_active":"2023-07-14T19:15:58.246672+0000","last_became_peered":"2023-07-14T19:15:58.246672+0000","last_unstale":"2023-07-14T19:15:59.938781+0000","last_undegraded":"2023-07-14T19:15:59.938781+0000","last_fullsized":"2023-07-14T19:15:59.938781+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_clean_scrub_stamp":"2023-07-14T19:15:57.219223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:59:58.787272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-07-14T19:15:59.720933+0000","last_change":"2023-07-14T19:15:58.246761+0000","last_active":"2023-07-14T19:15:59.720933+0000","last_peered":"2023-07-14T19:15:59.720933+0000","last_clean":"2023-07-14T19:15:59.720933+0000","last_became_active":"2023-07-14T19:15:58.246318+0000","last_became_peered":"2023-07-14T19:15:58.246318+0000","last_unstale":"2023-07-14T19:15:59.720933+0000","last_undegraded":"2023-07-14T19:15:59.720933+0000","last_fullsized":"2023-07-14T19:15:59.720933+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_clean_scrub_stamp":"2023-07-14T19:15:57.219223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T22:22:00.801887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-07-14T19:15:59.721079+0000","last_change":"2023-07-14T19:15:58.251465+0000","last_active":"2023-07-14T19:15:59.721079+0000","last_peered":"2023-07-14T19:15:59.721079+0000","last_clean":"2023-07-14T19:15:59.721079+0000","last_became_active":"2023-07-14T19:15:58.251138+0000","last_became_peered":"2023-07-14T19:15:58.251138+0000","last_unstale":"2023-07-14T19:15:59.721079+0000","last_undegraded":"2023-07-14T19:15:59.721079+0000","last_fullsized":"2023-07-14T19:15:59.721079+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_clean_scrub_stamp":"2023-07-14T19:15:57.219223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:53:28.483753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-07-14T19:15:59.573084+0000","last_change":"2023-07-14T19:15:58.244335+0000","last_active":"2023-07-14T19:15:59.573084+0000","last_peered":"2023-07-14T19:15:59.573084+0000","last_clean":"2023-07-14T19:15:59.573084+0000","last_became_active":"2023-07-14T19:15:58.243955+0000","last_became_peered":"2023-07-14T19:15:58.243955+0000","last_unstale":"2023-07-14T19:15:59.573084+0000","last_undegraded":"2023-07-14T19:15:59.573084+0000","last_fullsized":"2023-07-14T19:15:59.573084+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_clean_scrub_stamp":"2023-07-14T19:15:57.219223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T06:54:43.644336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-07-14T19:15:59.723074+0000","last_change":"2023-07-14T19:15:58.244373+0000","last_active":"2023-07-14T19:15:59.723074+0000","last_peered":"2023-07-14T19:15:59.723074+0000","last_clean":"2023-07-14T19:15:59.723074+0000","last_became_active":"2023-07-14T19:15:58.244042+0000","last_became_peered":"2023-07-14T19:15:58.244042+0000","last_unstale":"2023-07-14T19:15:59.723074+0000","last_undegraded":"2023-07-14T19:15:59.723074+0000","last_fullsized":"2023-07-14T19:15:59.723074+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_clean_scrub_stamp":"2023-07-14T19:15:57.219223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T03:40:02.041528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-07-14T19:15:59.938908+0000","last_change":"2023-07-14T19:15:58.240770+0000","last_active":"2023-07-14T19:15:59.938908+0000","last_peered":"2023-07-14T19:15:59.938908+0000","last_clean":"2023-07-14T19:15:59.938908+0000","last_became_active":"2023-07-14T19:15:58.240408+0000","last_became_peered":"2023-07-14T19:15:58.240408+0000","last_unstale":"2023-07-14T19:15:59.938908+0000","last_undegraded":"2023-07-14T19:15:59.938908+0000","last_fullsized":"2023-07-14T19:15:59.938908+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_clean_scrub_stamp":"2023-07-14T19:15:57.219223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T19:45:10.482496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-07-14T19:15:59.721267+0000","last_change":"2023-07-14T19:15:58.247557+0000","last_active":"2023-07-14T19:15:59.721267+0000","last_peered":"2023-07-14T19:15:59.721267+0000","last_clean":"2023-07-14T19:15:59.721267+0000","last_became_active":"2023-07-14T19:15:58.247228+0000","last_became_peered":"2023-07-14T19:15:58.247228+0000","last_unstale":"2023-07-14T19:15:59.721267+0000","last_undegraded":"2023-07-14T19:15:59.721267+0000","last_fullsized":"2023-07-14T19:15:59.721267+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_clean_scrub_stamp":"2023-07-14T19:15:57.219223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T05:18:52.409847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-07-14T19:15:59.722818+0000","last_change":"2023-07-14T19:15:58.247013+0000","last_active":"2023-07-14T19:15:59.722818+0000","last_peered":"2023-07-14T19:15:59.722818+0000","last_clean":"2023-07-14T19:15:59.722818+0000","last_became_active":"2023-07-14T19:15:58.246729+0000","last_became_peered":"2023-07-14T19:15:58.246729+0000","last_unstale":"2023-07-14T19:15:59.722818+0000","last_undegraded":"2023-07-14T19:15:59.722818+0000","last_fullsized":"2023-07-14T19:15:59.722818+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_clean_scrub_stamp":"2023-07-14T19:15:57.219223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T22:17:27.678424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-07-14T19:15:59.231044+0000","last_change":"2023-07-14T19:15:58.243023+0000","last_active":"2023-07-14T19:15:59.231044+0000","last_peered":"2023-07-14T19:15:59.231044+0000","last_clean":"2023-07-14T19:15:59.231044+0000","last_became_active":"2023-07-14T19:15:58.242630+0000","last_became_peered":"2023-07-14T19:15:58.242630+0000","last_unstale":"2023-07-14T19:15:59.231044+0000","last_undegraded":"2023-07-14T19:15:59.231044+0000","last_fullsized":"2023-07-14T19:15:59.231044+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_clean_scrub_stamp":"2023-07-14T19:15:57.219223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T23:27:40.017638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-07-14T19:15:59.723119+0000","last_change":"2023-07-14T19:15:58.246205+0000","last_active":"2023-07-14T19:15:59.723119+0000","last_peered":"2023-07-14T19:15:59.723119+0000","last_clean":"2023-07-14T19:15:59.723119+0000","last_became_active":"2023-07-14T19:15:58.245720+0000","last_became_peered":"2023-07-14T19:15:58.245720+0000","last_unstale":"2023-07-14T19:15:59.723119+0000","last_undegraded":"2023-07-14T19:15:59.723119+0000","last_fullsized":"2023-07-14T19:15:59.723119+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_clean_scrub_stamp":"2023-07-14T19:15:57.219223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T05:22:46.730021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-07-14T19:15:59.722477+0000","last_change":"2023-07-14T19:15:58.248611+0000","last_active":"2023-07-14T19:15:59.722477+0000","last_peered":"2023-07-14T19:15:59.722477+0000","last_clean":"2023-07-14T19:15:59.722477+0000","last_became_active":"2023-07-14T19:15:58.248352+0000","last_became_peered":"2023-07-14T19:15:58.248352+0000","last_unstale":"2023-07-14T19:15:59.722477+0000","last_undegraded":"2023-07-14T19:15:59.722477+0000","last_fullsized":"2023-07-14T19:15:59.722477+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_clean_scrub_stamp":"2023-07-14T19:15:57.219223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T06:43:46.682942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-07-14T19:15:59.722246+0000","last_change":"2023-07-14T19:15:58.243167+0000","last_active":"2023-07-14T19:15:59.722246+0000","last_peered":"2023-07-14T19:15:59.722246+0000","last_clean":"2023-07-14T19:15:59.722246+0000","last_became_active":"2023-07-14T19:15:58.241770+0000","last_became_peered":"2023-07-14T19:15:58.241770+0000","last_unstale":"2023-07-14T19:15:59.722246+0000","last_undegraded":"2023-07-14T19:15:59.722246+0000","last_fullsized":"2023-07-14T19:15:59.722246+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_clean_scrub_stamp":"2023-07-14T19:15:57.219223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T05:56:34.485106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-07-14T19:15:59.938811+0000","last_change":"2023-07-14T19:15:58.243972+0000","last_active":"2023-07-14T19:15:59.938811+0000","last_peered":"2023-07-14T19:15:59.938811+0000","last_clean":"2023-07-14T19:15:59.938811+0000","last_became_active":"2023-07-14T19:15:58.243685+0000","last_became_peered":"2023-07-14T19:15:58.243685+0000","last_unstale":"2023-07-14T19:15:59.938811+0000","last_undegraded":"2023-07-14T19:15:59.938811+0000","last_fullsized":"2023-07-14T19:15:59.938811+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_clean_scrub_stamp":"2023-07-14T19:15:57.219223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:44:54.955614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":246,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":442368,"data_stored":4722944,"data_compressed":280807,"data_compressed_allocated":360448,"data_compressed_original":4669440,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":63176,"num_objects":98,"num_object_clones":0,"num_object_copies":294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":0,"num_read":398,"num_read_kb":267,"num_write":227,"num_write_kb":124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":552960,"data_stored":189528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":199,"ondisk_log_size":199,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":76,"seq":326417514517,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27876,"kb_used_data":860,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717276,"statfs":{"total":95995035648,"available":95966490624,"internally_reserved":0,"allocated":880640,"data_stored":417064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jul 14 19:16:16 2023","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":0.56699999999999995,"15min":0.56699999999999995},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":1.1160000000000001,"5min":1.1160000000000001,"15min":1.1160000000000001},"last":0.45000000000000001},{"interface":"front","average":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":1.2150000000000001,"5min":1.2150000000000001,"15min":1.2150000000000001},"last":0.40600000000000003}]},{"osd":1,"last update":"Fri Jul 14 19:16:16 2023","interfaces":[{"interface":"back","average":{"1min":0.64000000000000001,"5min":0.64000000000000001,"15min":0.64000000000000001},"min":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":1.089,"5min":1.089,"15min":1.089},"last":0.752},{"interface":"front","average":{"1min":0.61899999999999999,"5min":0.61899999999999999,"15min":0.61899999999999999},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":1.1779999999999999,"5min":1.1779999999999999,"15min":1.1779999999999999},"last":0.55000000000000004}]},{"osd":2,"last update":"Fri Jul 14 19:16:16 2023","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.496,"15min":0.496},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.79700000000000004,"5min":0.79700000000000004,"15min":0.79700000000000004},"last":0.48499999999999999},{"interface":"front","average":{"1min":0.47999999999999998,"5min":0.47999999999999998,"15min":0.47999999999999998},"min":{"1min":0.30599999999999999,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.83699999999999997,"5min":0.83699999999999997,"15min":0.83699999999999997},"last":0.56599999999999995}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.436}]},{"osd":4,"last update":"Fri Jul 14 19:16:16 2023","interfaces":[{"interface":"back","average":{"1min":0.70399999999999996,"5min":0.70399999999999996,"15min":0.70399999999999996},"min":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":1.3919999999999999,"5min":1.3919999999999999,"15min":1.3919999999999999},"last":0.66700000000000004},{"interface":"front","average":{"1min":0.66600000000000004,"5min":0.66600000000000004,"15min":0.66600000000000004},"min":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":1.236,"5min":1.236,"15min":1.236},"last":0.58499999999999996}]},{"osd":5,"last update":"Fri Jul 14 19:16:16 2023","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":0.61599999999999999,"15min":0.61599999999999999},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":1.107,"5min":1.107,"15min":1.107},"last":0.60299999999999998},{"interface":"front","average":{"1min":0.60699999999999998,"5min":0.60699999999999998,"15min":0.60699999999999998},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.93700000000000006,"5min":0.93700000000000006,"15min":0.93700000000000006},"last":0.624}]},{"osd":6,"last update":"Fri Jul 14 19:16:16 2023","interfaces":[{"interface":"back","average":{"1min":0.66600000000000004,"5min":0.66600000000000004,"15min":0.66600000000000004},"min":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":1.196,"5min":1.196,"15min":1.196},"last":0.50800000000000001},{"interface":"front","average":{"1min":0.72199999999999998,"5min":0.72199999999999998,"15min":0.72199999999999998},"min":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"max":{"1min":1.379,"5min":1.379,"15min":1.379},"last":0.65000000000000002}]},{"osd":7,"last update":"Fri Jul 14 19:16:16 2023","interfaces":[{"interface":"back","average":{"1min":0.75700000000000001,"5min":0.75700000000000001,"15min":0.75700000000000001},"min":{"1min":0.54700000000000004,"5min":0.54700000000000004,"15min":0.54700000000000004},"max":{"1min":1.071,"5min":1.071,"15min":1.071},"last":0.70499999999999996},{"interface":"front","average":{"1min":0.75,"5min":0.75,"15min":0.75},"min":{"1min":0.57599999999999996,"5min":0.57599999999999996,"15min":0.57599999999999996},"max":{"1min":1.1259999999999999,"5min":1.1259999999999999,"15min":1.1259999999999999},"last":0.68500000000000005}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71899999999999997}]},{"osd":9,"last update":"Fri Jul 14 19:16:16 2023","interfaces":[{"interface":"back","average":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"min":{"1min":0.56100000000000005,"5min":0.56100000000000005,"15min":0.56100000000000005},"max":{"1min":1.462,"5min":1.462,"15min":1.462},"last":0.76600000000000001},{"interface":"front","average":{"1min":0.77200000000000002,"5min":0.77200000000000002,"15min":0.77200000000000002},"min":{"1min":0.56299999999999994,"5min":0.56299999999999994,"15min":0.56299999999999994},"max":{"1min":1.329,"5min":1.329,"15min":1.329},"last":0.82599999999999996}]},{"osd":10,"last update":"Fri Jul 14 19:16:16 2023","interfaces":[{"interface":"back","average":{"1min":0.76500000000000001,"5min":0.76500000000000001,"15min":0.76500000000000001},"min":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"max":{"1min":1.091,"5min":1.091,"15min":1.091},"last":0.73299999999999998},{"interface":"front","average":{"1min":0.82199999999999995,"5min":0.82199999999999995,"15min":0.82199999999999995},"min":{"1min":0.59399999999999997,"5min":0.59399999999999997,"15min":0.59399999999999997},"max":{"1min":1.5049999999999999,"5min":1.5049999999999999,"15min":1.5049999999999999},"last":0.80400000000000005}]}]},{"osd":10,"up_from":69,"seq":296352743448,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28024,"kb_used_data":1008,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717128,"statfs":{"total":95995035648,"available":95966339072,"internally_reserved":0,"allocated":1032192,"data_stored":1065045,"data_compressed":35714,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53700000000000003}]},{"osd":1,"last update":"Fri Jul 14 19:16:00 2023","interfaces":[{"interface":"back","average":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.66500000000000004,"5min":0.66500000000000004,"15min":0.66500000000000004},"last":0.42799999999999999},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.76900000000000002,"5min":0.76900000000000002,"15min":0.76900000000000002},"last":0.47699999999999998}]},{"osd":2,"last update":"Fri Jul 14 19:16:00 2023","interfaces":[{"interface":"back","average":{"1min":0.67100000000000004,"5min":0.67100000000000004,"15min":0.67100000000000004},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":1.0009999999999999,"5min":1.0009999999999999,"15min":1.0009999999999999},"last":0.68600000000000005},{"interface":"front","average":{"1min":0.751,"5min":0.751,"15min":0.751},"min":{"1min":0.51500000000000001,"5min":0.51500000000000001,"15min":0.51500000000000001},"max":{"1min":1.1379999999999999,"5min":1.1379999999999999,"15min":1.1379999999999999},"last":0.72199999999999998}]},{"osd":3,"last update":"Fri Jul 14 19:16:00 2023","interfaces":[{"interface":"back","average":{"1min":0.70099999999999996,"5min":0.70099999999999996,"15min":0.70099999999999996},"min":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":0.92200000000000004,"5min":0.92200000000000004,"15min":0.92200000000000004},"last":0.58099999999999996},{"interface":"front","average":{"1min":0.71599999999999997,"5min":0.71599999999999997,"15min":0.71599999999999997},"min":{"1min":0.45300000000000001,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":0.90900000000000003,"5min":0.90900000000000003,"15min":0.90900000000000003},"last":0.39900000000000002}]},{"osd":4,"last update":"Fri Jul 14 19:16:00 2023","interfaces":[{"interface":"back","average":{"1min":0.64700000000000002,"5min":0.64700000000000002,"15min":0.64700000000000002},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.83199999999999996,"5min":0.83199999999999996,"15min":0.83199999999999996},"last":0.56000000000000005},{"interface":"front","average":{"1min":0.59999999999999998,"5min":0.59999999999999998,"15min":0.59999999999999998},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.878,"5min":0.878,"15min":0.878},"last":0.51800000000000002}]},{"osd":5,"last update":"Fri Jul 14 19:16:00 2023","interfaces":[{"interface":"back","average":{"1min":0.76000000000000001,"5min":0.76000000000000001,"15min":0.76000000000000001},"min":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"max":{"1min":1.1259999999999999,"5min":1.1259999999999999,"15min":1.1259999999999999},"last":0.82899999999999996},{"interface":"front","average":{"1min":0.77900000000000003,"5min":0.77900000000000003,"15min":0.77900000000000003},"min":{"1min":0.56299999999999994,"5min":0.56299999999999994,"15min":0.56299999999999994},"max":{"1min":1.018,"5min":1.018,"15min":1.018},"last":0.76200000000000001}]},{"osd":6,"last update":"Fri Jul 14 19:16:00 2023","interfaces":[{"interface":"back","average":{"1min":0.82799999999999996,"5min":0.82799999999999996,"15min":0.82799999999999996},"min":{"1min":0.56899999999999995,"5min":0.56899999999999995,"15min":0.56899999999999995},"max":{"1min":1.294,"5min":1.294,"15min":1.294},"last":0.66900000000000004},{"interface":"front","average":{"1min":0.80400000000000005,"5min":0.80400000000000005,"15min":0.80400000000000005},"min":{"1min":0.57999999999999996,"5min":0.57999999999999996,"15min":0.57999999999999996},"max":{"1min":1.0960000000000001,"5min":1.0960000000000001,"15min":1.0960000000000001},"last":0.70199999999999996}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.49099999999999999}]},{"osd":8,"last update":"Fri Jul 14 19:16:00 2023","interfaces":[{"interface":"back","average":{"1min":0.84599999999999997,"5min":0.84599999999999997,"15min":0.84599999999999997},"min":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"max":{"1min":1.0349999999999999,"5min":1.0349999999999999,"15min":1.0349999999999999},"last":0.74399999999999999},{"interface":"front","average":{"1min":0.88900000000000001,"5min":0.88900000000000001,"15min":0.88900000000000001},"min":{"1min":0.60499999999999998,"5min":0.60499999999999998,"15min":0.60499999999999998},"max":{"1min":1.369,"5min":1.369,"15min":1.369},"last":0.77700000000000002}]},{"osd":9,"last update":"Fri Jul 14 19:16:00 2023","interfaces":[{"interface":"back","average":{"1min":0.88600000000000001,"5min":0.88600000000000001,"15min":0.88600000000000001},"min":{"1min":0.65000000000000002,"5min":0.65000000000000002,"15min":0.65000000000000002},"max":{"1min":1.1759999999999999,"5min":1.1759999999999999,"15min":1.1759999999999999},"last":0.64700000000000002},{"interface":"front","average":{"1min":0.88200000000000001,"5min":0.88200000000000001,"15min":0.88200000000000001},"min":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":1.468,"5min":1.468,"15min":1.468},"last":0.61199999999999999}]},{"osd":11,"last update":"Fri Jul 14 19:16:18 2023","interfaces":[{"interface":"back","average":{"1min":0.85899999999999999,"5min":0.85899999999999999,"15min":0.85899999999999999},"min":{"1min":0.69499999999999995,"5min":0.69499999999999995,"15min":0.69499999999999995},"max":{"1min":1.105,"5min":1.105,"15min":1.105},"last":0.81399999999999995},{"interface":"front","average":{"1min":0.82899999999999996,"5min":0.82899999999999996,"15min":0.82899999999999996},"min":{"1min":0.56799999999999995,"5min":0.56799999999999995,"15min":0.56799999999999995},"max":{"1min":1.0489999999999999,"5min":1.0489999999999999,"15min":1.0489999999999999},"last":0.84599999999999997}]}]},{"osd":9,"up_from":64,"seq":274877906973,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27900,"kb_used_data":884,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717252,"statfs":{"total":95995035648,"available":95966466048,"internally_reserved":0,"allocated":905216,"data_stored":417042,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jul 14 19:16:36 2023","interfaces":[{"interface":"back","average":{"1min":0.41299999999999998,"5min":0.47199999999999998,"15min":0.48199999999999998},"min":{"1min":0.28399999999999997,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.78900000000000003,"5min":0.82099999999999995,"15min":0.82099999999999995},"last":0.42599999999999999},{"interface":"front","average":{"1min":0.41899999999999998,"5min":0.45200000000000001,"15min":0.45800000000000002},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.61499999999999999,"5min":0.65300000000000002,"15min":0.65300000000000002},"last":0.36599999999999999}]},{"osd":1,"last update":"Fri Jul 14 19:16:36 2023","interfaces":[{"interface":"back","average":{"1min":0.58199999999999996,"5min":0.60999999999999999,"15min":0.61499999999999999},"min":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":1.1240000000000001,"5min":1.1240000000000001,"15min":1.1240000000000001},"last":0.47199999999999998},{"interface":"front","average":{"1min":0.60699999999999998,"5min":0.64700000000000002,"15min":0.65300000000000002},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.85399999999999998,"5min":0.85399999999999998,"15min":0.85399999999999998},"last":0.62}]},{"osd":2,"last update":"Fri Jul 14 19:16:36 2023","interfaces":[{"interface":"back","average":{"1min":0.60199999999999998,"5min":0.64600000000000002,"15min":0.65400000000000003},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.79700000000000004,"5min":0.92200000000000004,"15min":0.92200000000000004},"last":0.57399999999999995},{"interface":"front","average":{"1min":0.56100000000000005,"5min":0.63800000000000001,"15min":0.65100000000000002},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":0.71399999999999997,"5min":0.91000000000000003,"15min":0.91000000000000003},"last":0.49299999999999999}]},{"osd":3,"last update":"Fri Jul 14 19:16:36 2023","interfaces":[{"interface":"back","average":{"1min":0.40999999999999998,"5min":0.505,"15min":0.52100000000000002},"min":{"1min":0.30599999999999999,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.58599999999999997,"5min":0.876,"15min":0.876},"last":0.39100000000000001},{"interface":"front","average":{"1min":0.51600000000000001,"5min":0.59599999999999997,"15min":0.60999999999999999},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.73599999999999999,"5min":0.96499999999999997,"15min":0.96499999999999997},"last":0.52900000000000003}]},{"osd":4,"last update":"Fri Jul 14 19:16:36 2023","interfaces":[{"interface":"back","average":{"1min":0.60199999999999998,"5min":0.66800000000000004,"15min":0.67900000000000005},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.78000000000000003,"5min":1.0700000000000001,"15min":1.0700000000000001},"last":0.51800000000000002},{"interface":"front","average":{"1min":0.60799999999999998,"5min":0.65700000000000003,"15min":0.66500000000000004},"min":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":0.754,"5min":0.94099999999999995,"15min":0.94099999999999995},"last":0.54900000000000004}]},{"osd":5,"last update":"Fri Jul 14 19:16:36 2023","interfaces":[{"interface":"back","average":{"1min":0.68600000000000005,"5min":0.749,"15min":0.75900000000000001},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":1.8480000000000001,"5min":1.8480000000000001,"15min":1.8480000000000001},"last":0.58599999999999997},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.69599999999999995,"15min":0.71399999999999997},"min":{"1min":0.46000000000000002,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.81200000000000006,"5min":1.018,"15min":1.018},"last":0.50700000000000001}]},{"osd":6,"last update":"Fri Jul 14 19:16:36 2023","interfaces":[{"interface":"back","average":{"1min":0.71499999999999997,"5min":0.78100000000000003,"15min":0.79200000000000004},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":2.5720000000000001,"5min":2.5720000000000001,"15min":2.5720000000000001},"last":0.67800000000000005},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.73299999999999998,"15min":0.755},"min":{"1min":0.439,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.86199999999999999,"5min":1.145,"15min":1.145},"last":0.56299999999999994}]},{"osd":7,"last update":"Fri Jul 14 19:16:36 2023","interfaces":[{"interface":"back","average":{"1min":0.67700000000000005,"5min":0.76100000000000001,"15min":0.77500000000000002},"min":{"1min":0.55000000000000004,"5min":0.48699999999999999,"15min":0.48699999999999999},"max":{"1min":0.88700000000000001,"5min":1.0800000000000001,"15min":1.0800000000000001},"last":0.63700000000000001},{"interface":"front","average":{"1min":0.73799999999999999,"5min":0.81200000000000006,"15min":0.82399999999999995},"min":{"1min":0.45200000000000001,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":0.999,"5min":1.0469999999999999,"15min":1.0469999999999999},"last":0.71999999999999997}]},{"osd":8,"last update":"Fri Jul 14 19:16:36 2023","interfaces":[{"interface":"back","average":{"1min":0.71499999999999997,"5min":0.83899999999999997,"15min":0.85999999999999999},"min":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"max":{"1min":0.94999999999999996,"5min":1.242,"15min":1.242},"last":0.60899999999999999},{"interface":"front","average":{"1min":0.73699999999999999,"5min":0.80900000000000005,"15min":0.82099999999999995},"min":{"1min":0.59999999999999998,"5min":0.58499999999999996,"15min":0.58499999999999996},"max":{"1min":1.5600000000000001,"5min":1.5600000000000001,"15min":1.5600000000000001},"last":0.65800000000000003}]},{"osd":10,"last update":"Fri Jul 14 19:15:58 2023","interfaces":[{"interface":"back","average":{"1min":0.79200000000000004,"5min":0.79200000000000004,"15min":0.79200000000000004},"min":{"1min":0.54900000000000004,"5min":0.54900000000000004,"15min":0.54900000000000004},"max":{"1min":1.0820000000000001,"5min":1.0820000000000001,"15min":1.0820000000000001},"last":0.75700000000000001},{"interface":"front","average":{"1min":0.79100000000000004,"5min":0.79100000000000004,"15min":0.79100000000000004},"min":{"1min":0.59599999999999997,"5min":0.59599999999999997,"15min":0.59599999999999997},"max":{"1min":1.044,"5min":1.044,"15min":1.044},"last":0.64800000000000002}]}]},{"osd":8,"up_from":57,"seq":244813135906,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27892,"kb_used_data":876,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717260,"statfs":{"total":95995035648,"available":95966474240,"internally_reserved":0,"allocated":897024,"data_stored":417089,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jul 14 19:16:16 2023","interfaces":[{"interface":"back","average":{"1min":0.51500000000000001,"5min":0.51900000000000002,"15min":0.52000000000000002},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.746,"5min":0.78600000000000003,"15min":0.78600000000000003},"last":0.57599999999999996},{"interface":"front","average":{"1min":0.51300000000000001,"5min":0.57099999999999995,"15min":0.58099999999999996},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.72799999999999998,"5min":0.80500000000000005,"15min":0.80500000000000005},"last":0.60499999999999998}]},{"osd":1,"last update":"Fri Jul 14 19:16:16 2023","interfaces":[{"interface":"back","average":{"1min":0.64500000000000002,"5min":0.58999999999999997,"15min":0.58099999999999996},"min":{"1min":0.39700000000000002,"5min":0.375,"15min":0.375},"max":{"1min":0.81200000000000006,"5min":1.0069999999999999,"15min":1.0069999999999999},"last":0.82499999999999996},{"interface":"front","average":{"1min":0.64600000000000002,"5min":0.60299999999999998,"15min":0.59599999999999997},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":0.83399999999999996,"5min":0.91500000000000004,"15min":0.91500000000000004},"last":0.84299999999999997}]},{"osd":2,"last update":"Fri Jul 14 19:16:16 2023","interfaces":[{"interface":"back","average":{"1min":0.45900000000000002,"5min":0.43099999999999999,"15min":0.42599999999999999},"min":{"1min":0.28399999999999997,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.73799999999999999,"5min":0.73799999999999999,"15min":0.73799999999999999},"last":0.63900000000000001},{"interface":"front","average":{"1min":0.51600000000000001,"5min":0.49199999999999999,"15min":0.48799999999999999},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":0.76900000000000002,"5min":0.76900000000000002,"15min":0.76900000000000002},"last":0.754}]},{"osd":3,"last update":"Fri Jul 14 19:16:16 2023","interfaces":[{"interface":"back","average":{"1min":0.55500000000000005,"5min":0.59899999999999998,"15min":0.60599999999999998},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.68100000000000005,"5min":0.84799999999999998,"15min":0.84799999999999998},"last":0.71999999999999997},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.63400000000000001,"15min":0.64300000000000002},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.82599999999999996,"5min":0.82599999999999996,"15min":0.82599999999999996},"last":0.92100000000000004}]},{"osd":4,"last update":"Fri Jul 14 19:16:16 2023","interfaces":[{"interface":"back","average":{"1min":0.64100000000000001,"5min":0.59299999999999997,"15min":0.58499999999999996},"min":{"1min":0.44900000000000001,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.90700000000000003,"5min":1.069,"15min":1.069},"last":0.96299999999999997},{"interface":"front","average":{"1min":0.68799999999999994,"5min":0.68500000000000005,"15min":0.68500000000000005},"min":{"1min":0.46300000000000002,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.99099999999999999,"5min":1.474,"15min":1.474},"last":0.97699999999999998}]},{"osd":5,"last update":"Fri Jul 14 19:16:16 2023","interfaces":[{"interface":"back","average":{"1min":0.58799999999999997,"5min":0.55800000000000005,"15min":0.55300000000000005},"min":{"1min":0.32400000000000001,"5min":0.313,"15min":0.313},"max":{"1min":0.88900000000000001,"5min":0.88900000000000001,"15min":0.88900000000000001},"last":0.94099999999999995},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.56699999999999995,"15min":0.56799999999999995},"min":{"1min":0.33500000000000002,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.77300000000000002,"5min":1.0349999999999999,"15min":1.0349999999999999},"last":0.89900000000000002}]},{"osd":6,"last update":"Fri Jul 14 19:16:16 2023","interfaces":[{"interface":"back","average":{"1min":0.66100000000000003,"5min":0.66300000000000003,"15min":0.66300000000000003},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":0.87,"5min":0.87,"15min":0.87},"last":1.01},{"interface":"front","average":{"1min":0.69399999999999995,"5min":0.72399999999999998,"15min":0.72899999999999998},"min":{"1min":0.439,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":0.94299999999999995,"5min":1.1399999999999999,"15min":1.1399999999999999},"last":0.99399999999999999}]},{"osd":7,"last update":"Fri Jul 14 19:16:16 2023","interfaces":[{"interface":"back","average":{"1min":0.73899999999999999,"5min":0.72799999999999998,"15min":0.72599999999999998},"min":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":1.0289999999999999,"5min":1.1879999999999999,"15min":1.1879999999999999},"last":1.0700000000000001},{"interface":"front","average":{"1min":0.70299999999999996,"5min":0.72299999999999998,"15min":0.72599999999999998},"min":{"1min":0.46800000000000003,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":0.93400000000000005,"5min":1.093,"15min":1.093},"last":1.024}]},{"osd":9,"last update":"Fri Jul 14 19:16:36 2023","interfaces":[{"interface":"back","average":{"1min":0.75,"5min":0.753,"15min":0.753},"min":{"1min":0.47599999999999998,"5min":0.47599999999999998,"15min":0.47599999999999998},"max":{"1min":1.0129999999999999,"5min":1.0129999999999999,"15min":1.0129999999999999},"last":1.034},{"interface":"front","average":{"1min":0.75900000000000001,"5min":0.75600000000000001,"15min":0.75600000000000001},"min":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":0.95399999999999996,"5min":0.97099999999999997,"15min":0.97099999999999997},"last":1.0509999999999999}]},{"osd":10,"last update":"Fri Jul 14 19:15:55 2023","interfaces":[{"interface":"back","average":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"min":{"1min":0.52700000000000002,"5min":0.52700000000000002,"15min":0.52700000000000002},"max":{"1min":1.006,"5min":1.006,"15min":1.006},"last":1.1220000000000001},{"interface":"front","average":{"1min":0.73599999999999999,"5min":0.73599999999999999,"15min":0.73599999999999999},"min":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"max":{"1min":0.95999999999999996,"5min":0.95999999999999996,"15min":0.95999999999999996},"last":1.139}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.099}]}]},{"osd":7,"up_from":50,"seq":214748364837,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27944,"kb_used_data":928,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717208,"statfs":{"total":95995035648,"available":95966420992,"internally_reserved":0,"allocated":950272,"data_stored":1009136,"data_compressed":35714,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jul 14 19:15:51 2023","interfaces":[{"interface":"back","average":{"1min":0.57199999999999995,"5min":0.53300000000000003,"15min":0.52700000000000002},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.82999999999999996,"5min":0.82999999999999996,"15min":0.82999999999999996},"last":0.5},{"interface":"front","average":{"1min":0.55200000000000005,"5min":0.55600000000000005,"15min":0.55600000000000005},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.78100000000000003,"5min":0.89300000000000002,"15min":0.89300000000000002},"last":0.51800000000000002}]},{"osd":1,"last update":"Fri Jul 14 19:15:51 2023","interfaces":[{"interface":"back","average":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"min":{"1min":0.25700000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.67100000000000004,"5min":0.67100000000000004,"15min":0.67100000000000004},"last":0.375},{"interface":"front","average":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"min":{"1min":0.254,"5min":0.254,"15min":0.254},"max":{"1min":0.69099999999999995,"5min":0.91300000000000003,"15min":0.91300000000000003},"last":0.47399999999999998}]},{"osd":2,"last update":"Fri Jul 14 19:15:51 2023","interfaces":[{"interface":"back","average":{"1min":0.60199999999999998,"5min":0.59299999999999997,"15min":0.59099999999999997},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.78600000000000003,"5min":0.83999999999999997,"15min":0.83999999999999997},"last":0.53100000000000003},{"interface":"front","average":{"1min":0.65600000000000003,"5min":0.628,"15min":0.623},"min":{"1min":0.43099999999999999,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":0.96999999999999997,"5min":0.96999999999999997,"15min":0.96999999999999997},"last":0.57799999999999996}]},{"osd":3,"last update":"Fri Jul 14 19:15:51 2023","interfaces":[{"interface":"back","average":{"1min":0.61699999999999999,"5min":0.56599999999999995,"15min":0.55800000000000005},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":0.89200000000000002,"5min":0.89200000000000002,"15min":0.89200000000000002},"last":0.625},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.80300000000000005,"5min":0.91700000000000004,"15min":0.91700000000000004},"last":0.77500000000000002}]},{"osd":4,"last update":"Fri Jul 14 19:15:51 2023","interfaces":[{"interface":"back","average":{"1min":0.53800000000000003,"5min":0.54000000000000004,"15min":0.54000000000000004},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.81499999999999995,"5min":0.86399999999999999,"15min":0.86399999999999999},"last":0.40799999999999997},{"interface":"front","average":{"1min":0.497,"5min":0.48099999999999998,"15min":0.47899999999999998},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":0.80600000000000005,"5min":0.80600000000000005,"15min":0.80600000000000005},"last":0.432}]},{"osd":5,"last update":"Fri Jul 14 19:15:51 2023","interfaces":[{"interface":"back","average":{"1min":0.65100000000000002,"5min":0.67800000000000005,"15min":0.68200000000000005},"min":{"1min":0.43099999999999999,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":0.90600000000000003,"5min":1.355,"15min":1.355},"last":0.63500000000000001},{"interface":"front","average":{"1min":0.73599999999999999,"5min":0.71499999999999997,"15min":0.71099999999999997},"min":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":1.048,"5min":1.3009999999999999,"15min":1.3009999999999999},"last":0.51200000000000001}]},{"osd":6,"last update":"Fri Jul 14 19:15:51 2023","interfaces":[{"interface":"back","average":{"1min":0.69599999999999995,"5min":0.69399999999999995,"15min":0.69399999999999995},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":0.93200000000000005,"5min":1.2310000000000001,"15min":1.2310000000000001},"last":0.751},{"interface":"front","average":{"1min":0.70399999999999996,"5min":0.66800000000000004,"15min":0.66200000000000003},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":1.069,"5min":1.1140000000000001,"15min":1.1140000000000001},"last":0.79100000000000004}]},{"osd":8,"last update":"Fri Jul 14 19:16:14 2023","interfaces":[{"interface":"back","average":{"1min":0.71699999999999997,"5min":0.73799999999999999,"15min":0.74199999999999999},"min":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":1.1319999999999999,"5min":1.1319999999999999,"15min":1.1319999999999999},"last":0.71599999999999997},{"interface":"front","average":{"1min":0.72899999999999998,"5min":0.72899999999999998,"15min":0.72899999999999998},"min":{"1min":0.54700000000000004,"5min":0.495,"15min":0.495},"max":{"1min":0.99199999999999999,"5min":0.99199999999999999,"15min":0.99199999999999999},"last":0.60199999999999998}]},{"osd":9,"last update":"Fri Jul 14 19:16:38 2023","interfaces":[{"interface":"back","average":{"1min":0.61199999999999999,"5min":0.72499999999999998,"15min":0.74399999999999999},"min":{"1min":0.48899999999999999,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":0.79800000000000004,"5min":1.1599999999999999,"15min":1.1599999999999999},"last":0.67000000000000004},{"interface":"front","average":{"1min":0.69299999999999995,"5min":0.76100000000000001,"15min":0.77200000000000002},"min":{"1min":0.499,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":0.90500000000000003,"5min":1.214,"15min":1.214},"last":0.76500000000000001}]},{"osd":10,"last update":"Fri Jul 14 19:15:55 2023","interfaces":[{"interface":"back","average":{"1min":0.753,"5min":0.753,"15min":0.753},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":1.0960000000000001,"5min":1.0960000000000001,"15min":1.0960000000000001},"last":0.56499999999999995},{"interface":"front","average":{"1min":0.748,"5min":0.748,"15min":0.748},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":1.0980000000000001,"5min":1.0980000000000001,"15min":1.0980000000000001},"last":0.65200000000000002}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68500000000000005}]}]},{"osd":6,"up_from":45,"seq":193273528362,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27884,"kb_used_data":868,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717268,"statfs":{"total":95995035648,"available":95966482432,"internally_reserved":0,"allocated":888832,"data_stored":416717,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jul 14 19:16:34 2023","interfaces":[{"interface":"back","average":{"1min":0.39100000000000001,"5min":0.41699999999999998,"15min":0.40999999999999998},"min":{"1min":0.29499999999999998,"5min":0.222,"15min":0.222},"max":{"1min":0.57899999999999996,"5min":0.77100000000000002,"15min":0.77100000000000002},"last":0.38500000000000001},{"interface":"front","average":{"1min":0.40699999999999997,"5min":0.45100000000000001,"15min":0.443},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.68200000000000005,"5min":1.1459999999999999,"15min":1.1459999999999999},"last":0.41799999999999998}]},{"osd":1,"last update":"Fri Jul 14 19:16:34 2023","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.57899999999999996,"15min":0.56399999999999995},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":0.77900000000000003,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.61799999999999999},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.67000000000000004,"15min":0.69399999999999995},"min":{"1min":0.47099999999999997,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":0.85199999999999998,"5min":3.1930000000000001,"15min":3.1930000000000001},"last":0.52100000000000002}]},{"osd":2,"last update":"Fri Jul 14 19:16:34 2023","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":0.59499999999999997,"15min":0.58699999999999997},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.81100000000000005,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.629},{"interface":"front","average":{"1min":0.58199999999999996,"5min":0.58499999999999996,"15min":0.57699999999999996},"min":{"1min":0.40799999999999997,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.79400000000000004,"5min":0.85299999999999998,"15min":0.85299999999999998},"last":0.59299999999999997}]},{"osd":3,"last update":"Fri Jul 14 19:16:34 2023","interfaces":[{"interface":"back","average":{"1min":0.47799999999999998,"5min":0.501,"15min":0.497},"min":{"1min":0.317,"5min":0.314,"15min":0.314},"max":{"1min":0.72699999999999998,"5min":0.82799999999999996,"15min":0.82799999999999996},"last":0.36499999999999999},{"interface":"front","average":{"1min":0.51500000000000001,"5min":0.54200000000000004,"15min":0.54000000000000004},"min":{"1min":0.26700000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.89100000000000001,"5min":0.96199999999999997,"15min":0.96199999999999997},"last":0.45400000000000001}]},{"osd":4,"last update":"Fri Jul 14 19:16:34 2023","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.63500000000000001,"15min":0.625},"min":{"1min":0.42599999999999999,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.84399999999999997,"5min":0.91500000000000004,"15min":0.91500000000000004},"last":0.57199999999999995},{"interface":"front","average":{"1min":0.64500000000000002,"5min":0.66900000000000004,"15min":0.65700000000000003},"min":{"1min":0.48799999999999999,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":0.91000000000000003,"5min":1.0309999999999999,"15min":1.0309999999999999},"last":0.53700000000000003}]},{"osd":5,"last update":"Fri Jul 14 19:16:34 2023","interfaces":[{"interface":"back","average":{"1min":0.64600000000000002,"5min":0.73399999999999999,"15min":0.753},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.82899999999999996,"5min":3.1299999999999999,"15min":3.1299999999999999},"last":0.66100000000000003},{"interface":"front","average":{"1min":0.68400000000000005,"5min":0.69099999999999995,"15min":0.67200000000000004},"min":{"1min":0.41299999999999998,"5min":0.38,"15min":0.38},"max":{"1min":0.92500000000000004,"5min":1.105,"15min":1.105},"last":0.78000000000000003}]},{"osd":7,"last update":"Fri Jul 14 19:15:56 2023","interfaces":[{"interface":"back","average":{"1min":0.79000000000000004,"5min":0.84199999999999997,"15min":0.85099999999999998},"min":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"max":{"1min":1.0489999999999999,"5min":3.25,"15min":3.25},"last":0.55800000000000005},{"interface":"front","average":{"1min":0.79600000000000004,"5min":0.749,"15min":0.74099999999999999},"min":{"1min":0.58299999999999996,"5min":0.54000000000000004,"15min":0.54000000000000004},"max":{"1min":1.0589999999999999,"5min":1.0589999999999999,"15min":1.0589999999999999},"last":0.752}]},{"osd":8,"last update":"Fri Jul 14 19:16:15 2023","interfaces":[{"interface":"back","average":{"1min":0.70199999999999996,"5min":0.73299999999999998,"15min":0.73799999999999999},"min":{"1min":0.437,"5min":0.437,"15min":0.437},"max":{"1min":0.96199999999999997,"5min":1.044,"15min":1.044},"last":0.67700000000000005},{"interface":"front","average":{"1min":0.75600000000000001,"5min":0.91100000000000003,"15min":0.93700000000000006},"min":{"1min":0.60199999999999998,"5min":0.57999999999999996,"15min":0.57999999999999996},"max":{"1min":1.006,"5min":3.226,"15min":3.226},"last":0.70399999999999996}]},{"osd":9,"last update":"Fri Jul 14 19:15:38 2023","interfaces":[{"interface":"back","average":{"1min":0.85099999999999998,"5min":0.85099999999999998,"15min":0.85099999999999998},"min":{"1min":0.502,"5min":0.502,"15min":0.502},"max":{"1min":1.123,"5min":1.123,"15min":1.123},"last":0.68799999999999994},{"interface":"front","average":{"1min":0.83499999999999996,"5min":0.83499999999999996,"15min":0.83499999999999996},"min":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":1.135,"5min":1.135,"15min":1.135},"last":0.48899999999999999}]},{"osd":10,"last update":"Fri Jul 14 19:15:56 2023","interfaces":[{"interface":"back","average":{"1min":0.85899999999999999,"5min":0.85899999999999999,"15min":0.85899999999999999},"min":{"1min":0.63,"5min":0.63,"15min":0.63},"max":{"1min":1.0840000000000001,"5min":1.0840000000000001,"15min":1.0840000000000001},"last":0.60599999999999998},{"interface":"front","average":{"1min":0.85699999999999998,"5min":0.85699999999999998,"15min":0.85699999999999998},"min":{"1min":0.64100000000000001,"5min":0.64100000000000001,"15min":0.64100000000000001},"max":{"1min":1.0820000000000001,"5min":1.0820000000000001,"15min":1.0820000000000001},"last":0.72299999999999998}]}]},{"osd":1,"up_from":13,"seq":55834574911,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27936,"kb_used_data":920,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717216,"statfs":{"total":95995035648,"available":95966429184,"internally_reserved":0,"allocated":942080,"data_stored":474297,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jul 14 19:15:49 2023","interfaces":[{"interface":"back","average":{"1min":0.60199999999999998,"5min":0.51100000000000001,"15min":0.45400000000000001},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.81999999999999995,"5min":1.149,"15min":1.149},"last":0.53000000000000003},{"interface":"front","average":{"1min":0.63500000000000001,"5min":0.52500000000000002,"15min":0.46999999999999997},"min":{"1min":0.36099999999999999,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.97099999999999997,"5min":1.0740000000000001,"15min":1.0740000000000001},"last":0.63100000000000001}]},{"osd":2,"last update":"Fri Jul 14 19:16:09 2023","interfaces":[{"interface":"back","average":{"1min":0.61799999999999999,"5min":0.53400000000000003,"15min":0.51900000000000002},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.94299999999999995,"5min":0.94299999999999995,"15min":0.94299999999999995},"last":0.85599999999999998},{"interface":"front","average":{"1min":0.64000000000000001,"5min":0.54500000000000004,"15min":0.51400000000000001},"min":{"1min":0.45500000000000002,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.83799999999999997,"5min":0.83799999999999997,"15min":0.83799999999999997},"last":0.73199999999999998}]},{"osd":3,"last update":"Fri Jul 14 19:16:30 2023","interfaces":[{"interface":"back","average":{"1min":0.64900000000000002,"5min":0.59999999999999998,"15min":0.57099999999999995},"min":{"1min":0.41699999999999998,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.95199999999999996,"5min":1.1659999999999999,"15min":1.1659999999999999},"last":0.501},{"interface":"front","average":{"1min":0.62,"5min":0.59099999999999997,"15min":0.56799999999999995},"min":{"1min":0.46700000000000003,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.90000000000000002,"5min":1.177,"15min":1.177},"last":0.55200000000000005}]},{"osd":4,"last update":"Fri Jul 14 19:15:49 2023","interfaces":[{"interface":"back","average":{"1min":0.52800000000000002,"5min":0.435,"15min":0.42299999999999999},"min":{"1min":0.32100000000000001,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.77300000000000002,"5min":0.77300000000000002,"15min":0.77300000000000002},"last":0.47199999999999998},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.47099999999999997,"15min":0.45500000000000002},"min":{"1min":0.38500000000000001,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.80300000000000005,"5min":0.80300000000000005,"15min":0.80300000000000005},"last":0.69499999999999995}]},{"osd":5,"last update":"Fri Jul 14 19:16:09 2023","interfaces":[{"interface":"back","average":{"1min":0.56399999999999995,"5min":0.53500000000000003,"15min":0.52600000000000002},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.85099999999999998,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.71599999999999997},{"interface":"front","average":{"1min":0.68200000000000005,"5min":0.60399999999999998,"15min":0.59199999999999997},"min":{"1min":0.47499999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.89900000000000002,"5min":0.89900000000000002,"15min":0.89900000000000002},"last":0.78500000000000003}]},{"osd":6,"last update":"Fri Jul 14 19:16:37 2023","interfaces":[{"interface":"back","average":{"1min":0.85899999999999999,"5min":0.68400000000000005,"15min":0.65200000000000002},"min":{"1min":0.54000000000000004,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":2.7890000000000001,"5min":2.7890000000000001,"15min":2.7890000000000001},"last":0.76800000000000002},{"interface":"front","average":{"1min":0.78700000000000003,"5min":0.65800000000000003,"15min":0.63200000000000001},"min":{"1min":0.52200000000000002,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":2.6899999999999999,"5min":2.6899999999999999,"15min":2.6899999999999999},"last":0.67600000000000005}]},{"osd":7,"last update":"Fri Jul 14 19:15:56 2023","interfaces":[{"interface":"back","average":{"1min":0.65600000000000003,"5min":0.55000000000000004,"15min":0.53200000000000003},"min":{"1min":0.378,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.86799999999999999,"5min":0.86799999999999999,"15min":0.86799999999999999},"last":0.65600000000000003},{"interface":"front","average":{"1min":0.68799999999999994,"5min":0.61199999999999999,"15min":0.59899999999999998},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.93300000000000005,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.82199999999999995}]},{"osd":8,"last update":"Fri Jul 14 19:16:20 2023","interfaces":[{"interface":"back","average":{"1min":0.75800000000000001,"5min":0.67800000000000005,"15min":0.66400000000000003},"min":{"1min":0.502,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":1.036,"5min":1.036,"15min":1.036},"last":0.89300000000000002},{"interface":"front","average":{"1min":0.68000000000000005,"5min":0.64400000000000002,"15min":0.63800000000000001},"min":{"1min":0.495,"5min":0.47999999999999998,"15min":0.47999999999999998},"max":{"1min":0.95499999999999996,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.754}]},{"osd":9,"last update":"Fri Jul 14 19:16:37 2023","interfaces":[{"interface":"back","average":{"1min":0.80400000000000005,"5min":0.74299999999999999,"15min":0.73299999999999998},"min":{"1min":0.58299999999999996,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":1.022,"5min":1.022,"15min":1.022},"last":0.871},{"interface":"front","average":{"1min":0.86899999999999999,"5min":0.77100000000000002,"15min":0.755},"min":{"1min":0.66000000000000003,"5min":0.53900000000000003,"15min":0.53900000000000003},"max":{"1min":1.0740000000000001,"5min":1.0740000000000001,"15min":1.0740000000000001},"last":0.92200000000000004}]},{"osd":10,"last update":"Fri Jul 14 19:15:56 2023","interfaces":[{"interface":"back","average":{"1min":0.65800000000000003,"5min":0.65800000000000003,"15min":0.65800000000000003},"min":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":0.86799999999999999,"5min":0.86799999999999999,"15min":0.86799999999999999},"last":0.80800000000000005},{"interface":"front","average":{"1min":0.72599999999999998,"5min":0.72599999999999998,"15min":0.72599999999999998},"min":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"max":{"1min":0.88300000000000001,"5min":0.88300000000000001,"15min":0.88300000000000001},"last":0.83999999999999997}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.88200000000000001}]}]},{"osd":0,"up_from":8,"seq":34359738436,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27904,"kb_used_data":888,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717248,"statfs":{"total":95995035648,"available":95966461952,"internally_reserved":0,"allocated":909312,"data_stored":418050,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Jul 14 19:15:50 2023","interfaces":[{"interface":"back","average":{"1min":0.627,"5min":0.55900000000000005,"15min":0.52900000000000003},"min":{"1min":0.49099999999999999,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.88,"5min":0.95799999999999996,"15min":0.95799999999999996},"last":0.316},{"interface":"front","average":{"1min":0.64800000000000002,"5min":0.56399999999999995,"15min":0.54200000000000004},"min":{"1min":0.40600000000000003,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.91700000000000004,"5min":0.99099999999999999,"15min":0.99099999999999999},"last":0.45700000000000002}]},{"osd":2,"last update":"Fri Jul 14 19:16:08 2023","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.54100000000000004,"15min":0.51200000000000001},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.94999999999999996,"5min":0.94999999999999996,"15min":0.94999999999999996},"last":0.55500000000000005},{"interface":"front","average":{"1min":0.61899999999999999,"5min":0.56200000000000006,"15min":0.52500000000000002},"min":{"1min":0.37,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.91000000000000003,"5min":1.044,"15min":1.044},"last":0.83499999999999996}]},{"osd":3,"last update":"Fri Jul 14 19:14:30 2023","interfaces":[{"interface":"back","average":{"1min":0.437,"5min":0.40100000000000002,"15min":0.39500000000000002},"min":{"1min":0.315,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.623,"5min":0.623,"15min":0.623},"last":0.34000000000000002},{"interface":"front","average":{"1min":0.497,"5min":0.47299999999999998,"15min":0.46899999999999997},"min":{"1min":0.34100000000000003,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.59999999999999998,"5min":0.63200000000000001,"15min":0.63200000000000001},"last":0.36499999999999999}]},{"osd":4,"last update":"Fri Jul 14 19:15:50 2023","interfaces":[{"interface":"back","average":{"1min":0.65900000000000003,"5min":0.57899999999999996,"15min":0.55100000000000005},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":0.92900000000000005,"5min":0.95399999999999996,"15min":0.95399999999999996},"last":0.41699999999999998},{"interface":"front","average":{"1min":0.63200000000000001,"5min":0.59199999999999997,"15min":0.56999999999999995},"min":{"1min":0.44,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.86099999999999999,"5min":0.93400000000000005,"15min":0.93400000000000005},"last":0.49099999999999999}]},{"osd":5,"last update":"Fri Jul 14 19:16:16 2023","interfaces":[{"interface":"back","average":{"1min":0.63400000000000001,"5min":0.63600000000000001,"15min":0.62},"min":{"1min":0.443,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":0.96899999999999997,"5min":1.0169999999999999,"15min":1.0169999999999999},"last":0.871},{"interface":"front","average":{"1min":0.59999999999999998,"5min":0.61099999999999999,"15min":0.58899999999999997},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.94899999999999995,"5min":1.073,"15min":1.073},"last":0.75900000000000001}]},{"osd":6,"last update":"Fri Jul 14 19:16:36 2023","interfaces":[{"interface":"back","average":{"1min":0.56000000000000005,"5min":0.53800000000000003,"15min":0.53300000000000003},"min":{"1min":0.34100000000000003,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":1.163,"5min":1.163,"15min":1.163},"last":0.65400000000000003},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.62,"15min":0.60199999999999998},"min":{"1min":0.38700000000000001,"5min":0.312,"15min":0.312},"max":{"1min":0.86599999999999999,"5min":1.105,"15min":1.105},"last":0.67600000000000005}]},{"osd":7,"last update":"Fri Jul 14 19:15:55 2023","interfaces":[{"interface":"back","average":{"1min":0.77400000000000002,"5min":0.72499999999999998,"15min":0.71699999999999997},"min":{"1min":0.55700000000000005,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":1.0629999999999999,"5min":1.0629999999999999,"15min":1.0629999999999999},"last":0.81299999999999994},{"interface":"front","average":{"1min":0.68700000000000006,"5min":0.69999999999999996,"15min":0.70199999999999996},"min":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":0.99299999999999999,"5min":1.0229999999999999,"15min":1.0229999999999999},"last":0.77200000000000002}]},{"osd":8,"last update":"Fri Jul 14 19:16:16 2023","interfaces":[{"interface":"back","average":{"1min":0.65300000000000002,"5min":0.68500000000000005,"15min":0.68999999999999995},"min":{"1min":0.42299999999999999,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":1.04,"5min":1.04,"15min":1.04},"last":0.81299999999999994},{"interface":"front","average":{"1min":0.68400000000000005,"5min":0.81299999999999994,"15min":0.83499999999999996},"min":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":1.0900000000000001,"5min":1.097,"15min":1.097},"last":0.91500000000000004}]},{"osd":9,"last update":"Fri Jul 14 19:15:37 2023","interfaces":[{"interface":"back","average":{"1min":0.73099999999999998,"5min":0.73099999999999998,"15min":0.73099999999999998},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":1.111,"5min":1.111,"15min":1.111},"last":0.53700000000000003},{"interface":"front","average":{"1min":0.70399999999999996,"5min":0.70399999999999996,"15min":0.70399999999999996},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.98099999999999998,"5min":0.98099999999999998,"15min":0.98099999999999998},"last":0.51700000000000002}]},{"osd":10,"last update":"Fri Jul 14 19:15:59 2023","interfaces":[{"interface":"back","average":{"1min":0.82799999999999996,"5min":0.82799999999999996,"15min":0.82799999999999996},"min":{"1min":0.66200000000000003,"5min":0.66200000000000003,"15min":0.66200000000000003},"max":{"1min":1.1379999999999999,"5min":1.1379999999999999,"15min":1.1379999999999999},"last":0.745},{"interface":"front","average":{"1min":0.77800000000000002,"5min":0.77800000000000002,"15min":0.77800000000000002},"min":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"max":{"1min":0.99199999999999999,"5min":0.99199999999999999,"15min":0.99199999999999999},"last":0.59799999999999998}]},{"osd":11,"last update":"Fri Jul 14 19:16:16 2023","interfaces":[{"interface":"back","average":{"1min":0.72299999999999998,"5min":0.72299999999999998,"15min":0.72299999999999998},"min":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"max":{"1min":1.0669999999999999,"5min":1.0669999999999999,"15min":1.0669999999999999},"last":0.72899999999999998},{"interface":"front","average":{"1min":0.77300000000000002,"5min":0.77300000000000002,"15min":0.77300000000000002},"min":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"max":{"1min":1.111,"5min":1.111,"15min":1.111},"last":0.89500000000000002}]}]},{"osd":2,"up_from":18,"seq":77309411388,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27960,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717192,"statfs":{"total":95995035648,"available":95966404608,"internally_reserved":0,"allocated":966656,"data_stored":474742,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jul 14 19:16:11 2023","interfaces":[{"interface":"back","average":{"1min":0.47999999999999998,"5min":0.48899999999999999,"15min":0.47899999999999998},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.67300000000000004,"5min":0.76600000000000001,"15min":0.76600000000000001},"last":0.49099999999999999},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.51800000000000002,"15min":0.497},"min":{"1min":0.38600000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.72399999999999998,"5min":0.79500000000000004,"15min":0.79500000000000004},"last":0.41799999999999998}]},{"osd":1,"last update":"Fri Jul 14 19:16:11 2023","interfaces":[{"interface":"back","average":{"1min":0.60399999999999998,"5min":0.53100000000000003,"15min":0.49399999999999999},"min":{"1min":0.44600000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.77000000000000002,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.68700000000000006},{"interface":"front","average":{"1min":0.59699999999999998,"5min":0.53700000000000003,"15min":0.505},"min":{"1min":0.38,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":0.81399999999999995,"5min":0.90700000000000003,"15min":0.90700000000000003},"last":0.78500000000000003}]},{"osd":3,"last update":"Fri Jul 14 19:16:30 2023","interfaces":[{"interface":"back","average":{"1min":1.046,"5min":0.64000000000000001,"15min":0.54900000000000004},"min":{"1min":0.42799999999999999,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":7.7350000000000003,"5min":7.7350000000000003,"15min":7.7350000000000003},"last":0.60999999999999999},{"interface":"front","average":{"1min":1.0700000000000001,"5min":0.67200000000000004,"15min":0.58399999999999996},"min":{"1min":0.44500000000000001,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":7.7190000000000003,"5min":7.7190000000000003,"15min":7.7190000000000003},"last":0.58299999999999996}]},{"osd":4,"last update":"Fri Jul 14 19:15:53 2023","interfaces":[{"interface":"back","average":{"1min":0.57899999999999996,"5min":0.53400000000000003,"15min":0.51500000000000001},"min":{"1min":0.45200000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.78700000000000003,"5min":0.78700000000000003,"15min":0.78700000000000003},"last":0.63},{"interface":"front","average":{"1min":0.60599999999999998,"5min":0.54700000000000004,"15min":0.52700000000000002},"min":{"1min":0.436,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.81299999999999994,"5min":0.81299999999999994,"15min":0.81299999999999994},"last":0.73499999999999999}]},{"osd":5,"last update":"Fri Jul 14 19:16:14 2023","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.42499999999999999,"15min":0.40799999999999997},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":0.746,"5min":0.746,"15min":0.746},"last":0.54300000000000004},{"interface":"front","average":{"1min":0.51300000000000001,"5min":0.45100000000000001,"15min":0.42899999999999999},"min":{"1min":0.32100000000000001,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.83999999999999997,"5min":0.83999999999999997,"15min":0.83999999999999997},"last":0.55800000000000005}]},{"osd":6,"last update":"Fri Jul 14 19:16:37 2023","interfaces":[{"interface":"back","average":{"1min":0.66900000000000004,"5min":0.63800000000000001,"15min":0.62},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":1.6930000000000001,"5min":1.6930000000000001,"15min":1.6930000000000001},"last":0.46400000000000002},{"interface":"front","average":{"1min":0.74099999999999999,"5min":0.64700000000000002,"15min":0.61699999999999999},"min":{"1min":0.48199999999999998,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":1.6659999999999999,"5min":1.6659999999999999,"15min":1.6659999999999999},"last":0.749}]},{"osd":7,"last update":"Fri Jul 14 19:15:53 2023","interfaces":[{"interface":"back","average":{"1min":0.64700000000000002,"5min":0.65100000000000002,"15min":0.65200000000000002},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.96299999999999997,"5min":0.96299999999999997,"15min":0.96299999999999997},"last":0.56999999999999995},{"interface":"front","average":{"1min":0.67100000000000004,"5min":0.66700000000000004,"15min":0.66700000000000004},"min":{"1min":0.48799999999999999,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.94099999999999995,"5min":0.94099999999999995,"15min":0.94099999999999995},"last":0.628}]},{"osd":8,"last update":"Fri Jul 14 19:16:14 2023","interfaces":[{"interface":"back","average":{"1min":0.63400000000000001,"5min":0.59299999999999997,"15min":0.58599999999999997},"min":{"1min":0.41599999999999998,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.98799999999999999,"5min":0.98799999999999999,"15min":0.98799999999999999},"last":0.67100000000000004},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.60499999999999998,"15min":0.59699999999999998},"min":{"1min":0.436,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":1.0069999999999999,"5min":1.0069999999999999,"15min":1.0069999999999999},"last":0.64300000000000002}]},{"osd":9,"last update":"Fri Jul 14 19:16:42 2023","interfaces":[{"interface":"back","average":{"1min":0.78100000000000003,"5min":0.74199999999999999,"15min":0.73599999999999999},"min":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"max":{"1min":1.6459999999999999,"5min":1.6459999999999999,"15min":1.6459999999999999},"last":0.59799999999999998},{"interface":"front","average":{"1min":1.093,"5min":0.80100000000000005,"15min":0.752},"min":{"1min":0.51600000000000001,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":5.9029999999999996,"5min":5.9029999999999996,"15min":5.9029999999999996},"last":0.82599999999999996}]},{"osd":10,"last update":"Fri Jul 14 19:15:56 2023","interfaces":[{"interface":"back","average":{"1min":0.63900000000000001,"5min":0.63900000000000001,"15min":0.63900000000000001},"min":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":0.88500000000000001,"5min":0.88500000000000001,"15min":0.88500000000000001},"last":0.72399999999999998},{"interface":"front","average":{"1min":0.71799999999999997,"5min":0.71799999999999997,"15min":0.71799999999999997},"min":{"1min":0.56200000000000006,"5min":0.56200000000000006,"15min":0.56200000000000006},"max":{"1min":1.073,"5min":1.073,"15min":1.073},"last":0.80200000000000005}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65500000000000003}]}]},{"osd":3,"up_from":25,"seq":107374182455,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27896,"kb_used_data":880,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717256,"statfs":{"total":95995035648,"available":95966470144,"internally_reserved":0,"allocated":901120,"data_stored":416265,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jul 14 19:16:29 2023","interfaces":[{"interface":"back","average":{"1min":0.40699999999999997,"5min":0.374,"15min":0.35499999999999998},"min":{"1min":0.28299999999999997,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":0.98399999999999999,"5min":0.98399999999999999,"15min":0.98399999999999999},"last":0.34699999999999998},{"interface":"front","average":{"1min":0.41399999999999998,"5min":0.40000000000000002,"15min":0.38100000000000001},"min":{"1min":0.312,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.84599999999999997,"5min":0.85899999999999999,"15min":0.85899999999999999},"last":0.31900000000000001}]},{"osd":1,"last update":"Fri Jul 14 19:16:29 2023","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.57199999999999995,"15min":0.53500000000000003},"min":{"1min":0.44800000000000001,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.79000000000000004,"5min":0.94699999999999995,"15min":0.94699999999999995},"last":0.53900000000000003},{"interface":"front","average":{"1min":0.624,"5min":0.58599999999999997,"15min":0.55700000000000005},"min":{"1min":0.41599999999999998,"5min":0.372,"15min":0.372},"max":{"1min":1.0800000000000001,"5min":1.0800000000000001,"15min":1.0800000000000001},"last":0.66600000000000004}]},{"osd":2,"last update":"Fri Jul 14 19:16:29 2023","interfaces":[{"interface":"back","average":{"1min":0.60299999999999998,"5min":0.57999999999999996,"15min":0.56100000000000005},"min":{"1min":0.45800000000000002,"5min":0.372,"15min":0.372},"max":{"1min":0.84899999999999998,"5min":0.871,"15min":0.871},"last":0.56299999999999994},{"interface":"front","average":{"1min":0.57399999999999995,"5min":0.57899999999999996,"15min":0.55700000000000005},"min":{"1min":0.44400000000000001,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.80000000000000004,"5min":0.93700000000000006,"15min":0.93700000000000006},"last":0.49099999999999999}]},{"osd":4,"last update":"Fri Jul 14 19:15:56 2023","interfaces":[{"interface":"back","average":{"1min":0.64500000000000002,"5min":0.58399999999999996,"15min":0.56200000000000006},"min":{"1min":0.47499999999999998,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.98599999999999999,"5min":0.98599999999999999,"15min":0.98599999999999999},"last":0.61899999999999999},{"interface":"front","average":{"1min":0.66800000000000004,"5min":0.60199999999999998,"15min":0.58999999999999997},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":0.86799999999999999,"5min":0.86799999999999999,"15min":0.86799999999999999},"last":0.58699999999999997}]},{"osd":5,"last update":"Fri Jul 14 19:16:15 2023","interfaces":[{"interface":"back","average":{"1min":0.60299999999999998,"5min":0.59499999999999997,"15min":0.58099999999999996},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":0.91300000000000003,"5min":0.91300000000000003,"15min":0.91300000000000003},"last":0.55300000000000005},{"interface":"front","average":{"1min":0.621,"5min":0.61599999999999999,"15min":0.60999999999999999},"min":{"1min":0.44700000000000001,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.81999999999999995,"5min":0.91700000000000004,"15min":0.91700000000000004},"last":0.67700000000000005}]},{"osd":6,"last update":"Fri Jul 14 19:16:37 2023","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.51000000000000001,"15min":0.50600000000000001},"min":{"1min":0.38100000000000001,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.82399999999999995,"5min":0.86399999999999999,"15min":0.86399999999999999},"last":0.39500000000000002},{"interface":"front","average":{"1min":0.55300000000000005,"5min":0.56899999999999995,"15min":0.55100000000000005},"min":{"1min":0.41099999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.877,"5min":0.99199999999999999,"15min":0.99199999999999999},"last":0.442}]},{"osd":7,"last update":"Fri Jul 14 19:15:56 2023","interfaces":[{"interface":"back","average":{"1min":0.73599999999999999,"5min":0.71699999999999997,"15min":0.71399999999999997},"min":{"1min":0.47599999999999998,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":1.05,"5min":1.1080000000000001,"15min":1.1080000000000001},"last":0.68899999999999995},{"interface":"front","average":{"1min":0.70299999999999996,"5min":0.67600000000000005,"15min":0.67200000000000004},"min":{"1min":0.46500000000000002,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":0.89400000000000002,"5min":0.95599999999999996,"15min":0.95599999999999996},"last":0.59999999999999998}]},{"osd":8,"last update":"Fri Jul 14 19:15:15 2023","interfaces":[{"interface":"back","average":{"1min":0.747,"5min":0.747,"15min":0.747},"min":{"1min":0.55800000000000005,"5min":0.55800000000000005,"15min":0.55800000000000005},"max":{"1min":1.0640000000000001,"5min":1.0640000000000001,"15min":1.0640000000000001},"last":0.66700000000000004},{"interface":"front","average":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"min":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":0.98499999999999999,"5min":0.98499999999999999,"15min":0.98499999999999999},"last":0.68500000000000005}]},{"osd":9,"last update":"Fri Jul 14 19:16:37 2023","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.67500000000000004,"15min":0.68999999999999995},"min":{"1min":0.41799999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.78500000000000003,"5min":1.048,"15min":1.048},"last":0.41899999999999998},{"interface":"front","average":{"1min":0.65400000000000003,"5min":0.749,"15min":0.76500000000000001},"min":{"1min":0.46600000000000003,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":0.95399999999999996,"5min":1.0289999999999999,"15min":1.0289999999999999},"last":0.46600000000000003}]},{"osd":10,"last update":"Fri Jul 14 19:15:56 2023","interfaces":[{"interface":"back","average":{"1min":0.81299999999999994,"5min":0.81299999999999994,"15min":0.81299999999999994},"min":{"1min":0.58199999999999996,"5min":0.58199999999999996,"15min":0.58199999999999996},"max":{"1min":1.0469999999999999,"5min":1.0469999999999999,"15min":1.0469999999999999},"last":0.73099999999999998},{"interface":"front","average":{"1min":0.77200000000000002,"5min":0.77200000000000002,"15min":0.77200000000000002},"min":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":1.0069999999999999,"5min":1.0069999999999999,"15min":1.0069999999999999},"last":0.69999999999999996}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64200000000000002}]}]},{"osd":4,"up_from":32,"seq":137438953523,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27884,"kb_used_data":868,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717268,"statfs":{"total":95995035648,"available":95966482432,"internally_reserved":0,"allocated":888832,"data_stored":416572,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jul 14 19:14:53 2023","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.503,"15min":0.495},"min":{"1min":0.27200000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.81999999999999995,"5min":0.81999999999999995,"15min":0.81999999999999995},"last":0.56299999999999994},{"interface":"front","average":{"1min":0.58199999999999996,"5min":0.53000000000000003,"15min":0.52200000000000002},"min":{"1min":0.378,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.84599999999999997,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.58299999999999996}]},{"osd":1,"last update":"Fri Jul 14 19:15:56 2023","interfaces":[{"interface":"back","average":{"1min":0.46600000000000003,"5min":0.39000000000000001,"15min":0.36399999999999999},"min":{"1min":0.30499999999999999,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.78400000000000003,"5min":1.0049999999999999,"15min":1.0049999999999999},"last":0.38900000000000001},{"interface":"front","average":{"1min":0.46600000000000003,"5min":0.39700000000000002,"15min":0.36899999999999999},"min":{"1min":0.26900000000000002,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":0.71699999999999997,"5min":0.91500000000000004,"15min":0.91500000000000004},"last":0.36599999999999999}]},{"osd":2,"last update":"Fri Jul 14 19:15:56 2023","interfaces":[{"interface":"back","average":{"1min":0.55100000000000005,"5min":0.54600000000000004,"15min":0.53400000000000003},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.69099999999999995,"5min":0.98499999999999999,"15min":0.98499999999999999},"last":0.60799999999999998},{"interface":"front","average":{"1min":0.66000000000000003,"5min":0.59899999999999998,"15min":0.56899999999999995},"min":{"1min":0.5,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.88700000000000001,"5min":0.92900000000000005,"15min":0.92900000000000005},"last":0.53200000000000003}]},{"osd":3,"last update":"Fri Jul 14 19:15:56 2023","interfaces":[{"interface":"back","average":{"1min":0.63700000000000001,"5min":0.55000000000000004,"15min":0.52500000000000002},"min":{"1min":0.41299999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.89300000000000002,"5min":0.89300000000000002,"15min":0.89300000000000002},"last":0.63300000000000001},{"interface":"front","average":{"1min":0.56100000000000005,"5min":0.55000000000000004,"15min":0.53800000000000003},"min":{"1min":0.36899999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.73499999999999999,"5min":0.94199999999999995,"15min":0.94199999999999995},"last":0.46999999999999997}]},{"osd":5,"last update":"Fri Jul 14 19:16:16 2023","interfaces":[{"interface":"back","average":{"1min":0.68300000000000005,"5min":0.60299999999999998,"15min":0.57499999999999996},"min":{"1min":0.50600000000000001,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":0.88200000000000001,"5min":0.88200000000000001,"15min":0.88200000000000001},"last":0.65200000000000002},{"interface":"front","average":{"1min":0.70999999999999996,"5min":0.60799999999999998,"15min":0.58099999999999996},"min":{"1min":0.52700000000000002,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.89700000000000002,"5min":0.89700000000000002,"15min":0.89700000000000002},"last":0.51100000000000001}]},{"osd":6,"last update":"Fri Jul 14 19:16:34 2023","interfaces":[{"interface":"back","average":{"1min":0.73299999999999998,"5min":0.64800000000000002,"15min":0.63},"min":{"1min":0.49299999999999999,"5min":0.38,"15min":0.38},"max":{"1min":0.95799999999999996,"5min":0.95799999999999996,"15min":0.95799999999999996},"last":0.56699999999999995},{"interface":"front","average":{"1min":0.746,"5min":0.63800000000000001,"15min":0.60799999999999998},"min":{"1min":0.47999999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.91500000000000004,"5min":1.0049999999999999,"15min":1.0049999999999999},"last":0.58099999999999996}]},{"osd":7,"last update":"Fri Jul 14 19:15:56 2023","interfaces":[{"interface":"back","average":{"1min":0.68100000000000005,"5min":0.60499999999999998,"15min":0.59299999999999997},"min":{"1min":0.44500000000000001,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.97599999999999998,"5min":1.0529999999999999,"15min":1.0529999999999999},"last":0.48799999999999999},{"interface":"front","average":{"1min":0.70899999999999996,"5min":0.59599999999999997,"15min":0.57699999999999996},"min":{"1min":0.47599999999999998,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.93100000000000005,"5min":0.93100000000000005,"15min":0.93100000000000005},"last":0.67500000000000004}]},{"osd":8,"last update":"Fri Jul 14 19:16:16 2023","interfaces":[{"interface":"back","average":{"1min":0.64400000000000002,"5min":0.63600000000000001,"15min":0.63500000000000001},"min":{"1min":0.51000000000000001,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":0.83599999999999997,"5min":1.0169999999999999,"15min":1.0169999999999999},"last":0.55000000000000004},{"interface":"front","average":{"1min":0.79300000000000004,"5min":0.77000000000000002,"15min":0.76600000000000001},"min":{"1min":0.56299999999999994,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":0.98499999999999999,"5min":0.98499999999999999,"15min":0.98499999999999999},"last":0.71399999999999997}]},{"osd":9,"last update":"Fri Jul 14 19:16:38 2023","interfaces":[{"interface":"back","average":{"1min":0.83799999999999997,"5min":0.78600000000000003,"15min":0.77800000000000002},"min":{"1min":0.64200000000000002,"5min":0.57399999999999995,"15min":0.57399999999999995},"max":{"1min":1.0009999999999999,"5min":1.0009999999999999,"15min":1.0009999999999999},"last":0.69999999999999996},{"interface":"front","average":{"1min":0.871,"5min":0.78800000000000003,"15min":0.77400000000000002},"min":{"1min":0.63600000000000001,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":1.024,"5min":1.05,"15min":1.05},"last":0.69099999999999995}]},{"osd":10,"last update":"Fri Jul 14 19:15:56 2023","interfaces":[{"interface":"back","average":{"1min":0.82199999999999995,"5min":0.82199999999999995,"15min":0.82199999999999995},"min":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.56000000000000005},"max":{"1min":1.0669999999999999,"5min":1.0669999999999999,"15min":1.0669999999999999},"last":0.73599999999999999},{"interface":"front","average":{"1min":0.80900000000000005,"5min":0.80900000000000005,"15min":0.80900000000000005},"min":{"1min":0.54900000000000004,"5min":0.54900000000000004,"15min":0.54900000000000004},"max":{"1min":1.028,"5min":1.028,"15min":1.028},"last":0.72299999999999998}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76200000000000001}]}]},{"osd":5,"up_from":37,"seq":158913789999,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27960,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717192,"statfs":{"total":95995035648,"available":95966404608,"internally_reserved":0,"allocated":966656,"data_stored":1009183,"data_compressed":35714,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jul 14 19:16:13 2023","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.52900000000000003,"15min":0.51500000000000001},"min":{"1min":0.28399999999999997,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.63700000000000001,"5min":1.028,"15min":1.028},"last":0.46200000000000002},{"interface":"front","average":{"1min":0.48399999999999999,"5min":0.54900000000000004,"15min":0.54600000000000004},"min":{"1min":0.33700000000000002,"5min":0.307,"15min":0.307},"max":{"1min":0.68600000000000005,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.36799999999999999}]},{"osd":1,"last update":"Fri Jul 14 19:16:13 2023","interfaces":[{"interface":"back","average":{"1min":0.57699999999999996,"5min":0.52900000000000003,"15min":0.503},"min":{"1min":0.38800000000000001,"5min":0.307,"15min":0.307},"max":{"1min":0.747,"5min":1.2410000000000001,"15min":1.2410000000000001},"last":0.51800000000000002},{"interface":"front","average":{"1min":0.56299999999999994,"5min":0.503,"15min":0.47899999999999998},"min":{"1min":0.36299999999999999,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.753,"5min":0.85799999999999998,"15min":0.85799999999999998},"last":0.63100000000000001}]},{"osd":2,"last update":"Fri Jul 14 19:16:13 2023","interfaces":[{"interface":"back","average":{"1min":0.41699999999999998,"5min":0.41399999999999998,"15min":0.40999999999999998},"min":{"1min":0.29999999999999999,"5min":0.246,"15min":0.246},"max":{"1min":0.66600000000000004,"5min":0.68400000000000005,"15min":0.68400000000000005},"last":0.38800000000000001},{"interface":"front","average":{"1min":0.39300000000000002,"5min":0.40799999999999997,"15min":0.40200000000000002},"min":{"1min":0.28399999999999997,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.52700000000000002,"5min":0.73999999999999999,"15min":0.73999999999999999},"last":0.40799999999999997}]},{"osd":3,"last update":"Fri Jul 14 19:16:13 2023","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":0.57999999999999996,"15min":0.57699999999999996},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.72899999999999998,"5min":0.98199999999999998,"15min":0.98199999999999998},"last":1.2030000000000001},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.59499999999999997,"15min":0.59099999999999997},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.71999999999999997,"5min":0.95899999999999996,"15min":0.95899999999999996},"last":1.284}]},{"osd":4,"last update":"Fri Jul 14 19:16:13 2023","interfaces":[{"interface":"back","average":{"1min":0.55100000000000005,"5min":0.54800000000000004,"15min":0.53600000000000003},"min":{"1min":0.42899999999999999,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.71199999999999997,"5min":0.79400000000000004,"15min":0.79400000000000004},"last":0.66200000000000003},{"interface":"front","average":{"1min":0.61299999999999999,"5min":0.58799999999999997,"15min":0.57399999999999995},"min":{"1min":0.46100000000000002,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.77400000000000002,"5min":0.875,"15min":0.875},"last":0.498}]},{"osd":6,"last update":"Fri Jul 14 19:16:37 2023","interfaces":[{"interface":"back","average":{"1min":0.65400000000000003,"5min":0.628,"15min":0.61599999999999999},"min":{"1min":0.5,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":1.492,"5min":1.492,"15min":1.492},"last":1.492},{"interface":"front","average":{"1min":0.66100000000000003,"5min":0.64800000000000002,"15min":0.63200000000000001},"min":{"1min":0.41999999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":1.4199999999999999,"5min":1.4199999999999999,"15min":1.4199999999999999},"last":1.4199999999999999}]},{"osd":7,"last update":"Fri Jul 14 19:15:53 2023","interfaces":[{"interface":"back","average":{"1min":0.64500000000000002,"5min":0.66000000000000003,"15min":0.66200000000000003},"min":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":0.84599999999999997,"5min":0.91400000000000003,"15min":0.91400000000000003},"last":0.54400000000000004},{"interface":"front","average":{"1min":0.66300000000000003,"5min":0.68000000000000005,"15min":0.68300000000000005},"min":{"1min":0.45700000000000002,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":0.83499999999999996,"5min":1.1240000000000001,"15min":1.1240000000000001},"last":0.57899999999999996}]},{"osd":8,"last update":"Fri Jul 14 19:16:14 2023","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.57099999999999995,"15min":0.57699999999999996},"min":{"1min":0.36399999999999999,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.77600000000000002,"5min":0.85199999999999998,"15min":0.85199999999999998},"last":0.59899999999999998},{"interface":"front","average":{"1min":0.66300000000000003,"5min":0.66300000000000003,"15min":0.66300000000000003},"min":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":0.81999999999999995,"5min":0.94199999999999995,"15min":0.94199999999999995},"last":0.61399999999999999}]},{"osd":9,"last update":"Fri Jul 14 19:16:37 2023","interfaces":[{"interface":"back","average":{"1min":0.67400000000000004,"5min":0.72299999999999998,"15min":0.73099999999999998},"min":{"1min":0.54800000000000004,"5min":0.53100000000000003,"15min":0.53100000000000003},"max":{"1min":0.79900000000000004,"5min":1.014,"15min":1.014},"last":0.64600000000000002},{"interface":"front","average":{"1min":0.79200000000000004,"5min":0.76600000000000001,"15min":0.76200000000000001},"min":{"1min":0.51200000000000001,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":1.9510000000000001,"5min":1.9510000000000001,"15min":1.9510000000000001},"last":1.9510000000000001}]},{"osd":10,"last update":"Fri Jul 14 19:15:59 2023","interfaces":[{"interface":"back","average":{"1min":0.70599999999999996,"5min":0.70599999999999996,"15min":0.70599999999999996},"min":{"1min":0.56200000000000006,"5min":0.56200000000000006,"15min":0.56200000000000006},"max":{"1min":0.89300000000000002,"5min":0.89300000000000002,"15min":0.89300000000000002},"last":0.67100000000000004},{"interface":"front","average":{"1min":0.70999999999999996,"5min":0.70999999999999996,"15min":0.70999999999999996},"min":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":0.89000000000000001,"5min":0.89000000000000001,"15min":0.89000000000000001},"last":0.68300000000000005}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34079,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34079,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34079,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35714,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35714,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35714,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35714,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35714,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2130,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":58650,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":58822,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":2772,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":674,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":2998,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":110592,"data_stored":58634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":246,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":123,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":123,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":123,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":123,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-07-14T19:16:47.018 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell --fsid adc52f04-2279-11ee-9b33-001a4aab830c -- ceph pg dump --format=json 2023-07-14T19:16:47.222 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:16:46 smithi191 ceph-mon[98311]: pgmap v278: 97 pgs: 97 active+clean; 638 KiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2023-07-14T19:16:47.242 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:16:46 smithi130 ceph-mon[97832]: pgmap v278: 97 pgs: 97 active+clean; 638 KiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2023-07-14T19:16:47.580 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/adc52f04-2279-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T19:16:48.004 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:16:47 smithi040 ceph-mon[96518]: from='client.14910 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-07-14T19:16:48.222 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:16:47 smithi191 ceph-mon[98311]: from='client.14910 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-07-14T19:16:48.242 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:16:47 smithi130 ceph-mon[97832]: from='client.14910 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-07-14T19:16:48.930 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:48.929+0000 7f71d1b22700 1 -- 172.21.15.40:0/2858718379 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f71cc0fee40 msgr2=0x7f71cc0ff210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:16:48.930 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:48.929+0000 7f71d1b22700 1 --2- 172.21.15.40:0/2858718379 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f71cc0fee40 0x7f71cc0ff210 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f71bc0099c0 tx=0x7f71bc009cd0 comp rx=0 tx=0).stop 2023-07-14T19:16:48.931 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:48.929+0000 7f71d1b22700 1 -- 172.21.15.40:0/2858718379 shutdown_connections 2023-07-14T19:16:48.931 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:48.929+0000 7f71d1b22700 1 --2- 172.21.15.40:0/2858718379 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f71cc103ee0 0x7f71cc10ba60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:48.931 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:48.929+0000 7f71d1b22700 1 --2- 172.21.15.40:0/2858718379 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f71cc0fee40 0x7f71cc0ff210 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:48.931 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:48.929+0000 7f71d1b22700 1 --2- 172.21.15.40:0/2858718379 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f71cc0ff7e0 0x7f71cc1037b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:48.931 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:48.929+0000 7f71d1b22700 1 -- 172.21.15.40:0/2858718379 >> 172.21.15.40:0/2858718379 conn(0x7f71cc075ca0 msgr2=0x7f71cc0760a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:16:48.931 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:48.929+0000 7f71d1b22700 1 -- 172.21.15.40:0/2858718379 shutdown_connections 2023-07-14T19:16:48.931 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:48.929+0000 7f71d1b22700 1 -- 172.21.15.40:0/2858718379 wait complete. 2023-07-14T19:16:48.932 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:48.929+0000 7f71d1b22700 1 Processor -- start 2023-07-14T19:16:48.932 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:48.930+0000 7f71d1b22700 1 -- start start 2023-07-14T19:16:48.933 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:48.930+0000 7f71d1b22700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f71cc0fee40 0x7f71cc195df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:16:48.934 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:48.930+0000 7f71d1b22700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f71cc0ff7e0 0x7f71cc196330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:16:48.934 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:48.930+0000 7f71d1b22700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f71cc103ee0 0x7f71cc19a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:16:48.934 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:48.930+0000 7f71d1b22700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71cc19adb0 con 0x7f71cc0ff7e0 2023-07-14T19:16:48.934 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:48.930+0000 7f71d1b22700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71cc19af20 con 0x7f71cc0fee40 2023-07-14T19:16:48.934 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:48.930+0000 7f71d1b22700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71cc19b0e0 con 0x7f71cc103ee0 2023-07-14T19:16:48.934 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:48.930+0000 7f71caffd700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f71cc0ff7e0 0x7f71cc196330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:16:48.934 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:48.930+0000 7f71caffd700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f71cc0ff7e0 0x7f71cc196330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:33504/0 (socket says 172.21.15.40:33504) 2023-07-14T19:16:48.935 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:48.930+0000 7f71caffd700 1 -- 172.21.15.40:0/1159471217 learned_addr learned my addr 172.21.15.40:0/1159471217 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-07-14T19:16:48.935 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:48.930+0000 7f71cb7fe700 1 --2- 172.21.15.40:0/1159471217 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f71cc0fee40 0x7f71cc195df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:16:48.935 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:48.930+0000 7f71cbfff700 1 --2- 172.21.15.40:0/1159471217 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f71cc103ee0 0x7f71cc19a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:16:48.935 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:48.930+0000 7f71caffd700 1 -- 172.21.15.40:0/1159471217 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f71cc103ee0 msgr2=0x7f71cc19a7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:16:48.935 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:48.930+0000 7f71caffd700 1 --2- 172.21.15.40:0/1159471217 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f71cc103ee0 0x7f71cc19a7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:48.935 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:48.930+0000 7f71caffd700 1 -- 172.21.15.40:0/1159471217 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f71cc0fee40 msgr2=0x7f71cc195df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:16:48.935 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:48.930+0000 7f71caffd700 1 --2- 172.21.15.40:0/1159471217 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f71cc0fee40 0x7f71cc195df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:48.935 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:48.930+0000 7f71caffd700 1 -- 172.21.15.40:0/1159471217 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71bc0095f0 con 0x7f71cc0ff7e0 2023-07-14T19:16:48.936 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:48.931+0000 7f71caffd700 1 --2- 172.21.15.40:0/1159471217 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f71cc0ff7e0 0x7f71cc196330 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f71c000c9e0 tx=0x7f71c000ccf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:16:48.936 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:48.931+0000 7f71c8ff9700 1 -- 172.21.15.40:0/1159471217 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71c0007aa0 con 0x7f71cc0ff7e0 2023-07-14T19:16:48.936 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:48.931+0000 7f71c8ff9700 1 -- 172.21.15.40:0/1159471217 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f71c0007c00 con 0x7f71cc0ff7e0 2023-07-14T19:16:48.936 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:48.931+0000 7f71d1b22700 1 -- 172.21.15.40:0/1159471217 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f71cc19b3c0 con 0x7f71cc0ff7e0 2023-07-14T19:16:48.936 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:48.931+0000 7f71c8ff9700 1 -- 172.21.15.40:0/1159471217 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71c0018660 con 0x7f71cc0ff7e0 2023-07-14T19:16:48.936 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:48.931+0000 7f71d1b22700 1 -- 172.21.15.40:0/1159471217 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f71cc1a2c20 con 0x7f71cc0ff7e0 2023-07-14T19:16:48.936 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:48.932+0000 7f71d1b22700 1 -- 172.21.15.40:0/1159471217 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f71cc04e340 con 0x7f71cc0ff7e0 2023-07-14T19:16:48.937 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:48.933+0000 7f71c8ff9700 1 -- 172.21.15.40:0/1159471217 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7f71c000f450 con 0x7f71cc0ff7e0 2023-07-14T19:16:48.937 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:48.933+0000 7f71c8ff9700 1 --2- 172.21.15.40:0/1159471217 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f71b40a11e0 0x7f71b40a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:16:48.938 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:48.934+0000 7f71cb7fe700 1 --2- 172.21.15.40:0/1159471217 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f71b40a11e0 0x7f71b40a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:16:48.938 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:48.934+0000 7f71c8ff9700 1 -- 172.21.15.40:0/1159471217 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f71c00ccae0 con 0x7f71cc0ff7e0 2023-07-14T19:16:48.938 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:48.934+0000 7f71cb7fe700 1 --2- 172.21.15.40:0/1159471217 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f71b40a11e0 0x7f71b40a3690 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f71bc009990 tx=0x7f71bc003720 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:16:48.938 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:48.937+0000 7f71c8ff9700 1 -- 172.21.15.40:0/1159471217 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f71c0096310 con 0x7f71cc0ff7e0 2023-07-14T19:16:48.971 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:16:48 smithi040 ceph-mon[96518]: pgmap v279: 97 pgs: 97 active+clean; 638 KiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2023-07-14T19:16:49.128 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:49.127+0000 7f71d1b22700 1 -- 172.21.15.40:0/1159471217 --> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f71cc068810 con 0x7f71b40a11e0 2023-07-14T19:16:49.140 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:49.139+0000 7f71c8ff9700 1 -- 172.21.15.40:0/1159471217 <== mgr.14152 v2:172.21.15.40:6800/1693027118 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+341215 (secure 0 0 0) 0x7f71cc068810 con 0x7f71b40a11e0 2023-07-14T19:16:49.143 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-07-14T19:16:49.145 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:49.143+0000 7f71d1b22700 1 -- 172.21.15.40:0/1159471217 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f71b40a11e0 msgr2=0x7f71b40a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:16:49.145 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:49.143+0000 7f71d1b22700 1 --2- 172.21.15.40:0/1159471217 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f71b40a11e0 0x7f71b40a3690 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f71bc009990 tx=0x7f71bc003720 comp rx=0 tx=0).stop 2023-07-14T19:16:49.146 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:49.143+0000 7f71d1b22700 1 -- 172.21.15.40:0/1159471217 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f71cc0ff7e0 msgr2=0x7f71cc196330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:16:49.146 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:49.143+0000 7f71d1b22700 1 --2- 172.21.15.40:0/1159471217 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f71cc0ff7e0 0x7f71cc196330 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f71c000c9e0 tx=0x7f71c000ccf0 comp rx=0 tx=0).stop 2023-07-14T19:16:49.146 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:49.143+0000 7f71d1b22700 1 -- 172.21.15.40:0/1159471217 shutdown_connections 2023-07-14T19:16:49.146 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:49.143+0000 7f71d1b22700 1 --2- 172.21.15.40:0/1159471217 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f71b40a11e0 0x7f71b40a3690 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:49.146 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:49.143+0000 7f71d1b22700 1 --2- 172.21.15.40:0/1159471217 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f71cc103ee0 0x7f71cc19a7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:49.146 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:49.143+0000 7f71d1b22700 1 --2- 172.21.15.40:0/1159471217 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f71cc0fee40 0x7f71cc195df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:49.146 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:49.143+0000 7f71d1b22700 1 --2- 172.21.15.40:0/1159471217 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f71cc0ff7e0 0x7f71cc196330 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:49.147 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:49.143+0000 7f71d1b22700 1 -- 172.21.15.40:0/1159471217 >> 172.21.15.40:0/1159471217 conn(0x7f71cc075ca0 msgr2=0x7f71cc068100 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:16:49.147 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:49.143+0000 7f71d1b22700 1 -- 172.21.15.40:0/1159471217 shutdown_connections 2023-07-14T19:16:49.147 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:49.143+0000 7f71d1b22700 1 -- 172.21.15.40:0/1159471217 wait complete. 2023-07-14T19:16:49.147 INFO:teuthology.orchestra.run.smithi040.stderr:dumped all 2023-07-14T19:16:49.222 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:16:48 smithi191 ceph-mon[98311]: pgmap v279: 97 pgs: 97 active+clean; 638 KiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2023-07-14T19:16:49.242 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:16:48 smithi130 ceph-mon[97832]: pgmap v279: 97 pgs: 97 active+clean; 638 KiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2023-07-14T19:16:49.640 INFO:teuthology.orchestra.run.smithi040.stdout:{"pg_ready":true,"pg_map":{"version":280,"stamp":"2023-07-14T19:16:48.821852+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":653790,"num_objects":102,"num_object_clones":0,"num_object_copies":306,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":102,"num_whiteouts":0,"num_read":579,"num_read_kb":406,"num_write":410,"num_write_kb":3064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":338,"ondisk_log_size":338,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":335060,"kb_used_data":10868,"kb_used_omap":20,"kb_used_meta":324075,"kb_avail":1124606764,"statfs":{"total":1151940427776,"available":1151597326336,"internally_reserved":0,"allocated":11128832,"data_stored":6951202,"data_compressed":107142,"data_compressed_allocated":135168,"data_compressed_original":1769472,"omap_allocated":21372,"internal_metadata":331852932},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":115,"num_read_kb":50,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003449"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-07-14T19:15:59.938963+0000","last_change":"2023-07-14T19:15:58.250082+0000","last_active":"2023-07-14T19:15:59.938963+0000","last_peered":"2023-07-14T19:15:59.938963+0000","last_clean":"2023-07-14T19:15:59.938963+0000","last_became_active":"2023-07-14T19:15:58.249783+0000","last_became_peered":"2023-07-14T19:15:58.249783+0000","last_unstale":"2023-07-14T19:15:59.938963+0000","last_undegraded":"2023-07-14T19:15:59.938963+0000","last_fullsized":"2023-07-14T19:15:59.938963+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_clean_scrub_stamp":"2023-07-14T19:15:57.219223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T03:38:35.618742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-07-14T19:15:59.573145+0000","last_change":"2023-07-14T19:15:58.240082+0000","last_active":"2023-07-14T19:15:59.573145+0000","last_peered":"2023-07-14T19:15:59.573145+0000","last_clean":"2023-07-14T19:15:59.573145+0000","last_became_active":"2023-07-14T19:15:58.239788+0000","last_became_peered":"2023-07-14T19:15:58.239788+0000","last_unstale":"2023-07-14T19:15:59.573145+0000","last_undegraded":"2023-07-14T19:15:59.573145+0000","last_fullsized":"2023-07-14T19:15:59.573145+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_clean_scrub_stamp":"2023-07-14T19:15:57.219223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T05:20:02.889590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-07-14T19:15:59.573092+0000","last_change":"2023-07-14T19:15:58.248010+0000","last_active":"2023-07-14T19:15:59.573092+0000","last_peered":"2023-07-14T19:15:59.573092+0000","last_clean":"2023-07-14T19:15:59.573092+0000","last_became_active":"2023-07-14T19:15:58.247755+0000","last_became_peered":"2023-07-14T19:15:58.247755+0000","last_unstale":"2023-07-14T19:15:59.573092+0000","last_undegraded":"2023-07-14T19:15:59.573092+0000","last_fullsized":"2023-07-14T19:15:59.573092+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_clean_scrub_stamp":"2023-07-14T19:15:57.219223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T19:47:43.289654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-07-14T19:15:59.229952+0000","last_change":"2023-07-14T19:15:58.246946+0000","last_active":"2023-07-14T19:15:59.229952+0000","last_peered":"2023-07-14T19:15:59.229952+0000","last_clean":"2023-07-14T19:15:59.229952+0000","last_became_active":"2023-07-14T19:15:58.246598+0000","last_became_peered":"2023-07-14T19:15:58.246598+0000","last_unstale":"2023-07-14T19:15:59.229952+0000","last_undegraded":"2023-07-14T19:15:59.229952+0000","last_fullsized":"2023-07-14T19:15:59.229952+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_clean_scrub_stamp":"2023-07-14T19:15:57.219223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T06:05:42.069121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-07-14T19:15:59.232238+0000","last_change":"2023-07-14T19:15:58.252632+0000","last_active":"2023-07-14T19:15:59.232238+0000","last_peered":"2023-07-14T19:15:59.232238+0000","last_clean":"2023-07-14T19:15:59.232238+0000","last_became_active":"2023-07-14T19:15:58.252319+0000","last_became_peered":"2023-07-14T19:15:58.252319+0000","last_unstale":"2023-07-14T19:15:59.232238+0000","last_undegraded":"2023-07-14T19:15:59.232238+0000","last_fullsized":"2023-07-14T19:15:59.232238+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_clean_scrub_stamp":"2023-07-14T19:15:57.219223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T06:07:06.243539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-07-14T19:15:59.722095+0000","last_change":"2023-07-14T19:15:58.248305+0000","last_active":"2023-07-14T19:15:59.722095+0000","last_peered":"2023-07-14T19:15:59.722095+0000","last_clean":"2023-07-14T19:15:59.722095+0000","last_became_active":"2023-07-14T19:15:58.247347+0000","last_became_peered":"2023-07-14T19:15:58.247347+0000","last_unstale":"2023-07-14T19:15:59.722095+0000","last_undegraded":"2023-07-14T19:15:59.722095+0000","last_fullsized":"2023-07-14T19:15:59.722095+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_clean_scrub_stamp":"2023-07-14T19:15:57.219223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T05:16:26.859500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-07-14T19:15:59.721610+0000","last_change":"2023-07-14T19:15:58.240871+0000","last_active":"2023-07-14T19:15:59.721610+0000","last_peered":"2023-07-14T19:15:59.721610+0000","last_clean":"2023-07-14T19:15:59.721610+0000","last_became_active":"2023-07-14T19:15:58.240574+0000","last_became_peered":"2023-07-14T19:15:58.240574+0000","last_unstale":"2023-07-14T19:15:59.721610+0000","last_undegraded":"2023-07-14T19:15:59.721610+0000","last_fullsized":"2023-07-14T19:15:59.721610+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_clean_scrub_stamp":"2023-07-14T19:15:57.219223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T00:07:11.508517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-07-14T19:15:59.721987+0000","last_change":"2023-07-14T19:15:58.241979+0000","last_active":"2023-07-14T19:15:59.721987+0000","last_peered":"2023-07-14T19:15:59.721987+0000","last_clean":"2023-07-14T19:15:59.721987+0000","last_became_active":"2023-07-14T19:15:58.241111+0000","last_became_peered":"2023-07-14T19:15:58.241111+0000","last_unstale":"2023-07-14T19:15:59.721987+0000","last_undegraded":"2023-07-14T19:15:59.721987+0000","last_fullsized":"2023-07-14T19:15:59.721987+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_clean_scrub_stamp":"2023-07-14T19:15:57.219223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T07:04:12.177825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-07-14T19:15:59.721751+0000","last_change":"2023-07-14T19:15:58.248866+0000","last_active":"2023-07-14T19:15:59.721751+0000","last_peered":"2023-07-14T19:15:59.721751+0000","last_clean":"2023-07-14T19:15:59.721751+0000","last_became_active":"2023-07-14T19:15:58.248456+0000","last_became_peered":"2023-07-14T19:15:58.248456+0000","last_unstale":"2023-07-14T19:15:59.721751+0000","last_undegraded":"2023-07-14T19:15:59.721751+0000","last_fullsized":"2023-07-14T19:15:59.721751+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_clean_scrub_stamp":"2023-07-14T19:15:57.219223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:28:00.287289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-07-14T19:15:59.722575+0000","last_change":"2023-07-14T19:15:58.248296+0000","last_active":"2023-07-14T19:15:59.722575+0000","last_peered":"2023-07-14T19:15:59.722575+0000","last_clean":"2023-07-14T19:15:59.722575+0000","last_became_active":"2023-07-14T19:15:58.247810+0000","last_became_peered":"2023-07-14T19:15:58.247810+0000","last_unstale":"2023-07-14T19:15:59.722575+0000","last_undegraded":"2023-07-14T19:15:59.722575+0000","last_fullsized":"2023-07-14T19:15:59.722575+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_clean_scrub_stamp":"2023-07-14T19:15:57.219223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T03:06:56.929583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-07-14T19:15:59.722644+0000","last_change":"2023-07-14T19:15:58.252057+0000","last_active":"2023-07-14T19:15:59.722644+0000","last_peered":"2023-07-14T19:15:59.722644+0000","last_clean":"2023-07-14T19:15:59.722644+0000","last_became_active":"2023-07-14T19:15:58.251425+0000","last_became_peered":"2023-07-14T19:15:58.251425+0000","last_unstale":"2023-07-14T19:15:59.722644+0000","last_undegraded":"2023-07-14T19:15:59.722644+0000","last_fullsized":"2023-07-14T19:15:59.722644+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_clean_scrub_stamp":"2023-07-14T19:15:57.219223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T00:23:30.017734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-07-14T19:15:59.229817+0000","last_change":"2023-07-14T19:15:58.241009+0000","last_active":"2023-07-14T19:15:59.229817+0000","last_peered":"2023-07-14T19:15:59.229817+0000","last_clean":"2023-07-14T19:15:59.229817+0000","last_became_active":"2023-07-14T19:15:58.240622+0000","last_became_peered":"2023-07-14T19:15:58.240622+0000","last_unstale":"2023-07-14T19:15:59.229817+0000","last_undegraded":"2023-07-14T19:15:59.229817+0000","last_fullsized":"2023-07-14T19:15:59.229817+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_clean_scrub_stamp":"2023-07-14T19:15:57.219223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T05:28:57.780030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-07-14T19:15:59.722108+0000","last_change":"2023-07-14T19:15:58.243377+0000","last_active":"2023-07-14T19:15:59.722108+0000","last_peered":"2023-07-14T19:15:59.722108+0000","last_clean":"2023-07-14T19:15:59.722108+0000","last_became_active":"2023-07-14T19:15:58.242225+0000","last_became_peered":"2023-07-14T19:15:58.242225+0000","last_unstale":"2023-07-14T19:15:59.722108+0000","last_undegraded":"2023-07-14T19:15:59.722108+0000","last_fullsized":"2023-07-14T19:15:59.722108+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_clean_scrub_stamp":"2023-07-14T19:15:57.219223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:21:59.269854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-07-14T19:15:59.573307+0000","last_change":"2023-07-14T19:15:58.244556+0000","last_active":"2023-07-14T19:15:59.573307+0000","last_peered":"2023-07-14T19:15:59.573307+0000","last_clean":"2023-07-14T19:15:59.573307+0000","last_became_active":"2023-07-14T19:15:58.244228+0000","last_became_peered":"2023-07-14T19:15:58.244228+0000","last_unstale":"2023-07-14T19:15:59.573307+0000","last_undegraded":"2023-07-14T19:15:59.573307+0000","last_fullsized":"2023-07-14T19:15:59.573307+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_clean_scrub_stamp":"2023-07-14T19:15:57.219223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:11:22.589361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-07-14T19:15:59.572875+0000","last_change":"2023-07-14T19:15:58.250176+0000","last_active":"2023-07-14T19:15:59.572875+0000","last_peered":"2023-07-14T19:15:59.572875+0000","last_clean":"2023-07-14T19:15:59.572875+0000","last_became_active":"2023-07-14T19:15:58.249708+0000","last_became_peered":"2023-07-14T19:15:58.249708+0000","last_unstale":"2023-07-14T19:15:59.572875+0000","last_undegraded":"2023-07-14T19:15:59.572875+0000","last_fullsized":"2023-07-14T19:15:59.572875+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_clean_scrub_stamp":"2023-07-14T19:15:57.219223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T22:33:15.826269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-07-14T19:15:59.721419+0000","last_change":"2023-07-14T19:15:58.246859+0000","last_active":"2023-07-14T19:15:59.721419+0000","last_peered":"2023-07-14T19:15:59.721419+0000","last_clean":"2023-07-14T19:15:59.721419+0000","last_became_active":"2023-07-14T19:15:58.246541+0000","last_became_peered":"2023-07-14T19:15:58.246541+0000","last_unstale":"2023-07-14T19:15:59.721419+0000","last_undegraded":"2023-07-14T19:15:59.721419+0000","last_fullsized":"2023-07-14T19:15:59.721419+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_clean_scrub_stamp":"2023-07-14T19:15:57.219223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T03:07:22.056544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-07-14T19:15:59.939090+0000","last_change":"2023-07-14T19:15:58.250112+0000","last_active":"2023-07-14T19:15:59.939090+0000","last_peered":"2023-07-14T19:15:59.939090+0000","last_clean":"2023-07-14T19:15:59.939090+0000","last_became_active":"2023-07-14T19:15:58.249824+0000","last_became_peered":"2023-07-14T19:15:58.249824+0000","last_unstale":"2023-07-14T19:15:59.939090+0000","last_undegraded":"2023-07-14T19:15:59.939090+0000","last_fullsized":"2023-07-14T19:15:59.939090+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_clean_scrub_stamp":"2023-07-14T19:15:57.219223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T22:34:26.596011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-07-14T19:15:59.723229+0000","last_change":"2023-07-14T19:15:58.243011+0000","last_active":"2023-07-14T19:15:59.723229+0000","last_peered":"2023-07-14T19:15:59.723229+0000","last_clean":"2023-07-14T19:15:59.723229+0000","last_became_active":"2023-07-14T19:15:58.242693+0000","last_became_peered":"2023-07-14T19:15:58.242693+0000","last_unstale":"2023-07-14T19:15:59.723229+0000","last_undegraded":"2023-07-14T19:15:59.723229+0000","last_fullsized":"2023-07-14T19:15:59.723229+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_clean_scrub_stamp":"2023-07-14T19:15:57.219223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:04:37.187798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-07-14T19:15:59.721093+0000","last_change":"2023-07-14T19:15:58.251298+0000","last_active":"2023-07-14T19:15:59.721093+0000","last_peered":"2023-07-14T19:15:59.721093+0000","last_clean":"2023-07-14T19:15:59.721093+0000","last_became_active":"2023-07-14T19:15:58.250984+0000","last_became_peered":"2023-07-14T19:15:58.250984+0000","last_unstale":"2023-07-14T19:15:59.721093+0000","last_undegraded":"2023-07-14T19:15:59.721093+0000","last_fullsized":"2023-07-14T19:15:59.721093+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_clean_scrub_stamp":"2023-07-14T19:15:57.219223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T20:29:32.350678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"81'13","reported_seq":67,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-07-14T19:16:42.686648+0000","last_change":"2023-07-14T19:15:57.235142+0000","last_active":"2023-07-14T19:16:42.686648+0000","last_peered":"2023-07-14T19:16:42.686648+0000","last_clean":"2023-07-14T19:16:42.686648+0000","last_became_active":"2023-07-14T19:15:57.234803+0000","last_became_peered":"2023-07-14T19:15:57.234803+0000","last_unstale":"2023-07-14T19:16:42.686648+0000","last_undegraded":"2023-07-14T19:16:42.686648+0000","last_fullsized":"2023-07-14T19:16:42.686648+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:15:56.210166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:15:56.210166+0000","last_clean_scrub_stamp":"2023-07-14T19:15:56.210166+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:27:04.191229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":40,"num_read_kb":40,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-07-14T19:15:59.722681+0000","last_change":"2023-07-14T19:15:58.247688+0000","last_active":"2023-07-14T19:15:59.722681+0000","last_peered":"2023-07-14T19:15:59.722681+0000","last_clean":"2023-07-14T19:15:59.722681+0000","last_became_active":"2023-07-14T19:15:58.247400+0000","last_became_peered":"2023-07-14T19:15:58.247400+0000","last_unstale":"2023-07-14T19:15:59.722681+0000","last_undegraded":"2023-07-14T19:15:59.722681+0000","last_fullsized":"2023-07-14T19:15:59.722681+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_clean_scrub_stamp":"2023-07-14T19:15:57.219223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T21:28:57.315894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"81'2","reported_seq":16,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-07-14T19:16:00.704237+0000","last_change":"2023-07-14T19:15:57.240026+0000","last_active":"2023-07-14T19:16:00.704237+0000","last_peered":"2023-07-14T19:16:00.704237+0000","last_clean":"2023-07-14T19:16:00.704237+0000","last_became_active":"2023-07-14T19:15:57.239687+0000","last_became_peered":"2023-07-14T19:15:57.239687+0000","last_unstale":"2023-07-14T19:16:00.704237+0000","last_undegraded":"2023-07-14T19:16:00.704237+0000","last_fullsized":"2023-07-14T19:16:00.704237+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:15:56.210166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:15:56.210166+0000","last_clean_scrub_stamp":"2023-07-14T19:15:56.210166+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:18:22.780240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-07-14T19:15:59.573195+0000","last_change":"2023-07-14T19:15:58.247039+0000","last_active":"2023-07-14T19:15:59.573195+0000","last_peered":"2023-07-14T19:15:59.573195+0000","last_clean":"2023-07-14T19:15:59.573195+0000","last_became_active":"2023-07-14T19:15:58.246685+0000","last_became_peered":"2023-07-14T19:15:58.246685+0000","last_unstale":"2023-07-14T19:15:59.573195+0000","last_undegraded":"2023-07-14T19:15:59.573195+0000","last_fullsized":"2023-07-14T19:15:59.573195+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_clean_scrub_stamp":"2023-07-14T19:15:57.219223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T03:20:32.272596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"81'12","reported_seq":27,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-07-14T19:16:05.632654+0000","last_change":"2023-07-14T19:15:57.244612+0000","last_active":"2023-07-14T19:16:05.632654+0000","last_peered":"2023-07-14T19:16:05.632654+0000","last_clean":"2023-07-14T19:16:05.632654+0000","last_became_active":"2023-07-14T19:15:57.243761+0000","last_became_peered":"2023-07-14T19:15:57.243761+0000","last_unstale":"2023-07-14T19:16:05.632654+0000","last_undegraded":"2023-07-14T19:16:05.632654+0000","last_fullsized":"2023-07-14T19:16:05.632654+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:15:56.210166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:15:56.210166+0000","last_clean_scrub_stamp":"2023-07-14T19:15:56.210166+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T21:44:07.276569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-07-14T19:15:59.721874+0000","last_change":"2023-07-14T19:15:58.239903+0000","last_active":"2023-07-14T19:15:59.721874+0000","last_peered":"2023-07-14T19:15:59.721874+0000","last_clean":"2023-07-14T19:15:59.721874+0000","last_became_active":"2023-07-14T19:15:58.239627+0000","last_became_peered":"2023-07-14T19:15:58.239627+0000","last_unstale":"2023-07-14T19:15:59.721874+0000","last_undegraded":"2023-07-14T19:15:59.721874+0000","last_fullsized":"2023-07-14T19:15:59.721874+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_clean_scrub_stamp":"2023-07-14T19:15:57.219223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:56:30.086686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"81'5","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-07-14T19:16:04.619350+0000","last_change":"2023-07-14T19:15:57.241773+0000","last_active":"2023-07-14T19:16:04.619350+0000","last_peered":"2023-07-14T19:16:04.619350+0000","last_clean":"2023-07-14T19:16:04.619350+0000","last_became_active":"2023-07-14T19:15:57.241451+0000","last_became_peered":"2023-07-14T19:15:57.241451+0000","last_unstale":"2023-07-14T19:16:04.619350+0000","last_undegraded":"2023-07-14T19:16:04.619350+0000","last_fullsized":"2023-07-14T19:16:04.619350+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:15:56.210166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:15:56.210166+0000","last_clean_scrub_stamp":"2023-07-14T19:15:56.210166+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T00:53:37.977462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-07-14T19:15:59.231297+0000","last_change":"2023-07-14T19:15:58.250564+0000","last_active":"2023-07-14T19:15:59.231297+0000","last_peered":"2023-07-14T19:15:59.231297+0000","last_clean":"2023-07-14T19:15:59.231297+0000","last_became_active":"2023-07-14T19:15:58.250277+0000","last_became_peered":"2023-07-14T19:15:58.250277+0000","last_unstale":"2023-07-14T19:15:59.231297+0000","last_undegraded":"2023-07-14T19:15:59.231297+0000","last_fullsized":"2023-07-14T19:15:59.231297+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_clean_scrub_stamp":"2023-07-14T19:15:57.219223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:37:56.001097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"81'5","reported_seq":64,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-07-14T19:16:46.693280+0000","last_change":"2023-07-14T19:15:57.245586+0000","last_active":"2023-07-14T19:16:46.693280+0000","last_peered":"2023-07-14T19:16:46.693280+0000","last_clean":"2023-07-14T19:16:46.693280+0000","last_became_active":"2023-07-14T19:15:57.244767+0000","last_became_peered":"2023-07-14T19:15:57.244767+0000","last_unstale":"2023-07-14T19:16:46.693280+0000","last_undegraded":"2023-07-14T19:16:46.693280+0000","last_fullsized":"2023-07-14T19:16:46.693280+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:15:56.210166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:15:56.210166+0000","last_clean_scrub_stamp":"2023-07-14T19:15:56.210166+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T06:43:35.975084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":574,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":45,"num_read_kb":2,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-07-14T19:15:59.232307+0000","last_change":"2023-07-14T19:15:58.240509+0000","last_active":"2023-07-14T19:15:59.232307+0000","last_peered":"2023-07-14T19:15:59.232307+0000","last_clean":"2023-07-14T19:15:59.232307+0000","last_became_active":"2023-07-14T19:15:58.240198+0000","last_became_peered":"2023-07-14T19:15:58.240198+0000","last_unstale":"2023-07-14T19:15:59.232307+0000","last_undegraded":"2023-07-14T19:15:59.232307+0000","last_fullsized":"2023-07-14T19:15:59.232307+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_clean_scrub_stamp":"2023-07-14T19:15:57.219223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T19:29:25.962372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"81'7","reported_seq":21,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-07-14T19:16:03.612450+0000","last_change":"2023-07-14T19:15:57.239582+0000","last_active":"2023-07-14T19:16:03.612450+0000","last_peered":"2023-07-14T19:16:03.612450+0000","last_clean":"2023-07-14T19:16:03.612450+0000","last_became_active":"2023-07-14T19:15:57.239244+0000","last_became_peered":"2023-07-14T19:15:57.239244+0000","last_unstale":"2023-07-14T19:16:03.612450+0000","last_undegraded":"2023-07-14T19:16:03.612450+0000","last_fullsized":"2023-07-14T19:16:03.612450+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:15:56.210166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:15:56.210166+0000","last_clean_scrub_stamp":"2023-07-14T19:15:56.210166+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T03:16:58.719650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-07-14T19:15:59.231791+0000","last_change":"2023-07-14T19:15:58.249443+0000","last_active":"2023-07-14T19:15:59.231791+0000","last_peered":"2023-07-14T19:15:59.231791+0000","last_clean":"2023-07-14T19:15:59.231791+0000","last_became_active":"2023-07-14T19:15:58.249131+0000","last_became_peered":"2023-07-14T19:15:58.249131+0000","last_unstale":"2023-07-14T19:15:59.231791+0000","last_undegraded":"2023-07-14T19:15:59.231791+0000","last_fullsized":"2023-07-14T19:15:59.231791+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_clean_scrub_stamp":"2023-07-14T19:15:57.219223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T03:45:01.049663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"81'3","reported_seq":17,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-07-14T19:16:02.597743+0000","last_change":"2023-07-14T19:15:57.243725+0000","last_active":"2023-07-14T19:16:02.597743+0000","last_peered":"2023-07-14T19:16:02.597743+0000","last_clean":"2023-07-14T19:16:02.597743+0000","last_became_active":"2023-07-14T19:15:57.243431+0000","last_became_peered":"2023-07-14T19:15:57.243431+0000","last_unstale":"2023-07-14T19:16:02.597743+0000","last_undegraded":"2023-07-14T19:16:02.597743+0000","last_fullsized":"2023-07-14T19:16:02.597743+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:15:56.210166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:15:56.210166+0000","last_clean_scrub_stamp":"2023-07-14T19:15:56.210166+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:41:49.174581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-07-14T19:15:59.721836+0000","last_change":"2023-07-14T19:15:58.239460+0000","last_active":"2023-07-14T19:15:59.721836+0000","last_peered":"2023-07-14T19:15:59.721836+0000","last_clean":"2023-07-14T19:15:59.721836+0000","last_became_active":"2023-07-14T19:15:58.239086+0000","last_became_peered":"2023-07-14T19:15:58.239086+0000","last_unstale":"2023-07-14T19:15:59.721836+0000","last_undegraded":"2023-07-14T19:15:59.721836+0000","last_fullsized":"2023-07-14T19:15:59.721836+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_clean_scrub_stamp":"2023-07-14T19:15:57.219223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T23:25:07.452583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"81'3","reported_seq":18,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-07-14T19:16:04.617213+0000","last_change":"2023-07-14T19:15:57.240173+0000","last_active":"2023-07-14T19:16:04.617213+0000","last_peered":"2023-07-14T19:16:04.617213+0000","last_clean":"2023-07-14T19:16:04.617213+0000","last_became_active":"2023-07-14T19:15:57.239822+0000","last_became_peered":"2023-07-14T19:15:57.239822+0000","last_unstale":"2023-07-14T19:16:04.617213+0000","last_undegraded":"2023-07-14T19:16:04.617213+0000","last_fullsized":"2023-07-14T19:16:04.617213+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:15:56.210166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:15:56.210166+0000","last_clean_scrub_stamp":"2023-07-14T19:15:56.210166+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T20:22:32.066346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-07-14T19:15:59.722607+0000","last_change":"2023-07-14T19:15:58.241533+0000","last_active":"2023-07-14T19:15:59.722607+0000","last_peered":"2023-07-14T19:15:59.722607+0000","last_clean":"2023-07-14T19:15:59.722607+0000","last_became_active":"2023-07-14T19:15:58.241216+0000","last_became_peered":"2023-07-14T19:15:58.241216+0000","last_unstale":"2023-07-14T19:15:59.722607+0000","last_undegraded":"2023-07-14T19:15:59.722607+0000","last_fullsized":"2023-07-14T19:15:59.722607+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_clean_scrub_stamp":"2023-07-14T19:15:57.219223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:40:27.802949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"81'4","reported_seq":98,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-07-14T19:16:44.690064+0000","last_change":"2023-07-14T19:15:57.236808+0000","last_active":"2023-07-14T19:16:44.690064+0000","last_peered":"2023-07-14T19:16:44.690064+0000","last_clean":"2023-07-14T19:16:44.690064+0000","last_became_active":"2023-07-14T19:15:57.236480+0000","last_became_peered":"2023-07-14T19:15:57.236480+0000","last_unstale":"2023-07-14T19:16:44.690064+0000","last_undegraded":"2023-07-14T19:16:44.690064+0000","last_fullsized":"2023-07-14T19:16:44.690064+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:15:56.210166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:15:56.210166+0000","last_clean_scrub_stamp":"2023-07-14T19:15:56.210166+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T20:50:18.427213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":80,"num_read_kb":80,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-07-14T19:15:59.721741+0000","last_change":"2023-07-14T19:15:58.243236+0000","last_active":"2023-07-14T19:15:59.721741+0000","last_peered":"2023-07-14T19:15:59.721741+0000","last_clean":"2023-07-14T19:15:59.721741+0000","last_became_active":"2023-07-14T19:15:58.242951+0000","last_became_peered":"2023-07-14T19:15:58.242951+0000","last_unstale":"2023-07-14T19:15:59.721741+0000","last_undegraded":"2023-07-14T19:15:59.721741+0000","last_fullsized":"2023-07-14T19:15:59.721741+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_clean_scrub_stamp":"2023-07-14T19:15:57.219223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T05:20:59.047342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"81'4","reported_seq":18,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-07-14T19:16:04.617616+0000","last_change":"2023-07-14T19:15:57.242251+0000","last_active":"2023-07-14T19:16:04.617616+0000","last_peered":"2023-07-14T19:16:04.617616+0000","last_clean":"2023-07-14T19:16:04.617616+0000","last_became_active":"2023-07-14T19:15:57.241907+0000","last_became_peered":"2023-07-14T19:15:57.241907+0000","last_unstale":"2023-07-14T19:16:04.617616+0000","last_undegraded":"2023-07-14T19:16:04.617616+0000","last_fullsized":"2023-07-14T19:16:04.617616+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:15:56.210166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:15:56.210166+0000","last_clean_scrub_stamp":"2023-07-14T19:15:56.210166+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:52:53.425429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-07-14T19:15:59.573106+0000","last_change":"2023-07-14T19:15:58.248412+0000","last_active":"2023-07-14T19:15:59.573106+0000","last_peered":"2023-07-14T19:15:59.573106+0000","last_clean":"2023-07-14T19:15:59.573106+0000","last_became_active":"2023-07-14T19:15:58.247619+0000","last_became_peered":"2023-07-14T19:15:58.247619+0000","last_unstale":"2023-07-14T19:15:59.573106+0000","last_undegraded":"2023-07-14T19:15:59.573106+0000","last_fullsized":"2023-07-14T19:15:59.573106+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_clean_scrub_stamp":"2023-07-14T19:15:57.219223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T00:15:52.855579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"81'5","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-07-14T19:16:03.611516+0000","last_change":"2023-07-14T19:15:57.240436+0000","last_active":"2023-07-14T19:16:03.611516+0000","last_peered":"2023-07-14T19:16:03.611516+0000","last_clean":"2023-07-14T19:16:03.611516+0000","last_became_active":"2023-07-14T19:15:57.240148+0000","last_became_peered":"2023-07-14T19:15:57.240148+0000","last_unstale":"2023-07-14T19:16:03.611516+0000","last_undegraded":"2023-07-14T19:16:03.611516+0000","last_fullsized":"2023-07-14T19:16:03.611516+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:15:56.210166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:15:56.210166+0000","last_clean_scrub_stamp":"2023-07-14T19:15:56.210166+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T03:15:40.160793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-07-14T19:15:59.722062+0000","last_change":"2023-07-14T19:15:58.246127+0000","last_active":"2023-07-14T19:15:59.722062+0000","last_peered":"2023-07-14T19:15:59.722062+0000","last_clean":"2023-07-14T19:15:59.722062+0000","last_became_active":"2023-07-14T19:15:58.245778+0000","last_became_peered":"2023-07-14T19:15:58.245778+0000","last_unstale":"2023-07-14T19:15:59.722062+0000","last_undegraded":"2023-07-14T19:15:59.722062+0000","last_fullsized":"2023-07-14T19:15:59.722062+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_clean_scrub_stamp":"2023-07-14T19:15:57.219223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T00:21:57.683936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"81'28","reported_seq":127,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-07-14T19:16:44.690254+0000","last_change":"2023-07-14T19:15:57.243660+0000","last_active":"2023-07-14T19:16:44.690254+0000","last_peered":"2023-07-14T19:16:44.690254+0000","last_clean":"2023-07-14T19:16:44.690254+0000","last_became_active":"2023-07-14T19:15:57.243338+0000","last_became_peered":"2023-07-14T19:15:57.243338+0000","last_unstale":"2023-07-14T19:16:44.690254+0000","last_undegraded":"2023-07-14T19:16:44.690254+0000","last_fullsized":"2023-07-14T19:16:44.690254+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:15:56.210166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:15:56.210166+0000","last_clean_scrub_stamp":"2023-07-14T19:15:56.210166+0000","objects_scrubbed":0,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T06:12:23.930218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56946,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":85,"num_read_kb":98,"num_write":30,"num_write_kb":64,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-07-14T19:15:59.232525+0000","last_change":"2023-07-14T19:15:58.244162+0000","last_active":"2023-07-14T19:15:59.232525+0000","last_peered":"2023-07-14T19:15:59.232525+0000","last_clean":"2023-07-14T19:15:59.232525+0000","last_became_active":"2023-07-14T19:15:58.243912+0000","last_became_peered":"2023-07-14T19:15:58.243912+0000","last_unstale":"2023-07-14T19:15:59.232525+0000","last_undegraded":"2023-07-14T19:15:59.232525+0000","last_fullsized":"2023-07-14T19:15:59.232525+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_clean_scrub_stamp":"2023-07-14T19:15:57.219223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T23:05:24.992671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"81'9","reported_seq":25,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-07-14T19:16:05.632720+0000","last_change":"2023-07-14T19:15:57.246975+0000","last_active":"2023-07-14T19:16:05.632720+0000","last_peered":"2023-07-14T19:16:05.632720+0000","last_clean":"2023-07-14T19:16:05.632720+0000","last_became_active":"2023-07-14T19:15:57.246481+0000","last_became_peered":"2023-07-14T19:15:57.246481+0000","last_unstale":"2023-07-14T19:16:05.632720+0000","last_undegraded":"2023-07-14T19:16:05.632720+0000","last_fullsized":"2023-07-14T19:16:05.632720+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:15:56.210166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:15:56.210166+0000","last_clean_scrub_stamp":"2023-07-14T19:15:56.210166+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T06:50:24.165830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-07-14T19:15:59.721632+0000","last_change":"2023-07-14T19:15:58.241870+0000","last_active":"2023-07-14T19:15:59.721632+0000","last_peered":"2023-07-14T19:15:59.721632+0000","last_clean":"2023-07-14T19:15:59.721632+0000","last_became_active":"2023-07-14T19:15:58.241503+0000","last_became_peered":"2023-07-14T19:15:58.241503+0000","last_unstale":"2023-07-14T19:15:59.721632+0000","last_undegraded":"2023-07-14T19:15:59.721632+0000","last_fullsized":"2023-07-14T19:15:59.721632+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_clean_scrub_stamp":"2023-07-14T19:15:57.219223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T23:05:14.779450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"81'5","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-07-14T19:16:04.617233+0000","last_change":"2023-07-14T19:15:57.236715+0000","last_active":"2023-07-14T19:16:04.617233+0000","last_peered":"2023-07-14T19:16:04.617233+0000","last_clean":"2023-07-14T19:16:04.617233+0000","last_became_active":"2023-07-14T19:15:57.236391+0000","last_became_peered":"2023-07-14T19:15:57.236391+0000","last_unstale":"2023-07-14T19:16:04.617233+0000","last_undegraded":"2023-07-14T19:16:04.617233+0000","last_fullsized":"2023-07-14T19:16:04.617233+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:15:56.210166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:15:56.210166+0000","last_clean_scrub_stamp":"2023-07-14T19:15:56.210166+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:06:23.938675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-07-14T19:15:59.721870+0000","last_change":"2023-07-14T19:15:58.245110+0000","last_active":"2023-07-14T19:15:59.721870+0000","last_peered":"2023-07-14T19:15:59.721870+0000","last_clean":"2023-07-14T19:15:59.721870+0000","last_became_active":"2023-07-14T19:15:58.244779+0000","last_became_peered":"2023-07-14T19:15:58.244779+0000","last_unstale":"2023-07-14T19:15:59.721870+0000","last_undegraded":"2023-07-14T19:15:59.721870+0000","last_fullsized":"2023-07-14T19:15:59.721870+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_clean_scrub_stamp":"2023-07-14T19:15:57.219223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T00:24:23.826897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"81'5","reported_seq":63,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-07-14T19:16:45.691712+0000","last_change":"2023-07-14T19:15:57.236795+0000","last_active":"2023-07-14T19:16:45.691712+0000","last_peered":"2023-07-14T19:16:45.691712+0000","last_clean":"2023-07-14T19:16:45.691712+0000","last_became_active":"2023-07-14T19:15:57.236442+0000","last_became_peered":"2023-07-14T19:15:57.236442+0000","last_unstale":"2023-07-14T19:16:45.691712+0000","last_undegraded":"2023-07-14T19:16:45.691712+0000","last_fullsized":"2023-07-14T19:16:45.691712+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:15:56.210166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:15:56.210166+0000","last_clean_scrub_stamp":"2023-07-14T19:15:56.210166+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:05:41.764871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":540,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":45,"num_read_kb":3,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-07-14T19:15:59.721448+0000","last_change":"2023-07-14T19:15:58.247790+0000","last_active":"2023-07-14T19:15:59.721448+0000","last_peered":"2023-07-14T19:15:59.721448+0000","last_clean":"2023-07-14T19:15:59.721448+0000","last_became_active":"2023-07-14T19:15:58.247461+0000","last_became_peered":"2023-07-14T19:15:58.247461+0000","last_unstale":"2023-07-14T19:15:59.721448+0000","last_undegraded":"2023-07-14T19:15:59.721448+0000","last_fullsized":"2023-07-14T19:15:59.721448+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_clean_scrub_stamp":"2023-07-14T19:15:57.219223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:29:24.306834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"81'17","reported_seq":32,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-07-14T19:16:04.618370+0000","last_change":"2023-07-14T19:15:57.244534+0000","last_active":"2023-07-14T19:16:04.618370+0000","last_peered":"2023-07-14T19:16:04.618370+0000","last_clean":"2023-07-14T19:16:04.618370+0000","last_became_active":"2023-07-14T19:15:57.243107+0000","last_became_peered":"2023-07-14T19:15:57.243107+0000","last_unstale":"2023-07-14T19:16:04.618370+0000","last_undegraded":"2023-07-14T19:16:04.618370+0000","last_fullsized":"2023-07-14T19:16:04.618370+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:15:56.210166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:15:56.210166+0000","last_clean_scrub_stamp":"2023-07-14T19:15:56.210166+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T21:48:40.199873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-07-14T19:15:59.231092+0000","last_change":"2023-07-14T19:15:58.250511+0000","last_active":"2023-07-14T19:15:59.231092+0000","last_peered":"2023-07-14T19:15:59.231092+0000","last_clean":"2023-07-14T19:15:59.231092+0000","last_became_active":"2023-07-14T19:15:58.250159+0000","last_became_peered":"2023-07-14T19:15:58.250159+0000","last_unstale":"2023-07-14T19:15:59.231092+0000","last_undegraded":"2023-07-14T19:15:59.231092+0000","last_fullsized":"2023-07-14T19:15:59.231092+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_clean_scrub_stamp":"2023-07-14T19:15:57.219223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T21:32:01.440745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"81'2","reported_seq":17,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-07-14T19:16:05.632535+0000","last_change":"2023-07-14T19:15:57.238719+0000","last_active":"2023-07-14T19:16:05.632535+0000","last_peered":"2023-07-14T19:16:05.632535+0000","last_clean":"2023-07-14T19:16:05.632535+0000","last_became_active":"2023-07-14T19:15:57.238404+0000","last_became_peered":"2023-07-14T19:15:57.238404+0000","last_unstale":"2023-07-14T19:16:05.632535+0000","last_undegraded":"2023-07-14T19:16:05.632535+0000","last_fullsized":"2023-07-14T19:16:05.632535+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:15:56.210166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:15:56.210166+0000","last_clean_scrub_stamp":"2023-07-14T19:15:56.210166+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T03:14:38.948894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-07-14T19:15:59.938888+0000","last_change":"2023-07-14T19:15:58.245052+0000","last_active":"2023-07-14T19:15:59.938888+0000","last_peered":"2023-07-14T19:15:59.938888+0000","last_clean":"2023-07-14T19:15:59.938888+0000","last_became_active":"2023-07-14T19:15:58.244733+0000","last_became_peered":"2023-07-14T19:15:58.244733+0000","last_unstale":"2023-07-14T19:15:59.938888+0000","last_undegraded":"2023-07-14T19:15:59.938888+0000","last_fullsized":"2023-07-14T19:15:59.938888+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_clean_scrub_stamp":"2023-07-14T19:15:57.219223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T03:43:49.689439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"81'1","reported_seq":15,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-07-14T19:15:59.722595+0000","last_change":"2023-07-14T19:15:57.245402+0000","last_active":"2023-07-14T19:15:59.722595+0000","last_peered":"2023-07-14T19:15:59.722595+0000","last_clean":"2023-07-14T19:15:59.722595+0000","last_became_active":"2023-07-14T19:15:57.245092+0000","last_became_peered":"2023-07-14T19:15:57.245092+0000","last_unstale":"2023-07-14T19:15:59.722595+0000","last_undegraded":"2023-07-14T19:15:59.722595+0000","last_fullsized":"2023-07-14T19:15:59.722595+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:15:56.210166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:15:56.210166+0000","last_clean_scrub_stamp":"2023-07-14T19:15:56.210166+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:17:28.260009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"81'1","reported_seq":13,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-07-14T19:16:00.307970+0000","last_change":"2023-07-14T19:15:58.247611+0000","last_active":"2023-07-14T19:16:00.307970+0000","last_peered":"2023-07-14T19:16:00.307970+0000","last_clean":"2023-07-14T19:16:00.307970+0000","last_became_active":"2023-07-14T19:15:58.247313+0000","last_became_peered":"2023-07-14T19:15:58.247313+0000","last_unstale":"2023-07-14T19:16:00.307970+0000","last_undegraded":"2023-07-14T19:16:00.307970+0000","last_fullsized":"2023-07-14T19:16:00.307970+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_clean_scrub_stamp":"2023-07-14T19:15:57.219223+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T06:40:19.317380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":123,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"81'5","reported_seq":61,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-07-14T19:16:44.689861+0000","last_change":"2023-07-14T19:15:57.242149+0000","last_active":"2023-07-14T19:16:44.689861+0000","last_peered":"2023-07-14T19:16:44.689861+0000","last_clean":"2023-07-14T19:16:44.689861+0000","last_became_active":"2023-07-14T19:15:57.241847+0000","last_became_peered":"2023-07-14T19:15:57.241847+0000","last_unstale":"2023-07-14T19:16:44.689861+0000","last_undegraded":"2023-07-14T19:16:44.689861+0000","last_fullsized":"2023-07-14T19:16:44.689861+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:15:56.210166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:15:56.210166+0000","last_clean_scrub_stamp":"2023-07-14T19:15:56.210166+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:14:47.640280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":540,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":42,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-07-14T19:15:59.722352+0000","last_change":"2023-07-14T19:15:58.244573+0000","last_active":"2023-07-14T19:15:59.722352+0000","last_peered":"2023-07-14T19:15:59.722352+0000","last_clean":"2023-07-14T19:15:59.722352+0000","last_became_active":"2023-07-14T19:15:58.244165+0000","last_became_peered":"2023-07-14T19:15:58.244165+0000","last_unstale":"2023-07-14T19:15:59.722352+0000","last_undegraded":"2023-07-14T19:15:59.722352+0000","last_fullsized":"2023-07-14T19:15:59.722352+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_clean_scrub_stamp":"2023-07-14T19:15:57.219223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T05:36:19.851400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-07-14T19:15:59.231138+0000","last_change":"2023-07-14T19:15:58.246282+0000","last_active":"2023-07-14T19:15:59.231138+0000","last_peered":"2023-07-14T19:15:59.231138+0000","last_clean":"2023-07-14T19:15:59.231138+0000","last_became_active":"2023-07-14T19:15:58.245984+0000","last_became_peered":"2023-07-14T19:15:58.245984+0000","last_unstale":"2023-07-14T19:15:59.231138+0000","last_undegraded":"2023-07-14T19:15:59.231138+0000","last_fullsized":"2023-07-14T19:15:59.231138+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_clean_scrub_stamp":"2023-07-14T19:15:57.219223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T00:42:41.109802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"81'5","reported_seq":60,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-07-14T19:16:43.688451+0000","last_change":"2023-07-14T19:15:57.244731+0000","last_active":"2023-07-14T19:16:43.688451+0000","last_peered":"2023-07-14T19:16:43.688451+0000","last_clean":"2023-07-14T19:16:43.688451+0000","last_became_active":"2023-07-14T19:15:57.244384+0000","last_became_peered":"2023-07-14T19:15:57.244384+0000","last_unstale":"2023-07-14T19:16:43.688451+0000","last_undegraded":"2023-07-14T19:16:43.688451+0000","last_fullsized":"2023-07-14T19:16:43.688451+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:15:56.210166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:15:56.210166+0000","last_clean_scrub_stamp":"2023-07-14T19:15:56.210166+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T22:20:50.336578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":41,"num_read_kb":41,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-07-14T19:15:59.722302+0000","last_change":"2023-07-14T19:15:58.241457+0000","last_active":"2023-07-14T19:15:59.722302+0000","last_peered":"2023-07-14T19:15:59.722302+0000","last_clean":"2023-07-14T19:15:59.722302+0000","last_became_active":"2023-07-14T19:15:58.241113+0000","last_became_peered":"2023-07-14T19:15:58.241113+0000","last_unstale":"2023-07-14T19:15:59.722302+0000","last_undegraded":"2023-07-14T19:15:59.722302+0000","last_fullsized":"2023-07-14T19:15:59.722302+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_clean_scrub_stamp":"2023-07-14T19:15:57.219223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T21:28:59.714151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"81'6","reported_seq":23,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-07-14T19:16:05.632583+0000","last_change":"2023-07-14T19:15:57.235057+0000","last_active":"2023-07-14T19:16:05.632583+0000","last_peered":"2023-07-14T19:16:05.632583+0000","last_clean":"2023-07-14T19:16:05.632583+0000","last_became_active":"2023-07-14T19:15:57.234751+0000","last_became_peered":"2023-07-14T19:15:57.234751+0000","last_unstale":"2023-07-14T19:16:05.632583+0000","last_undegraded":"2023-07-14T19:16:05.632583+0000","last_fullsized":"2023-07-14T19:16:05.632583+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:15:56.210166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:15:56.210166+0000","last_clean_scrub_stamp":"2023-07-14T19:15:56.210166+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:12:00.277001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":336,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-07-14T19:15:59.721531+0000","last_change":"2023-07-14T19:14:58.969281+0000","last_active":"2023-07-14T19:15:59.721531+0000","last_peered":"2023-07-14T19:15:59.721531+0000","last_clean":"2023-07-14T19:15:59.721531+0000","last_became_active":"2023-07-14T19:14:58.968767+0000","last_became_peered":"2023-07-14T19:14:58.968767+0000","last_unstale":"2023-07-14T19:15:59.721531+0000","last_undegraded":"2023-07-14T19:15:59.721531+0000","last_fullsized":"2023-07-14T19:15:59.721531+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:12:03.589019+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:12:03.589019+0000","last_clean_scrub_stamp":"2023-07-14T19:12:03.589019+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T05:39:04.087637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-07-14T19:15:59.722476+0000","last_change":"2023-07-14T19:15:58.250737+0000","last_active":"2023-07-14T19:15:59.722476+0000","last_peered":"2023-07-14T19:15:59.722476+0000","last_clean":"2023-07-14T19:15:59.722476+0000","last_became_active":"2023-07-14T19:15:58.250379+0000","last_became_peered":"2023-07-14T19:15:58.250379+0000","last_unstale":"2023-07-14T19:15:59.722476+0000","last_undegraded":"2023-07-14T19:15:59.722476+0000","last_fullsized":"2023-07-14T19:15:59.722476+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_clean_scrub_stamp":"2023-07-14T19:15:57.219223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T19:57:17.147937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"81'4","reported_seq":18,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-07-14T19:16:02.592793+0000","last_change":"2023-07-14T19:15:57.236799+0000","last_active":"2023-07-14T19:16:02.592793+0000","last_peered":"2023-07-14T19:16:02.592793+0000","last_clean":"2023-07-14T19:16:02.592793+0000","last_became_active":"2023-07-14T19:15:57.236503+0000","last_became_peered":"2023-07-14T19:15:57.236503+0000","last_unstale":"2023-07-14T19:16:02.592793+0000","last_undegraded":"2023-07-14T19:16:02.592793+0000","last_fullsized":"2023-07-14T19:16:02.592793+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:15:56.210166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:15:56.210166+0000","last_clean_scrub_stamp":"2023-07-14T19:15:56.210166+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T00:26:24.967329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"81'4","reported_seq":19,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-07-14T19:16:04.618843+0000","last_change":"2023-07-14T19:15:57.245788+0000","last_active":"2023-07-14T19:16:04.618843+0000","last_peered":"2023-07-14T19:16:04.618843+0000","last_clean":"2023-07-14T19:16:04.618843+0000","last_became_active":"2023-07-14T19:15:57.244749+0000","last_became_peered":"2023-07-14T19:15:57.244749+0000","last_unstale":"2023-07-14T19:16:04.618843+0000","last_undegraded":"2023-07-14T19:16:04.618843+0000","last_fullsized":"2023-07-14T19:16:04.618843+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:15:56.210166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:15:56.210166+0000","last_clean_scrub_stamp":"2023-07-14T19:15:56.210166+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T00:56:58.078952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-07-14T19:15:59.722962+0000","last_change":"2023-07-14T19:15:58.250973+0000","last_active":"2023-07-14T19:15:59.722962+0000","last_peered":"2023-07-14T19:15:59.722962+0000","last_clean":"2023-07-14T19:15:59.722962+0000","last_became_active":"2023-07-14T19:15:58.250634+0000","last_became_peered":"2023-07-14T19:15:58.250634+0000","last_unstale":"2023-07-14T19:15:59.722962+0000","last_undegraded":"2023-07-14T19:15:59.722962+0000","last_fullsized":"2023-07-14T19:15:59.722962+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_clean_scrub_stamp":"2023-07-14T19:15:57.219223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T06:14:32.307565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"81'9","reported_seq":23,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-07-14T19:16:08.026032+0000","last_change":"2023-07-14T19:15:57.234996+0000","last_active":"2023-07-14T19:16:08.026032+0000","last_peered":"2023-07-14T19:16:08.026032+0000","last_clean":"2023-07-14T19:16:08.026032+0000","last_became_active":"2023-07-14T19:15:57.234697+0000","last_became_peered":"2023-07-14T19:15:57.234697+0000","last_unstale":"2023-07-14T19:16:08.026032+0000","last_undegraded":"2023-07-14T19:16:08.026032+0000","last_fullsized":"2023-07-14T19:16:08.026032+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:15:56.210166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:15:56.210166+0000","last_clean_scrub_stamp":"2023-07-14T19:15:56.210166+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T21:46:18.232628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-07-14T19:15:59.722957+0000","last_change":"2023-07-14T19:15:58.241753+0000","last_active":"2023-07-14T19:15:59.722957+0000","last_peered":"2023-07-14T19:15:59.722957+0000","last_clean":"2023-07-14T19:15:59.722957+0000","last_became_active":"2023-07-14T19:15:58.241485+0000","last_became_peered":"2023-07-14T19:15:58.241485+0000","last_unstale":"2023-07-14T19:15:59.722957+0000","last_undegraded":"2023-07-14T19:15:59.722957+0000","last_fullsized":"2023-07-14T19:15:59.722957+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_clean_scrub_stamp":"2023-07-14T19:15:57.219223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T03:02:48.399750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"81'7","reported_seq":22,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-07-14T19:16:04.618776+0000","last_change":"2023-07-14T19:15:57.240019+0000","last_active":"2023-07-14T19:16:04.618776+0000","last_peered":"2023-07-14T19:16:04.618776+0000","last_clean":"2023-07-14T19:16:04.618776+0000","last_became_active":"2023-07-14T19:15:57.239521+0000","last_became_peered":"2023-07-14T19:15:57.239521+0000","last_unstale":"2023-07-14T19:16:04.618776+0000","last_undegraded":"2023-07-14T19:16:04.618776+0000","last_fullsized":"2023-07-14T19:16:04.618776+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:15:56.210166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:15:56.210166+0000","last_clean_scrub_stamp":"2023-07-14T19:15:56.210166+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T23:32:40.283168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-07-14T19:15:59.573249+0000","last_change":"2023-07-14T19:15:58.248187+0000","last_active":"2023-07-14T19:15:59.573249+0000","last_peered":"2023-07-14T19:15:59.573249+0000","last_clean":"2023-07-14T19:15:59.573249+0000","last_became_active":"2023-07-14T19:15:58.247854+0000","last_became_peered":"2023-07-14T19:15:58.247854+0000","last_unstale":"2023-07-14T19:15:59.573249+0000","last_undegraded":"2023-07-14T19:15:59.573249+0000","last_fullsized":"2023-07-14T19:15:59.573249+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_clean_scrub_stamp":"2023-07-14T19:15:57.219223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T21:01:16.786236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"81'2","reported_seq":16,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-07-14T19:16:01.572678+0000","last_change":"2023-07-14T19:15:57.245012+0000","last_active":"2023-07-14T19:16:01.572678+0000","last_peered":"2023-07-14T19:16:01.572678+0000","last_clean":"2023-07-14T19:16:01.572678+0000","last_became_active":"2023-07-14T19:15:57.244776+0000","last_became_peered":"2023-07-14T19:15:57.244776+0000","last_unstale":"2023-07-14T19:16:01.572678+0000","last_undegraded":"2023-07-14T19:16:01.572678+0000","last_fullsized":"2023-07-14T19:16:01.572678+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:15:56.210166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:15:56.210166+0000","last_clean_scrub_stamp":"2023-07-14T19:15:56.210166+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T20:16:41.721374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-07-14T19:15:59.232170+0000","last_change":"2023-07-14T19:15:58.247704+0000","last_active":"2023-07-14T19:15:59.232170+0000","last_peered":"2023-07-14T19:15:59.232170+0000","last_clean":"2023-07-14T19:15:59.232170+0000","last_became_active":"2023-07-14T19:15:58.247364+0000","last_became_peered":"2023-07-14T19:15:58.247364+0000","last_unstale":"2023-07-14T19:15:59.232170+0000","last_undegraded":"2023-07-14T19:15:59.232170+0000","last_fullsized":"2023-07-14T19:15:59.232170+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_clean_scrub_stamp":"2023-07-14T19:15:57.219223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T19:44:25.096100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"81'7","reported_seq":21,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-07-14T19:16:04.619759+0000","last_change":"2023-07-14T19:15:57.234991+0000","last_active":"2023-07-14T19:16:04.619759+0000","last_peered":"2023-07-14T19:16:04.619759+0000","last_clean":"2023-07-14T19:16:04.619759+0000","last_became_active":"2023-07-14T19:15:57.234680+0000","last_became_peered":"2023-07-14T19:15:57.234680+0000","last_unstale":"2023-07-14T19:16:04.619759+0000","last_undegraded":"2023-07-14T19:16:04.619759+0000","last_fullsized":"2023-07-14T19:16:04.619759+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:15:56.210166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:15:56.210166+0000","last_clean_scrub_stamp":"2023-07-14T19:15:56.210166+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T03:09:07.479398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-07-14T19:15:59.938707+0000","last_change":"2023-07-14T19:15:58.247224+0000","last_active":"2023-07-14T19:15:59.938707+0000","last_peered":"2023-07-14T19:15:59.938707+0000","last_clean":"2023-07-14T19:15:59.938707+0000","last_became_active":"2023-07-14T19:15:58.246889+0000","last_became_peered":"2023-07-14T19:15:58.246889+0000","last_unstale":"2023-07-14T19:15:59.938707+0000","last_undegraded":"2023-07-14T19:15:59.938707+0000","last_fullsized":"2023-07-14T19:15:59.938707+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_clean_scrub_stamp":"2023-07-14T19:15:57.219223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:25:39.860559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"81'3","reported_seq":17,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-07-14T19:16:05.868855+0000","last_change":"2023-07-14T19:15:57.246491+0000","last_active":"2023-07-14T19:16:05.868855+0000","last_peered":"2023-07-14T19:16:05.868855+0000","last_clean":"2023-07-14T19:16:05.868855+0000","last_became_active":"2023-07-14T19:15:57.246178+0000","last_became_peered":"2023-07-14T19:15:57.246178+0000","last_unstale":"2023-07-14T19:16:05.868855+0000","last_undegraded":"2023-07-14T19:16:05.868855+0000","last_fullsized":"2023-07-14T19:16:05.868855+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:15:56.210166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:15:56.210166+0000","last_clean_scrub_stamp":"2023-07-14T19:15:56.210166+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T03:15:13.275825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-07-14T19:15:59.723335+0000","last_change":"2023-07-14T19:15:58.247005+0000","last_active":"2023-07-14T19:15:59.723335+0000","last_peered":"2023-07-14T19:15:59.723335+0000","last_clean":"2023-07-14T19:15:59.723335+0000","last_became_active":"2023-07-14T19:15:58.246156+0000","last_became_peered":"2023-07-14T19:15:58.246156+0000","last_unstale":"2023-07-14T19:15:59.723335+0000","last_undegraded":"2023-07-14T19:15:59.723335+0000","last_fullsized":"2023-07-14T19:15:59.723335+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_clean_scrub_stamp":"2023-07-14T19:15:57.219223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T20:11:25.410164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"81'4","reported_seq":19,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-07-14T19:16:01.572721+0000","last_change":"2023-07-14T19:15:57.245551+0000","last_active":"2023-07-14T19:16:01.572721+0000","last_peered":"2023-07-14T19:16:01.572721+0000","last_clean":"2023-07-14T19:16:01.572721+0000","last_became_active":"2023-07-14T19:15:57.245179+0000","last_became_peered":"2023-07-14T19:15:57.245179+0000","last_unstale":"2023-07-14T19:16:01.572721+0000","last_undegraded":"2023-07-14T19:16:01.572721+0000","last_fullsized":"2023-07-14T19:16:01.572721+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:15:56.210166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:15:56.210166+0000","last_clean_scrub_stamp":"2023-07-14T19:15:56.210166+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T00:35:19.482494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-07-14T19:15:59.722416+0000","last_change":"2023-07-14T19:15:58.250454+0000","last_active":"2023-07-14T19:15:59.722416+0000","last_peered":"2023-07-14T19:15:59.722416+0000","last_clean":"2023-07-14T19:15:59.722416+0000","last_became_active":"2023-07-14T19:15:58.250145+0000","last_became_peered":"2023-07-14T19:15:58.250145+0000","last_unstale":"2023-07-14T19:15:59.722416+0000","last_undegraded":"2023-07-14T19:15:59.722416+0000","last_fullsized":"2023-07-14T19:15:59.722416+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_clean_scrub_stamp":"2023-07-14T19:15:57.219223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T23:46:27.565214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"81'5","reported_seq":64,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-07-14T19:16:46.707740+0000","last_change":"2023-07-14T19:15:57.247545+0000","last_active":"2023-07-14T19:16:46.707740+0000","last_peered":"2023-07-14T19:16:46.707740+0000","last_clean":"2023-07-14T19:16:46.707740+0000","last_became_active":"2023-07-14T19:15:57.247201+0000","last_became_peered":"2023-07-14T19:15:57.247201+0000","last_unstale":"2023-07-14T19:16:46.707740+0000","last_undegraded":"2023-07-14T19:16:46.707740+0000","last_fullsized":"2023-07-14T19:16:46.707740+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:15:56.210166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:15:56.210166+0000","last_clean_scrub_stamp":"2023-07-14T19:15:56.210166+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T23:52:58.892773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1122,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":45,"num_read_kb":2,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-07-14T19:15:59.722674+0000","last_change":"2023-07-14T19:15:58.252144+0000","last_active":"2023-07-14T19:15:59.722674+0000","last_peered":"2023-07-14T19:15:59.722674+0000","last_clean":"2023-07-14T19:15:59.722674+0000","last_became_active":"2023-07-14T19:15:58.251844+0000","last_became_peered":"2023-07-14T19:15:58.251844+0000","last_unstale":"2023-07-14T19:15:59.722674+0000","last_undegraded":"2023-07-14T19:15:59.722674+0000","last_fullsized":"2023-07-14T19:15:59.722674+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_clean_scrub_stamp":"2023-07-14T19:15:57.219223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:19:09.690589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"81'3","reported_seq":18,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-07-14T19:16:04.617360+0000","last_change":"2023-07-14T19:15:57.239560+0000","last_active":"2023-07-14T19:16:04.617360+0000","last_peered":"2023-07-14T19:16:04.617360+0000","last_clean":"2023-07-14T19:16:04.617360+0000","last_became_active":"2023-07-14T19:15:57.239250+0000","last_became_peered":"2023-07-14T19:15:57.239250+0000","last_unstale":"2023-07-14T19:16:04.617360+0000","last_undegraded":"2023-07-14T19:16:04.617360+0000","last_fullsized":"2023-07-14T19:16:04.617360+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:15:56.210166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:15:56.210166+0000","last_clean_scrub_stamp":"2023-07-14T19:15:56.210166+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T21:04:10.055051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"81'1","reported_seq":13,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-07-14T19:16:01.578417+0000","last_change":"2023-07-14T19:15:58.242144+0000","last_active":"2023-07-14T19:16:01.578417+0000","last_peered":"2023-07-14T19:16:01.578417+0000","last_clean":"2023-07-14T19:16:01.578417+0000","last_became_active":"2023-07-14T19:15:58.241597+0000","last_became_peered":"2023-07-14T19:15:58.241597+0000","last_unstale":"2023-07-14T19:16:01.578417+0000","last_undegraded":"2023-07-14T19:16:01.578417+0000","last_fullsized":"2023-07-14T19:16:01.578417+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_clean_scrub_stamp":"2023-07-14T19:15:57.219223+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T22:43:21.380478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":123,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"81'5","reported_seq":19,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-07-14T19:16:04.616641+0000","last_change":"2023-07-14T19:15:57.247059+0000","last_active":"2023-07-14T19:16:04.616641+0000","last_peered":"2023-07-14T19:16:04.616641+0000","last_clean":"2023-07-14T19:16:04.616641+0000","last_became_active":"2023-07-14T19:15:57.246736+0000","last_became_peered":"2023-07-14T19:15:57.246736+0000","last_unstale":"2023-07-14T19:16:04.616641+0000","last_undegraded":"2023-07-14T19:16:04.616641+0000","last_fullsized":"2023-07-14T19:16:04.616641+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:15:56.210166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:15:56.210166+0000","last_clean_scrub_stamp":"2023-07-14T19:15:56.210166+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T20:57:38.014630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-07-14T19:15:59.722005+0000","last_change":"2023-07-14T19:15:58.244180+0000","last_active":"2023-07-14T19:15:59.722005+0000","last_peered":"2023-07-14T19:15:59.722005+0000","last_clean":"2023-07-14T19:15:59.722005+0000","last_became_active":"2023-07-14T19:15:58.243885+0000","last_became_peered":"2023-07-14T19:15:58.243885+0000","last_unstale":"2023-07-14T19:15:59.722005+0000","last_undegraded":"2023-07-14T19:15:59.722005+0000","last_fullsized":"2023-07-14T19:15:59.722005+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_clean_scrub_stamp":"2023-07-14T19:15:57.219223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:56:30.077276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-07-14T19:15:59.938781+0000","last_change":"2023-07-14T19:15:58.246944+0000","last_active":"2023-07-14T19:15:59.938781+0000","last_peered":"2023-07-14T19:15:59.938781+0000","last_clean":"2023-07-14T19:15:59.938781+0000","last_became_active":"2023-07-14T19:15:58.246672+0000","last_became_peered":"2023-07-14T19:15:58.246672+0000","last_unstale":"2023-07-14T19:15:59.938781+0000","last_undegraded":"2023-07-14T19:15:59.938781+0000","last_fullsized":"2023-07-14T19:15:59.938781+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_clean_scrub_stamp":"2023-07-14T19:15:57.219223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:59:58.787272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-07-14T19:15:59.720933+0000","last_change":"2023-07-14T19:15:58.246761+0000","last_active":"2023-07-14T19:15:59.720933+0000","last_peered":"2023-07-14T19:15:59.720933+0000","last_clean":"2023-07-14T19:15:59.720933+0000","last_became_active":"2023-07-14T19:15:58.246318+0000","last_became_peered":"2023-07-14T19:15:58.246318+0000","last_unstale":"2023-07-14T19:15:59.720933+0000","last_undegraded":"2023-07-14T19:15:59.720933+0000","last_fullsized":"2023-07-14T19:15:59.720933+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_clean_scrub_stamp":"2023-07-14T19:15:57.219223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T22:22:00.801887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-07-14T19:15:59.721079+0000","last_change":"2023-07-14T19:15:58.251465+0000","last_active":"2023-07-14T19:15:59.721079+0000","last_peered":"2023-07-14T19:15:59.721079+0000","last_clean":"2023-07-14T19:15:59.721079+0000","last_became_active":"2023-07-14T19:15:58.251138+0000","last_became_peered":"2023-07-14T19:15:58.251138+0000","last_unstale":"2023-07-14T19:15:59.721079+0000","last_undegraded":"2023-07-14T19:15:59.721079+0000","last_fullsized":"2023-07-14T19:15:59.721079+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_clean_scrub_stamp":"2023-07-14T19:15:57.219223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:53:28.483753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-07-14T19:15:59.573084+0000","last_change":"2023-07-14T19:15:58.244335+0000","last_active":"2023-07-14T19:15:59.573084+0000","last_peered":"2023-07-14T19:15:59.573084+0000","last_clean":"2023-07-14T19:15:59.573084+0000","last_became_active":"2023-07-14T19:15:58.243955+0000","last_became_peered":"2023-07-14T19:15:58.243955+0000","last_unstale":"2023-07-14T19:15:59.573084+0000","last_undegraded":"2023-07-14T19:15:59.573084+0000","last_fullsized":"2023-07-14T19:15:59.573084+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_clean_scrub_stamp":"2023-07-14T19:15:57.219223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T06:54:43.644336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-07-14T19:15:59.723074+0000","last_change":"2023-07-14T19:15:58.244373+0000","last_active":"2023-07-14T19:15:59.723074+0000","last_peered":"2023-07-14T19:15:59.723074+0000","last_clean":"2023-07-14T19:15:59.723074+0000","last_became_active":"2023-07-14T19:15:58.244042+0000","last_became_peered":"2023-07-14T19:15:58.244042+0000","last_unstale":"2023-07-14T19:15:59.723074+0000","last_undegraded":"2023-07-14T19:15:59.723074+0000","last_fullsized":"2023-07-14T19:15:59.723074+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_clean_scrub_stamp":"2023-07-14T19:15:57.219223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T03:40:02.041528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-07-14T19:15:59.938908+0000","last_change":"2023-07-14T19:15:58.240770+0000","last_active":"2023-07-14T19:15:59.938908+0000","last_peered":"2023-07-14T19:15:59.938908+0000","last_clean":"2023-07-14T19:15:59.938908+0000","last_became_active":"2023-07-14T19:15:58.240408+0000","last_became_peered":"2023-07-14T19:15:58.240408+0000","last_unstale":"2023-07-14T19:15:59.938908+0000","last_undegraded":"2023-07-14T19:15:59.938908+0000","last_fullsized":"2023-07-14T19:15:59.938908+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_clean_scrub_stamp":"2023-07-14T19:15:57.219223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T19:45:10.482496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-07-14T19:15:59.721267+0000","last_change":"2023-07-14T19:15:58.247557+0000","last_active":"2023-07-14T19:15:59.721267+0000","last_peered":"2023-07-14T19:15:59.721267+0000","last_clean":"2023-07-14T19:15:59.721267+0000","last_became_active":"2023-07-14T19:15:58.247228+0000","last_became_peered":"2023-07-14T19:15:58.247228+0000","last_unstale":"2023-07-14T19:15:59.721267+0000","last_undegraded":"2023-07-14T19:15:59.721267+0000","last_fullsized":"2023-07-14T19:15:59.721267+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_clean_scrub_stamp":"2023-07-14T19:15:57.219223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T05:18:52.409847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-07-14T19:15:59.722818+0000","last_change":"2023-07-14T19:15:58.247013+0000","last_active":"2023-07-14T19:15:59.722818+0000","last_peered":"2023-07-14T19:15:59.722818+0000","last_clean":"2023-07-14T19:15:59.722818+0000","last_became_active":"2023-07-14T19:15:58.246729+0000","last_became_peered":"2023-07-14T19:15:58.246729+0000","last_unstale":"2023-07-14T19:15:59.722818+0000","last_undegraded":"2023-07-14T19:15:59.722818+0000","last_fullsized":"2023-07-14T19:15:59.722818+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_clean_scrub_stamp":"2023-07-14T19:15:57.219223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T22:17:27.678424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-07-14T19:15:59.231044+0000","last_change":"2023-07-14T19:15:58.243023+0000","last_active":"2023-07-14T19:15:59.231044+0000","last_peered":"2023-07-14T19:15:59.231044+0000","last_clean":"2023-07-14T19:15:59.231044+0000","last_became_active":"2023-07-14T19:15:58.242630+0000","last_became_peered":"2023-07-14T19:15:58.242630+0000","last_unstale":"2023-07-14T19:15:59.231044+0000","last_undegraded":"2023-07-14T19:15:59.231044+0000","last_fullsized":"2023-07-14T19:15:59.231044+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_clean_scrub_stamp":"2023-07-14T19:15:57.219223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T23:27:40.017638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-07-14T19:15:59.723119+0000","last_change":"2023-07-14T19:15:58.246205+0000","last_active":"2023-07-14T19:15:59.723119+0000","last_peered":"2023-07-14T19:15:59.723119+0000","last_clean":"2023-07-14T19:15:59.723119+0000","last_became_active":"2023-07-14T19:15:58.245720+0000","last_became_peered":"2023-07-14T19:15:58.245720+0000","last_unstale":"2023-07-14T19:15:59.723119+0000","last_undegraded":"2023-07-14T19:15:59.723119+0000","last_fullsized":"2023-07-14T19:15:59.723119+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_clean_scrub_stamp":"2023-07-14T19:15:57.219223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T05:22:46.730021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-07-14T19:15:59.722477+0000","last_change":"2023-07-14T19:15:58.248611+0000","last_active":"2023-07-14T19:15:59.722477+0000","last_peered":"2023-07-14T19:15:59.722477+0000","last_clean":"2023-07-14T19:15:59.722477+0000","last_became_active":"2023-07-14T19:15:58.248352+0000","last_became_peered":"2023-07-14T19:15:58.248352+0000","last_unstale":"2023-07-14T19:15:59.722477+0000","last_undegraded":"2023-07-14T19:15:59.722477+0000","last_fullsized":"2023-07-14T19:15:59.722477+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_clean_scrub_stamp":"2023-07-14T19:15:57.219223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T06:43:46.682942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-07-14T19:15:59.722246+0000","last_change":"2023-07-14T19:15:58.243167+0000","last_active":"2023-07-14T19:15:59.722246+0000","last_peered":"2023-07-14T19:15:59.722246+0000","last_clean":"2023-07-14T19:15:59.722246+0000","last_became_active":"2023-07-14T19:15:58.241770+0000","last_became_peered":"2023-07-14T19:15:58.241770+0000","last_unstale":"2023-07-14T19:15:59.722246+0000","last_undegraded":"2023-07-14T19:15:59.722246+0000","last_fullsized":"2023-07-14T19:15:59.722246+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_clean_scrub_stamp":"2023-07-14T19:15:57.219223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T05:56:34.485106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-07-14T19:15:59.938811+0000","last_change":"2023-07-14T19:15:58.243972+0000","last_active":"2023-07-14T19:15:59.938811+0000","last_peered":"2023-07-14T19:15:59.938811+0000","last_clean":"2023-07-14T19:15:59.938811+0000","last_became_active":"2023-07-14T19:15:58.243685+0000","last_became_peered":"2023-07-14T19:15:58.243685+0000","last_unstale":"2023-07-14T19:15:59.938811+0000","last_undegraded":"2023-07-14T19:15:59.938811+0000","last_fullsized":"2023-07-14T19:15:59.938811+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:15:57.219223+0000","last_clean_scrub_stamp":"2023-07-14T19:15:57.219223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:44:54.955614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":246,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":442368,"data_stored":4722944,"data_compressed":280807,"data_compressed_allocated":360448,"data_compressed_original":4669440,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":63176,"num_objects":98,"num_object_clones":0,"num_object_copies":294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":0,"num_read":443,"num_read_kb":287,"num_write":227,"num_write_kb":124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":552960,"data_stored":189528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":199,"ondisk_log_size":199,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":76,"seq":326417514518,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27876,"kb_used_data":860,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717276,"statfs":{"total":95995035648,"available":95966490624,"internally_reserved":0,"allocated":880640,"data_stored":417064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jul 14 19:16:16 2023","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":0.56699999999999995,"15min":0.56699999999999995},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":1.1160000000000001,"5min":1.1160000000000001,"15min":1.1160000000000001},"last":0.73099999999999998},{"interface":"front","average":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":1.2150000000000001,"5min":1.2150000000000001,"15min":1.2150000000000001},"last":0.56599999999999995}]},{"osd":1,"last update":"Fri Jul 14 19:16:16 2023","interfaces":[{"interface":"back","average":{"1min":0.64000000000000001,"5min":0.64000000000000001,"15min":0.64000000000000001},"min":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":1.089,"5min":1.089,"15min":1.089},"last":0.82899999999999996},{"interface":"front","average":{"1min":0.61899999999999999,"5min":0.61899999999999999,"15min":0.61899999999999999},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":1.1779999999999999,"5min":1.1779999999999999,"15min":1.1779999999999999},"last":0.77600000000000002}]},{"osd":2,"last update":"Fri Jul 14 19:16:16 2023","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.496,"15min":0.496},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.79700000000000004,"5min":0.79700000000000004,"15min":0.79700000000000004},"last":0.49299999999999999},{"interface":"front","average":{"1min":0.47999999999999998,"5min":0.47999999999999998,"15min":0.47999999999999998},"min":{"1min":0.30599999999999999,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.83699999999999997,"5min":0.83699999999999997,"15min":0.83699999999999997},"last":0.432}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.436}]},{"osd":4,"last update":"Fri Jul 14 19:16:16 2023","interfaces":[{"interface":"back","average":{"1min":0.70399999999999996,"5min":0.70399999999999996,"15min":0.70399999999999996},"min":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":1.3919999999999999,"5min":1.3919999999999999,"15min":1.3919999999999999},"last":0.85899999999999999},{"interface":"front","average":{"1min":0.66600000000000004,"5min":0.66600000000000004,"15min":0.66600000000000004},"min":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":1.236,"5min":1.236,"15min":1.236},"last":0.65200000000000002}]},{"osd":5,"last update":"Fri Jul 14 19:16:16 2023","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":0.61599999999999999,"15min":0.61599999999999999},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":1.107,"5min":1.107,"15min":1.107},"last":0.40500000000000003},{"interface":"front","average":{"1min":0.60699999999999998,"5min":0.60699999999999998,"15min":0.60699999999999998},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.93700000000000006,"5min":0.93700000000000006,"15min":0.93700000000000006},"last":0.58999999999999997}]},{"osd":6,"last update":"Fri Jul 14 19:16:16 2023","interfaces":[{"interface":"back","average":{"1min":0.66600000000000004,"5min":0.66600000000000004,"15min":0.66600000000000004},"min":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":1.196,"5min":1.196,"15min":1.196},"last":0.70799999999999996},{"interface":"front","average":{"1min":0.72199999999999998,"5min":0.72199999999999998,"15min":0.72199999999999998},"min":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"max":{"1min":1.379,"5min":1.379,"15min":1.379},"last":0.67700000000000005}]},{"osd":7,"last update":"Fri Jul 14 19:16:16 2023","interfaces":[{"interface":"back","average":{"1min":0.75700000000000001,"5min":0.75700000000000001,"15min":0.75700000000000001},"min":{"1min":0.54700000000000004,"5min":0.54700000000000004,"15min":0.54700000000000004},"max":{"1min":1.071,"5min":1.071,"15min":1.071},"last":0.89100000000000001},{"interface":"front","average":{"1min":0.75,"5min":0.75,"15min":0.75},"min":{"1min":0.57599999999999996,"5min":0.57599999999999996,"15min":0.57599999999999996},"max":{"1min":1.1259999999999999,"5min":1.1259999999999999,"15min":1.1259999999999999},"last":0.81899999999999995}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63}]},{"osd":9,"last update":"Fri Jul 14 19:16:16 2023","interfaces":[{"interface":"back","average":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"min":{"1min":0.56100000000000005,"5min":0.56100000000000005,"15min":0.56100000000000005},"max":{"1min":1.462,"5min":1.462,"15min":1.462},"last":0.80700000000000005},{"interface":"front","average":{"1min":0.77200000000000002,"5min":0.77200000000000002,"15min":0.77200000000000002},"min":{"1min":0.56299999999999994,"5min":0.56299999999999994,"15min":0.56299999999999994},"max":{"1min":1.329,"5min":1.329,"15min":1.329},"last":0.92600000000000005}]},{"osd":10,"last update":"Fri Jul 14 19:16:16 2023","interfaces":[{"interface":"back","average":{"1min":0.76500000000000001,"5min":0.76500000000000001,"15min":0.76500000000000001},"min":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"max":{"1min":1.091,"5min":1.091,"15min":1.091},"last":0.75},{"interface":"front","average":{"1min":0.82199999999999995,"5min":0.82199999999999995,"15min":0.82199999999999995},"min":{"1min":0.59399999999999997,"5min":0.59399999999999997,"15min":0.59399999999999997},"max":{"1min":1.5049999999999999,"5min":1.5049999999999999,"15min":1.5049999999999999},"last":0.76300000000000001}]}]},{"osd":10,"up_from":69,"seq":296352743449,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28024,"kb_used_data":1008,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717128,"statfs":{"total":95995035648,"available":95966339072,"internally_reserved":0,"allocated":1032192,"data_stored":1065045,"data_compressed":35714,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51800000000000002}]},{"osd":1,"last update":"Fri Jul 14 19:16:00 2023","interfaces":[{"interface":"back","average":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.66500000000000004,"5min":0.66500000000000004,"15min":0.66500000000000004},"last":0.38600000000000001},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.76900000000000002,"5min":0.76900000000000002,"15min":0.76900000000000002},"last":0.41399999999999998}]},{"osd":2,"last update":"Fri Jul 14 19:16:00 2023","interfaces":[{"interface":"back","average":{"1min":0.67100000000000004,"5min":0.67100000000000004,"15min":0.67100000000000004},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":1.0009999999999999,"5min":1.0009999999999999,"15min":1.0009999999999999},"last":0.625},{"interface":"front","average":{"1min":0.751,"5min":0.751,"15min":0.751},"min":{"1min":0.51500000000000001,"5min":0.51500000000000001,"15min":0.51500000000000001},"max":{"1min":1.1379999999999999,"5min":1.1379999999999999,"15min":1.1379999999999999},"last":0.65700000000000003}]},{"osd":3,"last update":"Fri Jul 14 19:16:00 2023","interfaces":[{"interface":"back","average":{"1min":0.70099999999999996,"5min":0.70099999999999996,"15min":0.70099999999999996},"min":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":0.92200000000000004,"5min":0.92200000000000004,"15min":0.92200000000000004},"last":0.58799999999999997},{"interface":"front","average":{"1min":0.71599999999999997,"5min":0.71599999999999997,"15min":0.71599999999999997},"min":{"1min":0.45300000000000001,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":0.90900000000000003,"5min":0.90900000000000003,"15min":0.90900000000000003},"last":0.60299999999999998}]},{"osd":4,"last update":"Fri Jul 14 19:16:00 2023","interfaces":[{"interface":"back","average":{"1min":0.64700000000000002,"5min":0.64700000000000002,"15min":0.64700000000000002},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.83199999999999996,"5min":0.83199999999999996,"15min":0.83199999999999996},"last":0.53800000000000003},{"interface":"front","average":{"1min":0.59999999999999998,"5min":0.59999999999999998,"15min":0.59999999999999998},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.878,"5min":0.878,"15min":0.878},"last":0.55600000000000005}]},{"osd":5,"last update":"Fri Jul 14 19:16:00 2023","interfaces":[{"interface":"back","average":{"1min":0.76000000000000001,"5min":0.76000000000000001,"15min":0.76000000000000001},"min":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"max":{"1min":1.1259999999999999,"5min":1.1259999999999999,"15min":1.1259999999999999},"last":0.73799999999999999},{"interface":"front","average":{"1min":0.77900000000000003,"5min":0.77900000000000003,"15min":0.77900000000000003},"min":{"1min":0.56299999999999994,"5min":0.56299999999999994,"15min":0.56299999999999994},"max":{"1min":1.018,"5min":1.018,"15min":1.018},"last":0.753}]},{"osd":6,"last update":"Fri Jul 14 19:16:00 2023","interfaces":[{"interface":"back","average":{"1min":0.82799999999999996,"5min":0.82799999999999996,"15min":0.82799999999999996},"min":{"1min":0.56899999999999995,"5min":0.56899999999999995,"15min":0.56899999999999995},"max":{"1min":1.294,"5min":1.294,"15min":1.294},"last":0.70699999999999996},{"interface":"front","average":{"1min":0.80400000000000005,"5min":0.80400000000000005,"15min":0.80400000000000005},"min":{"1min":0.57999999999999996,"5min":0.57999999999999996,"15min":0.57999999999999996},"max":{"1min":1.0960000000000001,"5min":1.0960000000000001,"15min":1.0960000000000001},"last":0.67200000000000004}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.49099999999999999}]},{"osd":8,"last update":"Fri Jul 14 19:16:00 2023","interfaces":[{"interface":"back","average":{"1min":0.84599999999999997,"5min":0.84599999999999997,"15min":0.84599999999999997},"min":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"max":{"1min":1.0349999999999999,"5min":1.0349999999999999,"15min":1.0349999999999999},"last":0.77400000000000002},{"interface":"front","average":{"1min":0.88900000000000001,"5min":0.88900000000000001,"15min":0.88900000000000001},"min":{"1min":0.60499999999999998,"5min":0.60499999999999998,"15min":0.60499999999999998},"max":{"1min":1.369,"5min":1.369,"15min":1.369},"last":0.78800000000000003}]},{"osd":9,"last update":"Fri Jul 14 19:16:00 2023","interfaces":[{"interface":"back","average":{"1min":0.88600000000000001,"5min":0.88600000000000001,"15min":0.88600000000000001},"min":{"1min":0.65000000000000002,"5min":0.65000000000000002,"15min":0.65000000000000002},"max":{"1min":1.1759999999999999,"5min":1.1759999999999999,"15min":1.1759999999999999},"last":0.71999999999999997},{"interface":"front","average":{"1min":0.88200000000000001,"5min":0.88200000000000001,"15min":0.88200000000000001},"min":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":1.468,"5min":1.468,"15min":1.468},"last":0.68700000000000006}]},{"osd":11,"last update":"Fri Jul 14 19:16:18 2023","interfaces":[{"interface":"back","average":{"1min":0.85899999999999999,"5min":0.85899999999999999,"15min":0.85899999999999999},"min":{"1min":0.69499999999999995,"5min":0.69499999999999995,"15min":0.69499999999999995},"max":{"1min":1.105,"5min":1.105,"15min":1.105},"last":0.81899999999999995},{"interface":"front","average":{"1min":0.82899999999999996,"5min":0.82899999999999996,"15min":0.82899999999999996},"min":{"1min":0.56799999999999995,"5min":0.56799999999999995,"15min":0.56799999999999995},"max":{"1min":1.0489999999999999,"5min":1.0489999999999999,"15min":1.0489999999999999},"last":0.80700000000000005}]}]},{"osd":9,"up_from":64,"seq":274877906973,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27900,"kb_used_data":884,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717252,"statfs":{"total":95995035648,"available":95966466048,"internally_reserved":0,"allocated":905216,"data_stored":417042,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jul 14 19:16:36 2023","interfaces":[{"interface":"back","average":{"1min":0.41299999999999998,"5min":0.47199999999999998,"15min":0.48199999999999998},"min":{"1min":0.28399999999999997,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.78900000000000003,"5min":0.82099999999999995,"15min":0.82099999999999995},"last":0.42599999999999999},{"interface":"front","average":{"1min":0.41899999999999998,"5min":0.45200000000000001,"15min":0.45800000000000002},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.61499999999999999,"5min":0.65300000000000002,"15min":0.65300000000000002},"last":0.36599999999999999}]},{"osd":1,"last update":"Fri Jul 14 19:16:36 2023","interfaces":[{"interface":"back","average":{"1min":0.58199999999999996,"5min":0.60999999999999999,"15min":0.61499999999999999},"min":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":1.1240000000000001,"5min":1.1240000000000001,"15min":1.1240000000000001},"last":0.47199999999999998},{"interface":"front","average":{"1min":0.60699999999999998,"5min":0.64700000000000002,"15min":0.65300000000000002},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.85399999999999998,"5min":0.85399999999999998,"15min":0.85399999999999998},"last":0.62}]},{"osd":2,"last update":"Fri Jul 14 19:16:36 2023","interfaces":[{"interface":"back","average":{"1min":0.60199999999999998,"5min":0.64600000000000002,"15min":0.65400000000000003},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.79700000000000004,"5min":0.92200000000000004,"15min":0.92200000000000004},"last":0.57399999999999995},{"interface":"front","average":{"1min":0.56100000000000005,"5min":0.63800000000000001,"15min":0.65100000000000002},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":0.71399999999999997,"5min":0.91000000000000003,"15min":0.91000000000000003},"last":0.49299999999999999}]},{"osd":3,"last update":"Fri Jul 14 19:16:36 2023","interfaces":[{"interface":"back","average":{"1min":0.40999999999999998,"5min":0.505,"15min":0.52100000000000002},"min":{"1min":0.30599999999999999,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.58599999999999997,"5min":0.876,"15min":0.876},"last":0.39100000000000001},{"interface":"front","average":{"1min":0.51600000000000001,"5min":0.59599999999999997,"15min":0.60999999999999999},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.73599999999999999,"5min":0.96499999999999997,"15min":0.96499999999999997},"last":0.52900000000000003}]},{"osd":4,"last update":"Fri Jul 14 19:16:36 2023","interfaces":[{"interface":"back","average":{"1min":0.60199999999999998,"5min":0.66800000000000004,"15min":0.67900000000000005},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.78000000000000003,"5min":1.0700000000000001,"15min":1.0700000000000001},"last":0.51800000000000002},{"interface":"front","average":{"1min":0.60799999999999998,"5min":0.65700000000000003,"15min":0.66500000000000004},"min":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":0.754,"5min":0.94099999999999995,"15min":0.94099999999999995},"last":0.54900000000000004}]},{"osd":5,"last update":"Fri Jul 14 19:16:36 2023","interfaces":[{"interface":"back","average":{"1min":0.68600000000000005,"5min":0.749,"15min":0.75900000000000001},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":1.8480000000000001,"5min":1.8480000000000001,"15min":1.8480000000000001},"last":0.58599999999999997},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.69599999999999995,"15min":0.71399999999999997},"min":{"1min":0.46000000000000002,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.81200000000000006,"5min":1.018,"15min":1.018},"last":0.50700000000000001}]},{"osd":6,"last update":"Fri Jul 14 19:16:36 2023","interfaces":[{"interface":"back","average":{"1min":0.71499999999999997,"5min":0.78100000000000003,"15min":0.79200000000000004},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":2.5720000000000001,"5min":2.5720000000000001,"15min":2.5720000000000001},"last":0.67800000000000005},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.73299999999999998,"15min":0.755},"min":{"1min":0.439,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.86199999999999999,"5min":1.145,"15min":1.145},"last":0.56299999999999994}]},{"osd":7,"last update":"Fri Jul 14 19:16:36 2023","interfaces":[{"interface":"back","average":{"1min":0.67700000000000005,"5min":0.76100000000000001,"15min":0.77500000000000002},"min":{"1min":0.55000000000000004,"5min":0.48699999999999999,"15min":0.48699999999999999},"max":{"1min":0.88700000000000001,"5min":1.0800000000000001,"15min":1.0800000000000001},"last":0.63700000000000001},{"interface":"front","average":{"1min":0.73799999999999999,"5min":0.81200000000000006,"15min":0.82399999999999995},"min":{"1min":0.45200000000000001,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":0.999,"5min":1.0469999999999999,"15min":1.0469999999999999},"last":0.71999999999999997}]},{"osd":8,"last update":"Fri Jul 14 19:16:36 2023","interfaces":[{"interface":"back","average":{"1min":0.71499999999999997,"5min":0.83899999999999997,"15min":0.85999999999999999},"min":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"max":{"1min":0.94999999999999996,"5min":1.242,"15min":1.242},"last":0.60899999999999999},{"interface":"front","average":{"1min":0.73699999999999999,"5min":0.80900000000000005,"15min":0.82099999999999995},"min":{"1min":0.59999999999999998,"5min":0.58499999999999996,"15min":0.58499999999999996},"max":{"1min":1.5600000000000001,"5min":1.5600000000000001,"15min":1.5600000000000001},"last":0.65800000000000003}]},{"osd":10,"last update":"Fri Jul 14 19:15:58 2023","interfaces":[{"interface":"back","average":{"1min":0.79200000000000004,"5min":0.79200000000000004,"15min":0.79200000000000004},"min":{"1min":0.54900000000000004,"5min":0.54900000000000004,"15min":0.54900000000000004},"max":{"1min":1.0820000000000001,"5min":1.0820000000000001,"15min":1.0820000000000001},"last":0.75700000000000001},{"interface":"front","average":{"1min":0.79100000000000004,"5min":0.79100000000000004,"15min":0.79100000000000004},"min":{"1min":0.59599999999999997,"5min":0.59599999999999997,"15min":0.59599999999999997},"max":{"1min":1.044,"5min":1.044,"15min":1.044},"last":0.64800000000000002}]}]},{"osd":8,"up_from":57,"seq":244813135907,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27892,"kb_used_data":876,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717260,"statfs":{"total":95995035648,"available":95966474240,"internally_reserved":0,"allocated":897024,"data_stored":417089,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jul 14 19:16:16 2023","interfaces":[{"interface":"back","average":{"1min":0.51500000000000001,"5min":0.51900000000000002,"15min":0.52000000000000002},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.746,"5min":0.78600000000000003,"15min":0.78600000000000003},"last":0.65100000000000002},{"interface":"front","average":{"1min":0.51300000000000001,"5min":0.57099999999999995,"15min":0.58099999999999996},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.72799999999999998,"5min":0.80500000000000005,"15min":0.80500000000000005},"last":0.68200000000000005}]},{"osd":1,"last update":"Fri Jul 14 19:16:16 2023","interfaces":[{"interface":"back","average":{"1min":0.64500000000000002,"5min":0.58999999999999997,"15min":0.58099999999999996},"min":{"1min":0.39700000000000002,"5min":0.375,"15min":0.375},"max":{"1min":0.81200000000000006,"5min":1.0069999999999999,"15min":1.0069999999999999},"last":0.66700000000000004},{"interface":"front","average":{"1min":0.64600000000000002,"5min":0.60299999999999998,"15min":0.59599999999999997},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":0.83399999999999996,"5min":0.91500000000000004,"15min":0.91500000000000004},"last":0.70499999999999996}]},{"osd":2,"last update":"Fri Jul 14 19:16:16 2023","interfaces":[{"interface":"back","average":{"1min":0.45900000000000002,"5min":0.43099999999999999,"15min":0.42599999999999999},"min":{"1min":0.28399999999999997,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.73799999999999999,"5min":0.73799999999999999,"15min":0.73799999999999999},"last":0.48699999999999999},{"interface":"front","average":{"1min":0.51600000000000001,"5min":0.49199999999999999,"15min":0.48799999999999999},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":0.76900000000000002,"5min":0.76900000000000002,"15min":0.76900000000000002},"last":0.53700000000000003}]},{"osd":3,"last update":"Fri Jul 14 19:16:16 2023","interfaces":[{"interface":"back","average":{"1min":0.55500000000000005,"5min":0.59899999999999998,"15min":0.60599999999999998},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.68100000000000005,"5min":0.84799999999999998,"15min":0.84799999999999998},"last":0.61199999999999999},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.63400000000000001,"15min":0.64300000000000002},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.82599999999999996,"5min":0.82599999999999996,"15min":0.82599999999999996},"last":0.78600000000000003}]},{"osd":4,"last update":"Fri Jul 14 19:16:16 2023","interfaces":[{"interface":"back","average":{"1min":0.64100000000000001,"5min":0.59299999999999997,"15min":0.58499999999999996},"min":{"1min":0.44900000000000001,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.90700000000000003,"5min":1.069,"15min":1.069},"last":0.56599999999999995},{"interface":"front","average":{"1min":0.68799999999999994,"5min":0.68500000000000005,"15min":0.68500000000000005},"min":{"1min":0.46300000000000002,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.99099999999999999,"5min":1.474,"15min":1.474},"last":0.72899999999999998}]},{"osd":5,"last update":"Fri Jul 14 19:16:16 2023","interfaces":[{"interface":"back","average":{"1min":0.58799999999999997,"5min":0.55800000000000005,"15min":0.55300000000000005},"min":{"1min":0.32400000000000001,"5min":0.313,"15min":0.313},"max":{"1min":0.88900000000000001,"5min":0.88900000000000001,"15min":0.88900000000000001},"last":0.76800000000000002},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.56699999999999995,"15min":0.56799999999999995},"min":{"1min":0.33500000000000002,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.77300000000000002,"5min":1.0349999999999999,"15min":1.0349999999999999},"last":0.63500000000000001}]},{"osd":6,"last update":"Fri Jul 14 19:16:16 2023","interfaces":[{"interface":"back","average":{"1min":0.66100000000000003,"5min":0.66300000000000003,"15min":0.66300000000000003},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":0.87,"5min":0.87,"15min":0.87},"last":0.85899999999999999},{"interface":"front","average":{"1min":0.69399999999999995,"5min":0.72399999999999998,"15min":0.72899999999999998},"min":{"1min":0.439,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":0.94299999999999995,"5min":1.1399999999999999,"15min":1.1399999999999999},"last":0.81499999999999995}]},{"osd":7,"last update":"Fri Jul 14 19:16:16 2023","interfaces":[{"interface":"back","average":{"1min":0.73899999999999999,"5min":0.72799999999999998,"15min":0.72599999999999998},"min":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":1.0289999999999999,"5min":1.1879999999999999,"15min":1.1879999999999999},"last":0.84699999999999998},{"interface":"front","average":{"1min":0.70299999999999996,"5min":0.72299999999999998,"15min":0.72599999999999998},"min":{"1min":0.46800000000000003,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":0.93400000000000005,"5min":1.093,"15min":1.093},"last":0.75}]},{"osd":9,"last update":"Fri Jul 14 19:16:36 2023","interfaces":[{"interface":"back","average":{"1min":0.75,"5min":0.753,"15min":0.753},"min":{"1min":0.47599999999999998,"5min":0.47599999999999998,"15min":0.47599999999999998},"max":{"1min":1.0129999999999999,"5min":1.0129999999999999,"15min":1.0129999999999999},"last":0.91000000000000003},{"interface":"front","average":{"1min":0.75900000000000001,"5min":0.75600000000000001,"15min":0.75600000000000001},"min":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":0.95399999999999996,"5min":0.97099999999999997,"15min":0.97099999999999997},"last":0.89200000000000002}]},{"osd":10,"last update":"Fri Jul 14 19:15:55 2023","interfaces":[{"interface":"back","average":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"min":{"1min":0.52700000000000002,"5min":0.52700000000000002,"15min":0.52700000000000002},"max":{"1min":1.006,"5min":1.006,"15min":1.006},"last":0.877},{"interface":"front","average":{"1min":0.73599999999999999,"5min":0.73599999999999999,"15min":0.73599999999999999},"min":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"max":{"1min":0.95999999999999996,"5min":0.95999999999999996,"15min":0.95999999999999996},"last":0.92400000000000004}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82799999999999996}]}]},{"osd":7,"up_from":50,"seq":214748364838,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27944,"kb_used_data":928,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717208,"statfs":{"total":95995035648,"available":95966420992,"internally_reserved":0,"allocated":950272,"data_stored":1009136,"data_compressed":35714,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jul 14 19:15:51 2023","interfaces":[{"interface":"back","average":{"1min":0.57199999999999995,"5min":0.53300000000000003,"15min":0.52700000000000002},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.82999999999999996,"5min":0.82999999999999996,"15min":0.82999999999999996},"last":0.5},{"interface":"front","average":{"1min":0.55200000000000005,"5min":0.55600000000000005,"15min":0.55600000000000005},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.78100000000000003,"5min":0.89300000000000002,"15min":0.89300000000000002},"last":0.51800000000000002}]},{"osd":1,"last update":"Fri Jul 14 19:15:51 2023","interfaces":[{"interface":"back","average":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"min":{"1min":0.25700000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.67100000000000004,"5min":0.67100000000000004,"15min":0.67100000000000004},"last":0.42599999999999999},{"interface":"front","average":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"min":{"1min":0.254,"5min":0.254,"15min":0.254},"max":{"1min":0.69099999999999995,"5min":0.91300000000000003,"15min":0.91300000000000003},"last":0.58199999999999996}]},{"osd":2,"last update":"Fri Jul 14 19:15:51 2023","interfaces":[{"interface":"back","average":{"1min":0.60199999999999998,"5min":0.59299999999999997,"15min":0.59099999999999997},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.78600000000000003,"5min":0.83999999999999997,"15min":0.83999999999999997},"last":0.60299999999999998},{"interface":"front","average":{"1min":0.65600000000000003,"5min":0.628,"15min":0.623},"min":{"1min":0.43099999999999999,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":0.96999999999999997,"5min":0.96999999999999997,"15min":0.96999999999999997},"last":0.55700000000000005}]},{"osd":3,"last update":"Fri Jul 14 19:15:51 2023","interfaces":[{"interface":"back","average":{"1min":0.61699999999999999,"5min":0.56599999999999995,"15min":0.55800000000000005},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":0.89200000000000002,"5min":0.89200000000000002,"15min":0.89200000000000002},"last":0.70499999999999996},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.80300000000000005,"5min":0.91700000000000004,"15min":0.91700000000000004},"last":0.66400000000000003}]},{"osd":4,"last update":"Fri Jul 14 19:15:51 2023","interfaces":[{"interface":"back","average":{"1min":0.53800000000000003,"5min":0.54000000000000004,"15min":0.54000000000000004},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.81499999999999995,"5min":0.86399999999999999,"15min":0.86399999999999999},"last":0.48499999999999999},{"interface":"front","average":{"1min":0.497,"5min":0.48099999999999998,"15min":0.47899999999999998},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":0.80600000000000005,"5min":0.80600000000000005,"15min":0.80600000000000005},"last":0.45900000000000002}]},{"osd":5,"last update":"Fri Jul 14 19:15:51 2023","interfaces":[{"interface":"back","average":{"1min":0.65100000000000002,"5min":0.67800000000000005,"15min":0.68200000000000005},"min":{"1min":0.43099999999999999,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":0.90600000000000003,"5min":1.355,"15min":1.355},"last":0.53300000000000003},{"interface":"front","average":{"1min":0.73599999999999999,"5min":0.71499999999999997,"15min":0.71099999999999997},"min":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":1.048,"5min":1.3009999999999999,"15min":1.3009999999999999},"last":0.63800000000000001}]},{"osd":6,"last update":"Fri Jul 14 19:15:51 2023","interfaces":[{"interface":"back","average":{"1min":0.69599999999999995,"5min":0.69399999999999995,"15min":0.69399999999999995},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":0.93200000000000005,"5min":1.2310000000000001,"15min":1.2310000000000001},"last":0.74099999999999999},{"interface":"front","average":{"1min":0.70399999999999996,"5min":0.66800000000000004,"15min":0.66200000000000003},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":1.069,"5min":1.1140000000000001,"15min":1.1140000000000001},"last":0.85399999999999998}]},{"osd":8,"last update":"Fri Jul 14 19:16:14 2023","interfaces":[{"interface":"back","average":{"1min":0.71699999999999997,"5min":0.73799999999999999,"15min":0.74199999999999999},"min":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":1.1319999999999999,"5min":1.1319999999999999,"15min":1.1319999999999999},"last":0.753},{"interface":"front","average":{"1min":0.72899999999999998,"5min":0.72899999999999998,"15min":0.72899999999999998},"min":{"1min":0.54700000000000004,"5min":0.495,"15min":0.495},"max":{"1min":0.99199999999999999,"5min":0.99199999999999999,"15min":0.99199999999999999},"last":0.69099999999999995}]},{"osd":9,"last update":"Fri Jul 14 19:16:38 2023","interfaces":[{"interface":"back","average":{"1min":0.61199999999999999,"5min":0.72499999999999998,"15min":0.74399999999999999},"min":{"1min":0.48899999999999999,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":0.79800000000000004,"5min":1.1599999999999999,"15min":1.1599999999999999},"last":0.81799999999999995},{"interface":"front","average":{"1min":0.69299999999999995,"5min":0.76100000000000001,"15min":0.77200000000000002},"min":{"1min":0.499,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":0.90500000000000003,"5min":1.214,"15min":1.214},"last":0.83699999999999997}]},{"osd":10,"last update":"Fri Jul 14 19:15:55 2023","interfaces":[{"interface":"back","average":{"1min":0.753,"5min":0.753,"15min":0.753},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":1.0960000000000001,"5min":1.0960000000000001,"15min":1.0960000000000001},"last":0.72599999999999998},{"interface":"front","average":{"1min":0.748,"5min":0.748,"15min":0.748},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":1.0980000000000001,"5min":1.0980000000000001,"15min":1.0980000000000001},"last":0.80300000000000005}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.78500000000000003}]}]},{"osd":6,"up_from":45,"seq":193273528363,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27884,"kb_used_data":868,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717268,"statfs":{"total":95995035648,"available":95966482432,"internally_reserved":0,"allocated":888832,"data_stored":416717,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jul 14 19:16:34 2023","interfaces":[{"interface":"back","average":{"1min":0.39100000000000001,"5min":0.41699999999999998,"15min":0.40999999999999998},"min":{"1min":0.29499999999999998,"5min":0.222,"15min":0.222},"max":{"1min":0.57899999999999996,"5min":0.77100000000000002,"15min":0.77100000000000002},"last":0.29999999999999999},{"interface":"front","average":{"1min":0.40699999999999997,"5min":0.45100000000000001,"15min":0.443},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.68200000000000005,"5min":1.1459999999999999,"15min":1.1459999999999999},"last":0.28100000000000003}]},{"osd":1,"last update":"Fri Jul 14 19:16:34 2023","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.57899999999999996,"15min":0.56399999999999995},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":0.77900000000000003,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.502},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.67000000000000004,"15min":0.69399999999999995},"min":{"1min":0.47099999999999997,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":0.85199999999999998,"5min":3.1930000000000001,"15min":3.1930000000000001},"last":0.57399999999999995}]},{"osd":2,"last update":"Fri Jul 14 19:16:34 2023","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":0.59499999999999997,"15min":0.58699999999999997},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.81100000000000005,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.48399999999999999},{"interface":"front","average":{"1min":0.58199999999999996,"5min":0.58499999999999996,"15min":0.57699999999999996},"min":{"1min":0.40799999999999997,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.79400000000000004,"5min":0.85299999999999998,"15min":0.85299999999999998},"last":0.54600000000000004}]},{"osd":3,"last update":"Fri Jul 14 19:16:34 2023","interfaces":[{"interface":"back","average":{"1min":0.47799999999999998,"5min":0.501,"15min":0.497},"min":{"1min":0.317,"5min":0.314,"15min":0.314},"max":{"1min":0.72699999999999998,"5min":0.82799999999999996,"15min":0.82799999999999996},"last":0.32700000000000001},{"interface":"front","average":{"1min":0.51500000000000001,"5min":0.54200000000000004,"15min":0.54000000000000004},"min":{"1min":0.26700000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.89100000000000001,"5min":0.96199999999999997,"15min":0.96199999999999997},"last":0.35399999999999998}]},{"osd":4,"last update":"Fri Jul 14 19:16:34 2023","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.63500000000000001,"15min":0.625},"min":{"1min":0.42599999999999999,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.84399999999999997,"5min":0.91500000000000004,"15min":0.91500000000000004},"last":0.53800000000000003},{"interface":"front","average":{"1min":0.64500000000000002,"5min":0.66900000000000004,"15min":0.65700000000000003},"min":{"1min":0.48799999999999999,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":0.91000000000000003,"5min":1.0309999999999999,"15min":1.0309999999999999},"last":0.63600000000000001}]},{"osd":5,"last update":"Fri Jul 14 19:16:34 2023","interfaces":[{"interface":"back","average":{"1min":0.64600000000000002,"5min":0.73399999999999999,"15min":0.753},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.82899999999999996,"5min":3.1299999999999999,"15min":3.1299999999999999},"last":0.44900000000000001},{"interface":"front","average":{"1min":0.68400000000000005,"5min":0.69099999999999995,"15min":0.67200000000000004},"min":{"1min":0.41299999999999998,"5min":0.38,"15min":0.38},"max":{"1min":0.92500000000000004,"5min":1.105,"15min":1.105},"last":0.51100000000000001}]},{"osd":7,"last update":"Fri Jul 14 19:15:56 2023","interfaces":[{"interface":"back","average":{"1min":0.79000000000000004,"5min":0.84199999999999997,"15min":0.85099999999999998},"min":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"max":{"1min":1.0489999999999999,"5min":3.25,"15min":3.25},"last":0.55700000000000005},{"interface":"front","average":{"1min":0.79600000000000004,"5min":0.749,"15min":0.74099999999999999},"min":{"1min":0.58299999999999996,"5min":0.54000000000000004,"15min":0.54000000000000004},"max":{"1min":1.0589999999999999,"5min":1.0589999999999999,"15min":1.0589999999999999},"last":0.625}]},{"osd":8,"last update":"Fri Jul 14 19:16:15 2023","interfaces":[{"interface":"back","average":{"1min":0.70199999999999996,"5min":0.73299999999999998,"15min":0.73799999999999999},"min":{"1min":0.437,"5min":0.437,"15min":0.437},"max":{"1min":0.96199999999999997,"5min":1.044,"15min":1.044},"last":0.47399999999999998},{"interface":"front","average":{"1min":0.75600000000000001,"5min":0.91100000000000003,"15min":0.93700000000000006},"min":{"1min":0.60199999999999998,"5min":0.57999999999999996,"15min":0.57999999999999996},"max":{"1min":1.006,"5min":3.226,"15min":3.226},"last":0.52400000000000002}]},{"osd":9,"last update":"Fri Jul 14 19:16:41 2023","interfaces":[{"interface":"back","average":{"1min":0.61299999999999999,"5min":0.80300000000000005,"15min":0.83499999999999996},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.88500000000000001,"5min":1.123,"15min":1.123},"last":0.436},{"interface":"front","average":{"1min":0.64600000000000002,"5min":0.79700000000000004,"15min":0.82199999999999995},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.84499999999999997,"5min":1.135,"15min":1.135},"last":0.41699999999999998}]},{"osd":10,"last update":"Fri Jul 14 19:15:56 2023","interfaces":[{"interface":"back","average":{"1min":0.85899999999999999,"5min":0.85899999999999999,"15min":0.85899999999999999},"min":{"1min":0.63,"5min":0.63,"15min":0.63},"max":{"1min":1.0840000000000001,"5min":1.0840000000000001,"15min":1.0840000000000001},"last":0.71499999999999997},{"interface":"front","average":{"1min":0.85699999999999998,"5min":0.85699999999999998,"15min":0.85699999999999998},"min":{"1min":0.64100000000000001,"5min":0.64100000000000001,"15min":0.64100000000000001},"max":{"1min":1.0820000000000001,"5min":1.0820000000000001,"15min":1.0820000000000001},"last":0.69099999999999995}]}]},{"osd":1,"up_from":13,"seq":55834574912,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27936,"kb_used_data":920,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717216,"statfs":{"total":95995035648,"available":95966429184,"internally_reserved":0,"allocated":942080,"data_stored":474297,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jul 14 19:15:49 2023","interfaces":[{"interface":"back","average":{"1min":0.60199999999999998,"5min":0.51100000000000001,"15min":0.45400000000000001},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.81999999999999995,"5min":1.149,"15min":1.149},"last":0.45700000000000002},{"interface":"front","average":{"1min":0.63500000000000001,"5min":0.52500000000000002,"15min":0.46999999999999997},"min":{"1min":0.36099999999999999,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.97099999999999997,"5min":1.0740000000000001,"15min":1.0740000000000001},"last":0.42699999999999999}]},{"osd":2,"last update":"Fri Jul 14 19:16:09 2023","interfaces":[{"interface":"back","average":{"1min":0.61799999999999999,"5min":0.53400000000000003,"15min":0.51900000000000002},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.94299999999999995,"5min":0.94299999999999995,"15min":0.94299999999999995},"last":0.72799999999999998},{"interface":"front","average":{"1min":0.64000000000000001,"5min":0.54500000000000004,"15min":0.51400000000000001},"min":{"1min":0.45500000000000002,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.83799999999999997,"5min":0.83799999999999997,"15min":0.83799999999999997},"last":0.71099999999999997}]},{"osd":3,"last update":"Fri Jul 14 19:16:30 2023","interfaces":[{"interface":"back","average":{"1min":0.64900000000000002,"5min":0.59999999999999998,"15min":0.57099999999999995},"min":{"1min":0.41699999999999998,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.95199999999999996,"5min":1.1659999999999999,"15min":1.1659999999999999},"last":0.629},{"interface":"front","average":{"1min":0.62,"5min":0.59099999999999997,"15min":0.56799999999999995},"min":{"1min":0.46700000000000003,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.90000000000000002,"5min":1.177,"15min":1.177},"last":0.48399999999999999}]},{"osd":4,"last update":"Fri Jul 14 19:15:49 2023","interfaces":[{"interface":"back","average":{"1min":0.52800000000000002,"5min":0.435,"15min":0.42299999999999999},"min":{"1min":0.32100000000000001,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.77300000000000002,"5min":0.77300000000000002,"15min":0.77300000000000002},"last":0.39600000000000002},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.47099999999999997,"15min":0.45500000000000002},"min":{"1min":0.38500000000000001,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.80300000000000005,"5min":0.80300000000000005,"15min":0.80300000000000005},"last":0.56799999999999995}]},{"osd":5,"last update":"Fri Jul 14 19:16:09 2023","interfaces":[{"interface":"back","average":{"1min":0.56399999999999995,"5min":0.53500000000000003,"15min":0.52600000000000002},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.85099999999999998,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.69499999999999995},{"interface":"front","average":{"1min":0.68200000000000005,"5min":0.60399999999999998,"15min":0.59199999999999997},"min":{"1min":0.47499999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.89900000000000002,"5min":0.89900000000000002,"15min":0.89900000000000002},"last":0.65000000000000002}]},{"osd":6,"last update":"Fri Jul 14 19:16:37 2023","interfaces":[{"interface":"back","average":{"1min":0.85899999999999999,"5min":0.68400000000000005,"15min":0.65200000000000002},"min":{"1min":0.54000000000000004,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":2.7890000000000001,"5min":2.7890000000000001,"15min":2.7890000000000001},"last":0.68100000000000005},{"interface":"front","average":{"1min":0.78700000000000003,"5min":0.65800000000000003,"15min":0.63200000000000001},"min":{"1min":0.52200000000000002,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":2.6899999999999999,"5min":2.6899999999999999,"15min":2.6899999999999999},"last":0.54600000000000004}]},{"osd":7,"last update":"Fri Jul 14 19:15:56 2023","interfaces":[{"interface":"back","average":{"1min":0.65600000000000003,"5min":0.55000000000000004,"15min":0.53200000000000003},"min":{"1min":0.378,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.86799999999999999,"5min":0.86799999999999999,"15min":0.86799999999999999},"last":0.59799999999999998},{"interface":"front","average":{"1min":0.68799999999999994,"5min":0.61199999999999999,"15min":0.59899999999999998},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.93300000000000005,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.77300000000000002}]},{"osd":8,"last update":"Fri Jul 14 19:16:20 2023","interfaces":[{"interface":"back","average":{"1min":0.75800000000000001,"5min":0.67800000000000005,"15min":0.66400000000000003},"min":{"1min":0.502,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":1.036,"5min":1.036,"15min":1.036},"last":0.80800000000000005},{"interface":"front","average":{"1min":0.68000000000000005,"5min":0.64400000000000002,"15min":0.63800000000000001},"min":{"1min":0.495,"5min":0.47999999999999998,"15min":0.47999999999999998},"max":{"1min":0.95499999999999996,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.66800000000000004}]},{"osd":9,"last update":"Fri Jul 14 19:16:37 2023","interfaces":[{"interface":"back","average":{"1min":0.80400000000000005,"5min":0.74299999999999999,"15min":0.73299999999999998},"min":{"1min":0.58299999999999996,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":1.022,"5min":1.022,"15min":1.022},"last":0.79400000000000004},{"interface":"front","average":{"1min":0.86899999999999999,"5min":0.77100000000000002,"15min":0.755},"min":{"1min":0.66000000000000003,"5min":0.53900000000000003,"15min":0.53900000000000003},"max":{"1min":1.0740000000000001,"5min":1.0740000000000001,"15min":1.0740000000000001},"last":0.85899999999999999}]},{"osd":10,"last update":"Fri Jul 14 19:15:56 2023","interfaces":[{"interface":"back","average":{"1min":0.65800000000000003,"5min":0.65800000000000003,"15min":0.65800000000000003},"min":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":0.86799999999999999,"5min":0.86799999999999999,"15min":0.86799999999999999},"last":0.61899999999999999},{"interface":"front","average":{"1min":0.72599999999999998,"5min":0.72599999999999998,"15min":0.72599999999999998},"min":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"max":{"1min":0.88300000000000001,"5min":0.88300000000000001,"15min":0.88300000000000001},"last":0.753}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.745}]}]},{"osd":0,"up_from":8,"seq":34359738437,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27904,"kb_used_data":888,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717248,"statfs":{"total":95995035648,"available":95966461952,"internally_reserved":0,"allocated":909312,"data_stored":418050,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Jul 14 19:15:50 2023","interfaces":[{"interface":"back","average":{"1min":0.627,"5min":0.55900000000000005,"15min":0.52900000000000003},"min":{"1min":0.49099999999999999,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.88,"5min":0.95799999999999996,"15min":0.95799999999999996},"last":0.49299999999999999},{"interface":"front","average":{"1min":0.64800000000000002,"5min":0.56399999999999995,"15min":0.54200000000000004},"min":{"1min":0.40600000000000003,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.91700000000000004,"5min":0.99099999999999999,"15min":0.99099999999999999},"last":0.505}]},{"osd":2,"last update":"Fri Jul 14 19:16:08 2023","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.54100000000000004,"15min":0.51200000000000001},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.94999999999999996,"5min":0.94999999999999996,"15min":0.94999999999999996},"last":0.57099999999999995},{"interface":"front","average":{"1min":0.61899999999999999,"5min":0.56200000000000006,"15min":0.52500000000000002},"min":{"1min":0.37,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.91000000000000003,"5min":1.044,"15min":1.044},"last":0.69799999999999995}]},{"osd":3,"last update":"Fri Jul 14 19:14:30 2023","interfaces":[{"interface":"back","average":{"1min":0.437,"5min":0.40100000000000002,"15min":0.39500000000000002},"min":{"1min":0.315,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.623,"5min":0.623,"15min":0.623},"last":0.36799999999999999},{"interface":"front","average":{"1min":0.497,"5min":0.47299999999999998,"15min":0.46899999999999997},"min":{"1min":0.34100000000000003,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.59999999999999998,"5min":0.63200000000000001,"15min":0.63200000000000001},"last":0.39600000000000002}]},{"osd":4,"last update":"Fri Jul 14 19:15:50 2023","interfaces":[{"interface":"back","average":{"1min":0.65900000000000003,"5min":0.57899999999999996,"15min":0.55100000000000005},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":0.92900000000000005,"5min":0.95399999999999996,"15min":0.95399999999999996},"last":0.432},{"interface":"front","average":{"1min":0.63200000000000001,"5min":0.59199999999999997,"15min":0.56999999999999995},"min":{"1min":0.44,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.86099999999999999,"5min":0.93400000000000005,"15min":0.93400000000000005},"last":0.54000000000000004}]},{"osd":5,"last update":"Fri Jul 14 19:16:16 2023","interfaces":[{"interface":"back","average":{"1min":0.63400000000000001,"5min":0.63600000000000001,"15min":0.62},"min":{"1min":0.443,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":0.96899999999999997,"5min":1.0169999999999999,"15min":1.0169999999999999},"last":0.66400000000000003},{"interface":"front","average":{"1min":0.59999999999999998,"5min":0.61099999999999999,"15min":0.58899999999999997},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.94899999999999995,"5min":1.073,"15min":1.073},"last":0.60599999999999998}]},{"osd":6,"last update":"Fri Jul 14 19:16:36 2023","interfaces":[{"interface":"back","average":{"1min":0.56000000000000005,"5min":0.53800000000000003,"15min":0.53300000000000003},"min":{"1min":0.34100000000000003,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":1.163,"5min":1.163,"15min":1.163},"last":0.47599999999999998},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.62,"15min":0.60199999999999998},"min":{"1min":0.38700000000000001,"5min":0.312,"15min":0.312},"max":{"1min":0.86599999999999999,"5min":1.105,"15min":1.105},"last":0.46200000000000002}]},{"osd":7,"last update":"Fri Jul 14 19:15:55 2023","interfaces":[{"interface":"back","average":{"1min":0.77400000000000002,"5min":0.72499999999999998,"15min":0.71699999999999997},"min":{"1min":0.55700000000000005,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":1.0629999999999999,"5min":1.0629999999999999,"15min":1.0629999999999999},"last":0.81299999999999994},{"interface":"front","average":{"1min":0.68700000000000006,"5min":0.69999999999999996,"15min":0.70199999999999996},"min":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":0.99299999999999999,"5min":1.0229999999999999,"15min":1.0229999999999999},"last":0.77200000000000002}]},{"osd":8,"last update":"Fri Jul 14 19:16:16 2023","interfaces":[{"interface":"back","average":{"1min":0.65300000000000002,"5min":0.68500000000000005,"15min":0.68999999999999995},"min":{"1min":0.42299999999999999,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":1.04,"5min":1.04,"15min":1.04},"last":0.65200000000000002},{"interface":"front","average":{"1min":0.68400000000000005,"5min":0.81299999999999994,"15min":0.83499999999999996},"min":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":1.0900000000000001,"5min":1.097,"15min":1.097},"last":0.55700000000000005}]},{"osd":9,"last update":"Fri Jul 14 19:16:42 2023","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.70499999999999996,"15min":0.72199999999999998},"min":{"1min":0.40600000000000003,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.92000000000000004,"5min":1.111,"15min":1.111},"last":0.64200000000000002},{"interface":"front","average":{"1min":0.66800000000000004,"5min":0.69599999999999995,"15min":0.70099999999999996},"min":{"1min":0.40200000000000002,"5min":0.377,"15min":0.377},"max":{"1min":1.206,"5min":1.206,"15min":1.206},"last":0.63100000000000001}]},{"osd":10,"last update":"Fri Jul 14 19:15:59 2023","interfaces":[{"interface":"back","average":{"1min":0.82799999999999996,"5min":0.82799999999999996,"15min":0.82799999999999996},"min":{"1min":0.66200000000000003,"5min":0.66200000000000003,"15min":0.66200000000000003},"max":{"1min":1.1379999999999999,"5min":1.1379999999999999,"15min":1.1379999999999999},"last":0.52700000000000002},{"interface":"front","average":{"1min":0.77800000000000002,"5min":0.77800000000000002,"15min":0.77800000000000002},"min":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"max":{"1min":0.99199999999999999,"5min":0.99199999999999999,"15min":0.99199999999999999},"last":0.67600000000000005}]},{"osd":11,"last update":"Fri Jul 14 19:16:16 2023","interfaces":[{"interface":"back","average":{"1min":0.72299999999999998,"5min":0.72299999999999998,"15min":0.72299999999999998},"min":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"max":{"1min":1.0669999999999999,"5min":1.0669999999999999,"15min":1.0669999999999999},"last":0.68700000000000006},{"interface":"front","average":{"1min":0.77300000000000002,"5min":0.77300000000000002,"15min":0.77300000000000002},"min":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"max":{"1min":1.111,"5min":1.111,"15min":1.111},"last":0.61699999999999999}]}]},{"osd":2,"up_from":18,"seq":77309411389,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27960,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717192,"statfs":{"total":95995035648,"available":95966404608,"internally_reserved":0,"allocated":966656,"data_stored":474742,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jul 14 19:16:11 2023","interfaces":[{"interface":"back","average":{"1min":0.47999999999999998,"5min":0.48899999999999999,"15min":0.47899999999999998},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.67300000000000004,"5min":0.76600000000000001,"15min":0.76600000000000001},"last":0.39700000000000002},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.51800000000000002,"15min":0.497},"min":{"1min":0.38600000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.72399999999999998,"5min":0.79500000000000004,"15min":0.79500000000000004},"last":0.78000000000000003}]},{"osd":1,"last update":"Fri Jul 14 19:16:11 2023","interfaces":[{"interface":"back","average":{"1min":0.60399999999999998,"5min":0.53100000000000003,"15min":0.49399999999999999},"min":{"1min":0.44600000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.77000000000000002,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.52800000000000002},{"interface":"front","average":{"1min":0.59699999999999998,"5min":0.53700000000000003,"15min":0.505},"min":{"1min":0.38,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":0.81399999999999995,"5min":0.90700000000000003,"15min":0.90700000000000003},"last":0.76700000000000002}]},{"osd":3,"last update":"Fri Jul 14 19:16:30 2023","interfaces":[{"interface":"back","average":{"1min":1.046,"5min":0.64000000000000001,"15min":0.54900000000000004},"min":{"1min":0.42799999999999999,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":7.7350000000000003,"5min":7.7350000000000003,"15min":7.7350000000000003},"last":0.48499999999999999},{"interface":"front","average":{"1min":1.0700000000000001,"5min":0.67200000000000004,"15min":0.58399999999999996},"min":{"1min":0.44500000000000001,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":7.7190000000000003,"5min":7.7190000000000003,"15min":7.7190000000000003},"last":0.59199999999999997}]},{"osd":4,"last update":"Fri Jul 14 19:15:53 2023","interfaces":[{"interface":"back","average":{"1min":0.57899999999999996,"5min":0.53400000000000003,"15min":0.51500000000000001},"min":{"1min":0.45200000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.78700000000000003,"5min":0.78700000000000003,"15min":0.78700000000000003},"last":0.67100000000000004},{"interface":"front","average":{"1min":0.60599999999999998,"5min":0.54700000000000004,"15min":0.52700000000000002},"min":{"1min":0.436,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.81299999999999994,"5min":0.81299999999999994,"15min":0.81299999999999994},"last":0.61199999999999999}]},{"osd":5,"last update":"Fri Jul 14 19:16:14 2023","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.42499999999999999,"15min":0.40799999999999997},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":0.746,"5min":0.746,"15min":0.746},"last":0.498},{"interface":"front","average":{"1min":0.51300000000000001,"5min":0.45100000000000001,"15min":0.42899999999999999},"min":{"1min":0.32100000000000001,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.83999999999999997,"5min":0.83999999999999997,"15min":0.83999999999999997},"last":0.46100000000000002}]},{"osd":6,"last update":"Fri Jul 14 19:16:37 2023","interfaces":[{"interface":"back","average":{"1min":0.66900000000000004,"5min":0.63800000000000001,"15min":0.62},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":1.6930000000000001,"5min":1.6930000000000001,"15min":1.6930000000000001},"last":0.65300000000000002},{"interface":"front","average":{"1min":0.74099999999999999,"5min":0.64700000000000002,"15min":0.61699999999999999},"min":{"1min":0.48199999999999998,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":1.6659999999999999,"5min":1.6659999999999999,"15min":1.6659999999999999},"last":0.71299999999999997}]},{"osd":7,"last update":"Fri Jul 14 19:15:53 2023","interfaces":[{"interface":"back","average":{"1min":0.64700000000000002,"5min":0.65100000000000002,"15min":0.65200000000000002},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.96299999999999997,"5min":0.96299999999999997,"15min":0.96299999999999997},"last":0.56999999999999995},{"interface":"front","average":{"1min":0.67100000000000004,"5min":0.66700000000000004,"15min":0.66700000000000004},"min":{"1min":0.48799999999999999,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.94099999999999995,"5min":0.94099999999999995,"15min":0.94099999999999995},"last":0.628}]},{"osd":8,"last update":"Fri Jul 14 19:16:14 2023","interfaces":[{"interface":"back","average":{"1min":0.63400000000000001,"5min":0.59299999999999997,"15min":0.58599999999999997},"min":{"1min":0.41599999999999998,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.98799999999999999,"5min":0.98799999999999999,"15min":0.98799999999999999},"last":0.629},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.60499999999999998,"15min":0.59699999999999998},"min":{"1min":0.436,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":1.0069999999999999,"5min":1.0069999999999999,"15min":1.0069999999999999},"last":0.69399999999999995}]},{"osd":9,"last update":"Fri Jul 14 19:16:42 2023","interfaces":[{"interface":"back","average":{"1min":0.78100000000000003,"5min":0.74199999999999999,"15min":0.73599999999999999},"min":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"max":{"1min":1.6459999999999999,"5min":1.6459999999999999,"15min":1.6459999999999999},"last":0.72899999999999998},{"interface":"front","average":{"1min":1.093,"5min":0.80100000000000005,"15min":0.752},"min":{"1min":0.51600000000000001,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":5.9029999999999996,"5min":5.9029999999999996,"15min":5.9029999999999996},"last":0.79300000000000004}]},{"osd":10,"last update":"Fri Jul 14 19:15:56 2023","interfaces":[{"interface":"back","average":{"1min":0.63900000000000001,"5min":0.63900000000000001,"15min":0.63900000000000001},"min":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":0.88500000000000001,"5min":0.88500000000000001,"15min":0.88500000000000001},"last":0.74099999999999999},{"interface":"front","average":{"1min":0.71799999999999997,"5min":0.71799999999999997,"15min":0.71799999999999997},"min":{"1min":0.56200000000000006,"5min":0.56200000000000006,"15min":0.56200000000000006},"max":{"1min":1.073,"5min":1.073,"15min":1.073},"last":0.753}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82099999999999995}]}]},{"osd":3,"up_from":25,"seq":107374182456,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27896,"kb_used_data":880,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717256,"statfs":{"total":95995035648,"available":95966470144,"internally_reserved":0,"allocated":901120,"data_stored":416265,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jul 14 19:16:29 2023","interfaces":[{"interface":"back","average":{"1min":0.40699999999999997,"5min":0.374,"15min":0.35499999999999998},"min":{"1min":0.28299999999999997,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":0.98399999999999999,"5min":0.98399999999999999,"15min":0.98399999999999999},"last":0.45600000000000002},{"interface":"front","average":{"1min":0.41399999999999998,"5min":0.40000000000000002,"15min":0.38100000000000001},"min":{"1min":0.312,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.84599999999999997,"5min":0.85899999999999999,"15min":0.85899999999999999},"last":0.434}]},{"osd":1,"last update":"Fri Jul 14 19:16:29 2023","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.57199999999999995,"15min":0.53500000000000003},"min":{"1min":0.44800000000000001,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.79000000000000004,"5min":0.94699999999999995,"15min":0.94699999999999995},"last":0.56799999999999995},{"interface":"front","average":{"1min":0.624,"5min":0.58599999999999997,"15min":0.55700000000000005},"min":{"1min":0.41599999999999998,"5min":0.372,"15min":0.372},"max":{"1min":1.0800000000000001,"5min":1.0800000000000001,"15min":1.0800000000000001},"last":0.54800000000000004}]},{"osd":2,"last update":"Fri Jul 14 19:16:29 2023","interfaces":[{"interface":"back","average":{"1min":0.60299999999999998,"5min":0.57999999999999996,"15min":0.56100000000000005},"min":{"1min":0.45800000000000002,"5min":0.372,"15min":0.372},"max":{"1min":0.84899999999999998,"5min":0.871,"15min":0.871},"last":0.71199999999999997},{"interface":"front","average":{"1min":0.57399999999999995,"5min":0.57899999999999996,"15min":0.55700000000000005},"min":{"1min":0.44400000000000001,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.80000000000000004,"5min":0.93700000000000006,"15min":0.93700000000000006},"last":0.61799999999999999}]},{"osd":4,"last update":"Fri Jul 14 19:15:56 2023","interfaces":[{"interface":"back","average":{"1min":0.64500000000000002,"5min":0.58399999999999996,"15min":0.56200000000000006},"min":{"1min":0.47499999999999998,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.98599999999999999,"5min":0.98599999999999999,"15min":0.98599999999999999},"last":0.60299999999999998},{"interface":"front","average":{"1min":0.66800000000000004,"5min":0.60199999999999998,"15min":0.58999999999999997},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":0.86799999999999999,"5min":0.86799999999999999,"15min":0.86799999999999999},"last":0.64500000000000002}]},{"osd":5,"last update":"Fri Jul 14 19:16:15 2023","interfaces":[{"interface":"back","average":{"1min":0.60299999999999998,"5min":0.59499999999999997,"15min":0.58099999999999996},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":0.91300000000000003,"5min":0.91300000000000003,"15min":0.91300000000000003},"last":0.627},{"interface":"front","average":{"1min":0.621,"5min":0.61599999999999999,"15min":0.60999999999999999},"min":{"1min":0.44700000000000001,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.81999999999999995,"5min":0.91700000000000004,"15min":0.91700000000000004},"last":0.77400000000000002}]},{"osd":6,"last update":"Fri Jul 14 19:16:37 2023","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.51000000000000001,"15min":0.50600000000000001},"min":{"1min":0.38100000000000001,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.82399999999999995,"5min":0.86399999999999999,"15min":0.86399999999999999},"last":0.48599999999999999},{"interface":"front","average":{"1min":0.55300000000000005,"5min":0.56899999999999995,"15min":0.55100000000000005},"min":{"1min":0.41099999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.877,"5min":0.99199999999999999,"15min":0.99199999999999999},"last":0.51900000000000002}]},{"osd":7,"last update":"Fri Jul 14 19:15:56 2023","interfaces":[{"interface":"back","average":{"1min":0.73599999999999999,"5min":0.71699999999999997,"15min":0.71399999999999997},"min":{"1min":0.47599999999999998,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":1.05,"5min":1.1080000000000001,"15min":1.1080000000000001},"last":0.70199999999999996},{"interface":"front","average":{"1min":0.70299999999999996,"5min":0.67600000000000005,"15min":0.67200000000000004},"min":{"1min":0.46500000000000002,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":0.89400000000000002,"5min":0.95599999999999996,"15min":0.95599999999999996},"last":0.66000000000000003}]},{"osd":8,"last update":"Fri Jul 14 19:15:15 2023","interfaces":[{"interface":"back","average":{"1min":0.747,"5min":0.747,"15min":0.747},"min":{"1min":0.55800000000000005,"5min":0.55800000000000005,"15min":0.55800000000000005},"max":{"1min":1.0640000000000001,"5min":1.0640000000000001,"15min":1.0640000000000001},"last":0.66700000000000004},{"interface":"front","average":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"min":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":0.98499999999999999,"5min":0.98499999999999999,"15min":0.98499999999999999},"last":0.68500000000000005}]},{"osd":9,"last update":"Fri Jul 14 19:16:37 2023","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.67500000000000004,"15min":0.68999999999999995},"min":{"1min":0.41799999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.78500000000000003,"5min":1.048,"15min":1.048},"last":0.59099999999999997},{"interface":"front","average":{"1min":0.65400000000000003,"5min":0.749,"15min":0.76500000000000001},"min":{"1min":0.46600000000000003,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":0.95399999999999996,"5min":1.0289999999999999,"15min":1.0289999999999999},"last":0.67200000000000004}]},{"osd":10,"last update":"Fri Jul 14 19:15:56 2023","interfaces":[{"interface":"back","average":{"1min":0.81299999999999994,"5min":0.81299999999999994,"15min":0.81299999999999994},"min":{"1min":0.58199999999999996,"5min":0.58199999999999996,"15min":0.58199999999999996},"max":{"1min":1.0469999999999999,"5min":1.0469999999999999,"15min":1.0469999999999999},"last":0.72299999999999998},{"interface":"front","average":{"1min":0.77200000000000002,"5min":0.77200000000000002,"15min":0.77200000000000002},"min":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":1.0069999999999999,"5min":1.0069999999999999,"15min":1.0069999999999999},"last":0.745}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79500000000000004}]}]},{"osd":4,"up_from":32,"seq":137438953524,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27884,"kb_used_data":868,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717268,"statfs":{"total":95995035648,"available":95966482432,"internally_reserved":0,"allocated":888832,"data_stored":416572,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jul 14 19:14:53 2023","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.503,"15min":0.495},"min":{"1min":0.27200000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.81999999999999995,"5min":0.81999999999999995,"15min":0.81999999999999995},"last":0.56299999999999994},{"interface":"front","average":{"1min":0.58199999999999996,"5min":0.53000000000000003,"15min":0.52200000000000002},"min":{"1min":0.378,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.84599999999999997,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.58299999999999996}]},{"osd":1,"last update":"Fri Jul 14 19:15:56 2023","interfaces":[{"interface":"back","average":{"1min":0.46600000000000003,"5min":0.39000000000000001,"15min":0.36399999999999999},"min":{"1min":0.30499999999999999,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.78400000000000003,"5min":1.0049999999999999,"15min":1.0049999999999999},"last":0.42499999999999999},{"interface":"front","average":{"1min":0.46600000000000003,"5min":0.39700000000000002,"15min":0.36899999999999999},"min":{"1min":0.26900000000000002,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":0.71699999999999997,"5min":0.91500000000000004,"15min":0.91500000000000004},"last":0.39200000000000002}]},{"osd":2,"last update":"Fri Jul 14 19:15:56 2023","interfaces":[{"interface":"back","average":{"1min":0.55100000000000005,"5min":0.54600000000000004,"15min":0.53400000000000003},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.69099999999999995,"5min":0.98499999999999999,"15min":0.98499999999999999},"last":0.63500000000000001},{"interface":"front","average":{"1min":0.66000000000000003,"5min":0.59899999999999998,"15min":0.56899999999999995},"min":{"1min":0.5,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.88700000000000001,"5min":0.92900000000000005,"15min":0.92900000000000005},"last":0.754}]},{"osd":3,"last update":"Fri Jul 14 19:15:56 2023","interfaces":[{"interface":"back","average":{"1min":0.63700000000000001,"5min":0.55000000000000004,"15min":0.52500000000000002},"min":{"1min":0.41299999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.89300000000000002,"5min":0.89300000000000002,"15min":0.89300000000000002},"last":0.52200000000000002},{"interface":"front","average":{"1min":0.56100000000000005,"5min":0.55000000000000004,"15min":0.53800000000000003},"min":{"1min":0.36899999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.73499999999999999,"5min":0.94199999999999995,"15min":0.94199999999999995},"last":0.46100000000000002}]},{"osd":5,"last update":"Fri Jul 14 19:16:16 2023","interfaces":[{"interface":"back","average":{"1min":0.68300000000000005,"5min":0.60299999999999998,"15min":0.57499999999999996},"min":{"1min":0.50600000000000001,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":0.88200000000000001,"5min":0.88200000000000001,"15min":0.88200000000000001},"last":0.72099999999999997},{"interface":"front","average":{"1min":0.70999999999999996,"5min":0.60799999999999998,"15min":0.58099999999999996},"min":{"1min":0.52700000000000002,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.89700000000000002,"5min":0.89700000000000002,"15min":0.89700000000000002},"last":0.623}]},{"osd":6,"last update":"Fri Jul 14 19:16:34 2023","interfaces":[{"interface":"back","average":{"1min":0.73299999999999998,"5min":0.64800000000000002,"15min":0.63},"min":{"1min":0.49299999999999999,"5min":0.38,"15min":0.38},"max":{"1min":0.95799999999999996,"5min":0.95799999999999996,"15min":0.95799999999999996},"last":0.57699999999999996},{"interface":"front","average":{"1min":0.746,"5min":0.63800000000000001,"15min":0.60799999999999998},"min":{"1min":0.47999999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.91500000000000004,"5min":1.0049999999999999,"15min":1.0049999999999999},"last":0.65100000000000002}]},{"osd":7,"last update":"Fri Jul 14 19:15:56 2023","interfaces":[{"interface":"back","average":{"1min":0.68100000000000005,"5min":0.60499999999999998,"15min":0.59299999999999997},"min":{"1min":0.44500000000000001,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.97599999999999998,"5min":1.0529999999999999,"15min":1.0529999999999999},"last":0.59799999999999998},{"interface":"front","average":{"1min":0.70899999999999996,"5min":0.59599999999999997,"15min":0.57699999999999996},"min":{"1min":0.47599999999999998,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.93100000000000005,"5min":0.93100000000000005,"15min":0.93100000000000005},"last":0.67200000000000004}]},{"osd":8,"last update":"Fri Jul 14 19:16:16 2023","interfaces":[{"interface":"back","average":{"1min":0.64400000000000002,"5min":0.63600000000000001,"15min":0.63500000000000001},"min":{"1min":0.51000000000000001,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":0.83599999999999997,"5min":1.0169999999999999,"15min":1.0169999999999999},"last":0.68999999999999995},{"interface":"front","average":{"1min":0.79300000000000004,"5min":0.77000000000000002,"15min":0.76600000000000001},"min":{"1min":0.56299999999999994,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":0.98499999999999999,"5min":0.98499999999999999,"15min":0.98499999999999999},"last":0.78500000000000003}]},{"osd":9,"last update":"Fri Jul 14 19:16:38 2023","interfaces":[{"interface":"back","average":{"1min":0.83799999999999997,"5min":0.78600000000000003,"15min":0.77800000000000002},"min":{"1min":0.64200000000000002,"5min":0.57399999999999995,"15min":0.57399999999999995},"max":{"1min":1.0009999999999999,"5min":1.0009999999999999,"15min":1.0009999999999999},"last":0.74099999999999999},{"interface":"front","average":{"1min":0.871,"5min":0.78800000000000003,"15min":0.77400000000000002},"min":{"1min":0.63600000000000001,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":1.024,"5min":1.05,"15min":1.05},"last":0.81000000000000005}]},{"osd":10,"last update":"Fri Jul 14 19:15:56 2023","interfaces":[{"interface":"back","average":{"1min":0.82199999999999995,"5min":0.82199999999999995,"15min":0.82199999999999995},"min":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.56000000000000005},"max":{"1min":1.0669999999999999,"5min":1.0669999999999999,"15min":1.0669999999999999},"last":0.83399999999999996},{"interface":"front","average":{"1min":0.80900000000000005,"5min":0.80900000000000005,"15min":0.80900000000000005},"min":{"1min":0.54900000000000004,"5min":0.54900000000000004,"15min":0.54900000000000004},"max":{"1min":1.028,"5min":1.028,"15min":1.028},"last":0.70599999999999996}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.85899999999999999}]}]},{"osd":5,"up_from":37,"seq":158913790000,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27960,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717192,"statfs":{"total":95995035648,"available":95966404608,"internally_reserved":0,"allocated":966656,"data_stored":1009183,"data_compressed":35714,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jul 14 19:16:13 2023","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.52900000000000003,"15min":0.51500000000000001},"min":{"1min":0.28399999999999997,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.63700000000000001,"5min":1.028,"15min":1.028},"last":0.57099999999999995},{"interface":"front","average":{"1min":0.48399999999999999,"5min":0.54900000000000004,"15min":0.54600000000000004},"min":{"1min":0.33700000000000002,"5min":0.307,"15min":0.307},"max":{"1min":0.68600000000000005,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.77300000000000002}]},{"osd":1,"last update":"Fri Jul 14 19:16:13 2023","interfaces":[{"interface":"back","average":{"1min":0.57699999999999996,"5min":0.52900000000000003,"15min":0.503},"min":{"1min":0.38800000000000001,"5min":0.307,"15min":0.307},"max":{"1min":0.747,"5min":1.2410000000000001,"15min":1.2410000000000001},"last":0.52100000000000002},{"interface":"front","average":{"1min":0.56299999999999994,"5min":0.503,"15min":0.47899999999999998},"min":{"1min":0.36299999999999999,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.753,"5min":0.85799999999999998,"15min":0.85799999999999998},"last":0.58799999999999997}]},{"osd":2,"last update":"Fri Jul 14 19:16:13 2023","interfaces":[{"interface":"back","average":{"1min":0.41699999999999998,"5min":0.41399999999999998,"15min":0.40999999999999998},"min":{"1min":0.29999999999999999,"5min":0.246,"15min":0.246},"max":{"1min":0.66600000000000004,"5min":0.68400000000000005,"15min":0.68400000000000005},"last":0.33600000000000002},{"interface":"front","average":{"1min":0.39300000000000002,"5min":0.40799999999999997,"15min":0.40200000000000002},"min":{"1min":0.28399999999999997,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.52700000000000002,"5min":0.73999999999999999,"15min":0.73999999999999999},"last":0.35699999999999998}]},{"osd":3,"last update":"Fri Jul 14 19:16:13 2023","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":0.57999999999999996,"15min":0.57699999999999996},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.72899999999999998,"5min":0.98199999999999998,"15min":0.98199999999999998},"last":0.61199999999999999},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.59499999999999997,"15min":0.59099999999999997},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.71999999999999997,"5min":0.95899999999999996,"15min":0.95899999999999996},"last":0.68000000000000005}]},{"osd":4,"last update":"Fri Jul 14 19:16:13 2023","interfaces":[{"interface":"back","average":{"1min":0.55100000000000005,"5min":0.54800000000000004,"15min":0.53600000000000003},"min":{"1min":0.42899999999999999,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.71199999999999997,"5min":0.79400000000000004,"15min":0.79400000000000004},"last":0.54200000000000004},{"interface":"front","average":{"1min":0.61299999999999999,"5min":0.58799999999999997,"15min":0.57399999999999995},"min":{"1min":0.46100000000000002,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.77400000000000002,"5min":0.875,"15min":0.875},"last":0.69199999999999995}]},{"osd":6,"last update":"Fri Jul 14 19:16:37 2023","interfaces":[{"interface":"back","average":{"1min":0.65400000000000003,"5min":0.628,"15min":0.61599999999999999},"min":{"1min":0.5,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":1.492,"5min":1.492,"15min":1.492},"last":0.66100000000000003},{"interface":"front","average":{"1min":0.66100000000000003,"5min":0.64800000000000002,"15min":0.63200000000000001},"min":{"1min":0.41999999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":1.4199999999999999,"5min":1.4199999999999999,"15min":1.4199999999999999},"last":0.72199999999999998}]},{"osd":7,"last update":"Fri Jul 14 19:15:53 2023","interfaces":[{"interface":"back","average":{"1min":0.64500000000000002,"5min":0.66000000000000003,"15min":0.66200000000000003},"min":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":0.84599999999999997,"5min":0.91400000000000003,"15min":0.91400000000000003},"last":0.55800000000000005},{"interface":"front","average":{"1min":0.66300000000000003,"5min":0.68000000000000005,"15min":0.68300000000000005},"min":{"1min":0.45700000000000002,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":0.83499999999999996,"5min":1.1240000000000001,"15min":1.1240000000000001},"last":0.63600000000000001}]},{"osd":8,"last update":"Fri Jul 14 19:16:14 2023","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.57099999999999995,"15min":0.57699999999999996},"min":{"1min":0.36399999999999999,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.77600000000000002,"5min":0.85199999999999998,"15min":0.85199999999999998},"last":0.47599999999999998},{"interface":"front","average":{"1min":0.66300000000000003,"5min":0.66300000000000003,"15min":0.66300000000000003},"min":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":0.81999999999999995,"5min":0.94199999999999995,"15min":0.94199999999999995},"last":0.498}]},{"osd":9,"last update":"Fri Jul 14 19:16:37 2023","interfaces":[{"interface":"back","average":{"1min":0.67400000000000004,"5min":0.72299999999999998,"15min":0.73099999999999998},"min":{"1min":0.54800000000000004,"5min":0.53100000000000003,"15min":0.53100000000000003},"max":{"1min":0.79900000000000004,"5min":1.014,"15min":1.014},"last":0.81000000000000005},{"interface":"front","average":{"1min":0.79200000000000004,"5min":0.76600000000000001,"15min":0.76200000000000001},"min":{"1min":0.51200000000000001,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":1.9510000000000001,"5min":1.9510000000000001,"15min":1.9510000000000001},"last":0.79100000000000004}]},{"osd":10,"last update":"Fri Jul 14 19:15:59 2023","interfaces":[{"interface":"back","average":{"1min":0.70599999999999996,"5min":0.70599999999999996,"15min":0.70599999999999996},"min":{"1min":0.56200000000000006,"5min":0.56200000000000006,"15min":0.56200000000000006},"max":{"1min":0.89300000000000002,"5min":0.89300000000000002,"15min":0.89300000000000002},"last":0.73599999999999999},{"interface":"front","average":{"1min":0.70999999999999996,"5min":0.70999999999999996,"15min":0.70999999999999996},"min":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":0.89000000000000001,"5min":0.89000000000000001,"15min":0.89000000000000001},"last":0.755}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34079,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34079,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34079,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35714,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35714,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35714,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35714,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35714,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2130,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":58650,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":58822,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":2772,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":674,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":2998,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":110592,"data_stored":58634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":246,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":123,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":123,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":123,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":123,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-07-14T19:16:49.648 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2023-07-14T19:16:49.649 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2023-07-14T19:16:49.649 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2023-07-14T19:16:49.649 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell --fsid adc52f04-2279-11ee-9b33-001a4aab830c -- ceph health --format=json 2023-07-14T19:16:50.210 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/adc52f04-2279-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T19:16:51.031 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:16:50 smithi040 ceph-mon[96518]: pgmap v280: 97 pgs: 97 active+clean; 638 KiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2023-07-14T19:16:51.031 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:16:50 smithi040 ceph-mon[96518]: from='client.14916 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-07-14T19:16:51.222 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:16:50 smithi191 ceph-mon[98311]: pgmap v280: 97 pgs: 97 active+clean; 638 KiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2023-07-14T19:16:51.222 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:16:50 smithi191 ceph-mon[98311]: from='client.14916 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-07-14T19:16:51.242 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:16:50 smithi130 ceph-mon[97832]: pgmap v280: 97 pgs: 97 active+clean; 638 KiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2023-07-14T19:16:51.242 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:16:50 smithi130 ceph-mon[97832]: from='client.14916 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-07-14T19:16:51.563 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:51.560+0000 7fea213be700 1 -- 172.21.15.40:0/1535330345 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fea1c103ee0 msgr2=0x7fea1c10ba60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:16:51.563 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:51.560+0000 7fea213be700 1 --2- 172.21.15.40:0/1535330345 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fea1c103ee0 0x7fea1c10ba60 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fea140099c0 tx=0x7fea14009cd0 comp rx=0 tx=0).stop 2023-07-14T19:16:51.563 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:51.560+0000 7fea213be700 1 -- 172.21.15.40:0/1535330345 shutdown_connections 2023-07-14T19:16:51.564 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:51.560+0000 7fea213be700 1 --2- 172.21.15.40:0/1535330345 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fea1c103ee0 0x7fea1c10ba60 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:51.564 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:51.560+0000 7fea213be700 1 --2- 172.21.15.40:0/1535330345 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fea1c0fee40 0x7fea1c0ff210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:51.564 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:51.560+0000 7fea213be700 1 --2- 172.21.15.40:0/1535330345 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fea1c0ff7e0 0x7fea1c1037b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:51.564 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:51.560+0000 7fea213be700 1 -- 172.21.15.40:0/1535330345 >> 172.21.15.40:0/1535330345 conn(0x7fea1c075ca0 msgr2=0x7fea1c0760a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:16:51.564 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:51.560+0000 7fea213be700 1 -- 172.21.15.40:0/1535330345 shutdown_connections 2023-07-14T19:16:51.564 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:51.560+0000 7fea213be700 1 -- 172.21.15.40:0/1535330345 wait complete. 2023-07-14T19:16:51.565 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:51.561+0000 7fea213be700 1 Processor -- start 2023-07-14T19:16:51.565 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:51.561+0000 7fea213be700 1 -- start start 2023-07-14T19:16:51.565 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:51.561+0000 7fea213be700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fea1c0fee40 0x7fea1c196b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:16:51.565 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:51.561+0000 7fea213be700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fea1c0ff7e0 0x7fea1c197050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:16:51.565 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:51.561+0000 7fea213be700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fea1c103ee0 0x7fea1c1a1b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:16:51.565 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:51.561+0000 7fea213be700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea1c197960 con 0x7fea1c0ff7e0 2023-07-14T19:16:51.565 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:51.561+0000 7fea213be700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea1c1a20d0 con 0x7fea1c103ee0 2023-07-14T19:16:51.566 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:51.561+0000 7fea213be700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea1c1a2240 con 0x7fea1c0fee40 2023-07-14T19:16:51.566 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:51.561+0000 7fea1a7fc700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fea1c0ff7e0 0x7fea1c197050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:16:51.566 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:51.561+0000 7fea1a7fc700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fea1c0ff7e0 0x7fea1c197050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:33516/0 (socket says 172.21.15.40:33516) 2023-07-14T19:16:51.566 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:51.561+0000 7fea1a7fc700 1 -- 172.21.15.40:0/1021997328 learned_addr learned my addr 172.21.15.40:0/1021997328 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-07-14T19:16:51.566 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:51.561+0000 7fea1affd700 1 --2- 172.21.15.40:0/1021997328 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fea1c0fee40 0x7fea1c196b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:16:51.566 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:51.562+0000 7fea1b7fe700 1 --2- 172.21.15.40:0/1021997328 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fea1c103ee0 0x7fea1c1a1b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:16:51.566 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:51.562+0000 7fea1a7fc700 1 -- 172.21.15.40:0/1021997328 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fea1c0fee40 msgr2=0x7fea1c196b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:16:51.566 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:51.562+0000 7fea1a7fc700 1 --2- 172.21.15.40:0/1021997328 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fea1c0fee40 0x7fea1c196b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:51.567 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:51.562+0000 7fea1a7fc700 1 -- 172.21.15.40:0/1021997328 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fea1c103ee0 msgr2=0x7fea1c1a1b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:16:51.567 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:51.562+0000 7fea1a7fc700 1 --2- 172.21.15.40:0/1021997328 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fea1c103ee0 0x7fea1c1a1b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:51.567 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:51.562+0000 7fea1a7fc700 1 -- 172.21.15.40:0/1021997328 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea140095f0 con 0x7fea1c0ff7e0 2023-07-14T19:16:51.567 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:51.562+0000 7fea1b7fe700 1 --2- 172.21.15.40:0/1021997328 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fea1c103ee0 0x7fea1c1a1b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-14T19:16:51.567 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:51.562+0000 7fea1a7fc700 1 --2- 172.21.15.40:0/1021997328 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fea1c0ff7e0 0x7fea1c197050 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7fea1000c8f0 tx=0x7fea1000cc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:16:51.567 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:51.562+0000 7fea03fff700 1 -- 172.21.15.40:0/1021997328 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea10007940 con 0x7fea1c0ff7e0 2023-07-14T19:16:51.567 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:51.562+0000 7fea03fff700 1 -- 172.21.15.40:0/1021997328 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fea1000f450 con 0x7fea1c0ff7e0 2023-07-14T19:16:51.568 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:51.562+0000 7fea213be700 1 -- 172.21.15.40:0/1021997328 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fea1c1a2520 con 0x7fea1c0ff7e0 2023-07-14T19:16:51.568 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:51.562+0000 7fea03fff700 1 -- 172.21.15.40:0/1021997328 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea10018660 con 0x7fea1c0ff7e0 2023-07-14T19:16:51.568 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:51.562+0000 7fea213be700 1 -- 172.21.15.40:0/1021997328 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fea1c1a2a20 con 0x7fea1c0ff7e0 2023-07-14T19:16:51.569 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:51.563+0000 7fea213be700 1 -- 172.21.15.40:0/1021997328 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fea1c04e340 con 0x7fea1c0ff7e0 2023-07-14T19:16:51.569 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:51.564+0000 7fea03fff700 1 -- 172.21.15.40:0/1021997328 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7fea1000f5c0 con 0x7fea1c0ff7e0 2023-07-14T19:16:51.569 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:51.565+0000 7fea03fff700 1 --2- 172.21.15.40:0/1021997328 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7fea040a11e0 0x7fea040a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:16:51.569 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:51.565+0000 7fea1affd700 1 --2- 172.21.15.40:0/1021997328 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7fea040a11e0 0x7fea040a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:16:51.569 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:51.565+0000 7fea03fff700 1 -- 172.21.15.40:0/1021997328 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7979+0+0 (secure 0 0 0) 0x7fea100cc780 con 0x7fea1c0ff7e0 2023-07-14T19:16:51.569 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:51.565+0000 7fea1affd700 1 --2- 172.21.15.40:0/1021997328 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7fea040a11e0 0x7fea040a3690 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fea0c00f440 tx=0x7fea0c0079e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:16:51.570 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:51.569+0000 7fea03fff700 1 -- 172.21.15.40:0/1021997328 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fea10095f20 con 0x7fea1c0ff7e0 2023-07-14T19:16:51.815 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:51.814+0000 7fea213be700 1 -- 172.21.15.40:0/1021997328 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fea1c1a2db0 con 0x7fea1c0ff7e0 2023-07-14T19:16:51.815 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:51.814+0000 7fea03fff700 1 -- 172.21.15.40:0/1021997328 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+46 (secure 0 0 0) 0x7fea10099540 con 0x7fea1c0ff7e0 2023-07-14T19:16:51.815 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-07-14T19:16:51.816 INFO:teuthology.orchestra.run.smithi040.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2023-07-14T19:16:51.817 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:51.816+0000 7fea213be700 1 -- 172.21.15.40:0/1021997328 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7fea040a11e0 msgr2=0x7fea040a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:16:51.817 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:51.816+0000 7fea213be700 1 --2- 172.21.15.40:0/1021997328 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7fea040a11e0 0x7fea040a3690 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fea0c00f440 tx=0x7fea0c0079e0 comp rx=0 tx=0).stop 2023-07-14T19:16:51.817 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:51.816+0000 7fea213be700 1 -- 172.21.15.40:0/1021997328 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fea1c0ff7e0 msgr2=0x7fea1c197050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:16:51.818 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:51.816+0000 7fea213be700 1 --2- 172.21.15.40:0/1021997328 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fea1c0ff7e0 0x7fea1c197050 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7fea1000c8f0 tx=0x7fea1000cc00 comp rx=0 tx=0).stop 2023-07-14T19:16:51.818 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:51.816+0000 7fea213be700 1 -- 172.21.15.40:0/1021997328 shutdown_connections 2023-07-14T19:16:51.818 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:51.816+0000 7fea213be700 1 --2- 172.21.15.40:0/1021997328 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7fea040a11e0 0x7fea040a3690 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:51.818 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:51.816+0000 7fea213be700 1 --2- 172.21.15.40:0/1021997328 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fea1c0fee40 0x7fea1c196b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:51.818 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:51.816+0000 7fea213be700 1 --2- 172.21.15.40:0/1021997328 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fea1c103ee0 0x7fea1c1a1b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:51.818 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:51.816+0000 7fea213be700 1 --2- 172.21.15.40:0/1021997328 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fea1c0ff7e0 0x7fea1c197050 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:51.818 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:51.816+0000 7fea213be700 1 -- 172.21.15.40:0/1021997328 >> 172.21.15.40:0/1021997328 conn(0x7fea1c075ca0 msgr2=0x7fea1c102790 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:16:51.819 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:51.817+0000 7fea213be700 1 -- 172.21.15.40:0/1021997328 shutdown_connections 2023-07-14T19:16:51.819 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:51.817+0000 7fea213be700 1 -- 172.21.15.40:0/1021997328 wait complete. 2023-07-14T19:16:52.221 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:16:51 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.a", "id": [2, 11]}]: dispatch 2023-07-14T19:16:52.221 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:16:51 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.b", "id": [8, 11]}]: dispatch 2023-07-14T19:16:52.221 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:16:51 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.c", "id": [2, 4]}]: dispatch 2023-07-14T19:16:52.221 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:16:51 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1f", "id": [8, 1]}]: dispatch 2023-07-14T19:16:52.222 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:16:51 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.e", "id": [4, 7]}]: dispatch 2023-07-14T19:16:52.222 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:16:51 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.17", "id": [4, 8]}]: dispatch 2023-07-14T19:16:52.222 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:16:51 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1a", "id": [5, 6]}]: dispatch 2023-07-14T19:16:52.222 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:16:51 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.20", "id": [5, 9]}]: dispatch 2023-07-14T19:16:52.222 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:16:51 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.27", "id": [3, 9]}]: dispatch 2023-07-14T19:16:52.222 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:16:51 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2c", "id": [4, 9]}]: dispatch 2023-07-14T19:16:52.223 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:16:51 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.a", "id": [2, 11]}]: dispatch 2023-07-14T19:16:52.223 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:16:51 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.b", "id": [8, 11]}]: dispatch 2023-07-14T19:16:52.223 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:16:51 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.c", "id": [2, 4]}]: dispatch 2023-07-14T19:16:52.224 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:16:51 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1f", "id": [8, 1]}]: dispatch 2023-07-14T19:16:52.224 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:16:51 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.e", "id": [4, 7]}]: dispatch 2023-07-14T19:16:52.224 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:16:51 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.17", "id": [4, 8]}]: dispatch 2023-07-14T19:16:52.224 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:16:51 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1a", "id": [5, 6]}]: dispatch 2023-07-14T19:16:52.224 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:16:51 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.20", "id": [5, 9]}]: dispatch 2023-07-14T19:16:52.224 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:16:51 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.27", "id": [3, 9]}]: dispatch 2023-07-14T19:16:52.224 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:16:51 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2c", "id": [4, 9]}]: dispatch 2023-07-14T19:16:52.242 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:16:51 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.a", "id": [2, 11]}]: dispatch 2023-07-14T19:16:52.243 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:16:51 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.b", "id": [8, 11]}]: dispatch 2023-07-14T19:16:52.243 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:16:51 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.c", "id": [2, 4]}]: dispatch 2023-07-14T19:16:52.243 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:16:51 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1f", "id": [8, 1]}]: dispatch 2023-07-14T19:16:52.243 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:16:51 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.e", "id": [4, 7]}]: dispatch 2023-07-14T19:16:52.244 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:16:51 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.17", "id": [4, 8]}]: dispatch 2023-07-14T19:16:52.244 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:16:51 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1a", "id": [5, 6]}]: dispatch 2023-07-14T19:16:52.244 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:16:51 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.20", "id": [5, 9]}]: dispatch 2023-07-14T19:16:52.245 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:16:51 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.27", "id": [3, 9]}]: dispatch 2023-07-14T19:16:52.245 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:16:51 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2c", "id": [4, 9]}]: dispatch 2023-07-14T19:16:52.261 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2023-07-14T19:16:52.261 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-07-14T19:16:52.391 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:52.390+0000 7f3272475700 1 -- 172.21.15.40:0/1895248234 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f326c1006c0 msgr2=0x7f326c106f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:16:52.391 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:52.390+0000 7f3272475700 1 --2- 172.21.15.40:0/1895248234 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f326c1006c0 0x7f326c106f80 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f326400b0a0 tx=0x7f326400b3b0 comp rx=0 tx=0).stop 2023-07-14T19:16:52.392 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:52.390+0000 7f3272475700 1 -- 172.21.15.40:0/1895248234 shutdown_connections 2023-07-14T19:16:52.392 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:52.390+0000 7f3272475700 1 --2- 172.21.15.40:0/1895248234 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f326c1006c0 0x7f326c106f80 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:52.392 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:52.390+0000 7f3272475700 1 --2- 172.21.15.40:0/1895248234 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f326c0feb40 0x7f326c0fef50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:52.392 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:52.390+0000 7f3272475700 1 --2- 172.21.15.40:0/1895248234 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f326c0ffd30 0x7f326c100180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:52.392 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:52.390+0000 7f3272475700 1 -- 172.21.15.40:0/1895248234 >> 172.21.15.40:0/1895248234 conn(0x7f326c0fa160 msgr2=0x7f326c0fc590 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:16:52.392 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:52.390+0000 7f3272475700 1 -- 172.21.15.40:0/1895248234 shutdown_connections 2023-07-14T19:16:52.393 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:52.390+0000 7f3272475700 1 -- 172.21.15.40:0/1895248234 wait complete. 2023-07-14T19:16:52.393 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:52.390+0000 7f3272475700 1 Processor -- start 2023-07-14T19:16:52.393 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:52.390+0000 7f3272475700 1 -- start start 2023-07-14T19:16:52.394 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:52.391+0000 7f3272475700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f326c0feb40 0x7f326c0692e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:16:52.395 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:52.391+0000 7f3272475700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f326c0ffd30 0x7f326c19a820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:16:52.395 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:52.391+0000 7f3272475700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f326c1006c0 0x7f326c19ce60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:16:52.395 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:52.391+0000 7f3272475700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f326c19d600 con 0x7f326c0ffd30 2023-07-14T19:16:52.395 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:52.391+0000 7f3272475700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f326c19d770 con 0x7f326c0feb40 2023-07-14T19:16:52.395 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:52.391+0000 7f3272475700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f326c19d930 con 0x7f326c1006c0 2023-07-14T19:16:52.395 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:52.391+0000 7f326b7fe700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f326c0ffd30 0x7f326c19a820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:16:52.396 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:52.391+0000 7f326b7fe700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f326c0ffd30 0x7f326c19a820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:33530/0 (socket says 172.21.15.40:33530) 2023-07-14T19:16:52.396 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:52.391+0000 7f326b7fe700 1 -- 172.21.15.40:0/4107412990 learned_addr learned my addr 172.21.15.40:0/4107412990 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-07-14T19:16:52.396 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:52.391+0000 7f326bfff700 1 --2- 172.21.15.40:0/4107412990 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f326c0feb40 0x7f326c0692e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:16:52.396 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:52.391+0000 7f3270a12700 1 --2- 172.21.15.40:0/4107412990 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f326c1006c0 0x7f326c19ce60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:16:52.396 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:52.391+0000 7f326b7fe700 1 -- 172.21.15.40:0/4107412990 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f326c1006c0 msgr2=0x7f326c19ce60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:16:52.396 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:52.391+0000 7f326b7fe700 1 --2- 172.21.15.40:0/4107412990 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f326c1006c0 0x7f326c19ce60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:52.396 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:52.391+0000 7f326b7fe700 1 -- 172.21.15.40:0/4107412990 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f326c0feb40 msgr2=0x7f326c0692e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:16:52.397 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:52.391+0000 7f326b7fe700 1 --2- 172.21.15.40:0/4107412990 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f326c0feb40 0x7f326c0692e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:52.397 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:52.391+0000 7f326b7fe700 1 -- 172.21.15.40:0/4107412990 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3264009d00 con 0x7f326c0ffd30 2023-07-14T19:16:52.397 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:52.391+0000 7f326b7fe700 1 --2- 172.21.15.40:0/4107412990 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f326c0ffd30 0x7f326c19a820 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f326000b530 tx=0x7f326000b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:16:52.397 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:52.392+0000 7f32697fa700 1 -- 172.21.15.40:0/4107412990 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3260011830 con 0x7f326c0ffd30 2023-07-14T19:16:52.397 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:52.392+0000 7f32697fa700 1 -- 172.21.15.40:0/4107412990 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3260011e50 con 0x7f326c0ffd30 2023-07-14T19:16:52.397 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:52.392+0000 7f32697fa700 1 -- 172.21.15.40:0/4107412990 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f326000f2e0 con 0x7f326c0ffd30 2023-07-14T19:16:52.397 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:52.392+0000 7f3272475700 1 -- 172.21.15.40:0/4107412990 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f326c19dc10 con 0x7f326c0ffd30 2023-07-14T19:16:52.397 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:52.392+0000 7f3272475700 1 -- 172.21.15.40:0/4107412990 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f326c19e120 con 0x7f326c0ffd30 2023-07-14T19:16:52.398 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:52.393+0000 7f3272475700 1 -- 172.21.15.40:0/4107412990 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f326c105fe0 con 0x7f326c0ffd30 2023-07-14T19:16:52.398 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:52.394+0000 7f32697fa700 1 -- 172.21.15.40:0/4107412990 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7f3260010380 con 0x7f326c0ffd30 2023-07-14T19:16:52.398 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:52.394+0000 7f32697fa700 1 --2- 172.21.15.40:0/4107412990 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f32540a12a0 0x7f32540a3750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:16:52.399 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:52.395+0000 7f326bfff700 1 --2- 172.21.15.40:0/4107412990 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f32540a12a0 0x7f32540a3750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:16:52.399 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:52.395+0000 7f32697fa700 1 -- 172.21.15.40:0/4107412990 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 8269+0+0 (secure 0 0 0) 0x7f32600cc460 con 0x7f326c0ffd30 2023-07-14T19:16:52.399 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:52.395+0000 7f326bfff700 1 --2- 172.21.15.40:0/4107412990 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f32540a12a0 0x7f32540a3750 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f326c0ffb90 tx=0x7f325c009190 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:16:52.399 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:52.399+0000 7f32697fa700 1 -- 172.21.15.40:0/4107412990 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f3260095b40 con 0x7f326c0ffd30 2023-07-14T19:16:52.603 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:52.602+0000 7f3272475700 1 -- 172.21.15.40:0/4107412990 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f326c02cae0 con 0x7f326c0ffd30 2023-07-14T19:16:52.604 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:52.604+0000 7f32697fa700 1 -- 172.21.15.40:0/4107412990 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v82) v1 ==== 74+0+22647 (secure 0 0 0) 0x7f3260099160 con 0x7f326c0ffd30 2023-07-14T19:16:52.605 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-07-14T19:16:52.605 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":82,"fsid":"adc52f04-2279-11ee-9b33-001a4aab830c","created":"2023-07-14T19:08:56.893730+0000","modified":"2023-07-14T19:16:51.757987+0000","last_up_change":"2023-07-14T19:15:16.632979+0000","last_in_change":"2023-07-14T19:14:58.961702+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-07-14T19:12:03.271333+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-07-14T19:15:56.073464+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-07-14T19:15:57.018726+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"0d79356a-95d2-49ad-857b-2092ae9f7e13","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":3599703125},{"type":"v1","addr":"172.21.15.40:6803","nonce":3599703125}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":3599703125},{"type":"v1","addr":"172.21.15.40:6805","nonce":3599703125}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":3599703125},{"type":"v1","addr":"172.21.15.40:6809","nonce":3599703125}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":3599703125},{"type":"v1","addr":"172.21.15.40:6807","nonce":3599703125}]},"public_addr":"172.21.15.40:6803/3599703125","cluster_addr":"172.21.15.40:6805/3599703125","heartbeat_back_addr":"172.21.15.40:6809/3599703125","heartbeat_front_addr":"172.21.15.40:6807/3599703125","state":["exists","up"]},{"osd":1,"uuid":"40f8a3d1-a5e4-473c-b9f2-7600ec22be0d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6800","nonce":2512760702},{"type":"v1","addr":"172.21.15.130:6801","nonce":2512760702}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6802","nonce":2512760702},{"type":"v1","addr":"172.21.15.130:6803","nonce":2512760702}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6806","nonce":2512760702},{"type":"v1","addr":"172.21.15.130:6807","nonce":2512760702}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6804","nonce":2512760702},{"type":"v1","addr":"172.21.15.130:6805","nonce":2512760702}]},"public_addr":"172.21.15.130:6801/2512760702","cluster_addr":"172.21.15.130:6803/2512760702","heartbeat_back_addr":"172.21.15.130:6807/2512760702","heartbeat_front_addr":"172.21.15.130:6805/2512760702","state":["exists","up"]},{"osd":2,"uuid":"5a157891-cb7a-4ea6-9239-f48a7c67c003","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6800","nonce":589569135},{"type":"v1","addr":"172.21.15.191:6801","nonce":589569135}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6802","nonce":589569135},{"type":"v1","addr":"172.21.15.191:6803","nonce":589569135}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6806","nonce":589569135},{"type":"v1","addr":"172.21.15.191:6807","nonce":589569135}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6804","nonce":589569135},{"type":"v1","addr":"172.21.15.191:6805","nonce":589569135}]},"public_addr":"172.21.15.191:6801/589569135","cluster_addr":"172.21.15.191:6803/589569135","heartbeat_back_addr":"172.21.15.191:6807/589569135","heartbeat_front_addr":"172.21.15.191:6805/589569135","state":["exists","up"]},{"osd":3,"uuid":"8020e7dc-ac40-487f-857a-8da40bccb459","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":996287860},{"type":"v1","addr":"172.21.15.40:6811","nonce":996287860}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6812","nonce":996287860},{"type":"v1","addr":"172.21.15.40:6813","nonce":996287860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6816","nonce":996287860},{"type":"v1","addr":"172.21.15.40:6817","nonce":996287860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":996287860},{"type":"v1","addr":"172.21.15.40:6815","nonce":996287860}]},"public_addr":"172.21.15.40:6811/996287860","cluster_addr":"172.21.15.40:6813/996287860","heartbeat_back_addr":"172.21.15.40:6817/996287860","heartbeat_front_addr":"172.21.15.40:6815/996287860","state":["exists","up"]},{"osd":4,"uuid":"c5501b35-f62d-482d-8d22-115c88f0298e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6808","nonce":3507448720},{"type":"v1","addr":"172.21.15.130:6809","nonce":3507448720}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6810","nonce":3507448720},{"type":"v1","addr":"172.21.15.130:6811","nonce":3507448720}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6814","nonce":3507448720},{"type":"v1","addr":"172.21.15.130:6815","nonce":3507448720}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6812","nonce":3507448720},{"type":"v1","addr":"172.21.15.130:6813","nonce":3507448720}]},"public_addr":"172.21.15.130:6809/3507448720","cluster_addr":"172.21.15.130:6811/3507448720","heartbeat_back_addr":"172.21.15.130:6815/3507448720","heartbeat_front_addr":"172.21.15.130:6813/3507448720","state":["exists","up"]},{"osd":5,"uuid":"7770b6d6-5874-493a-986a-84db482cee42","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6808","nonce":1781944663},{"type":"v1","addr":"172.21.15.191:6809","nonce":1781944663}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6810","nonce":1781944663},{"type":"v1","addr":"172.21.15.191:6811","nonce":1781944663}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6814","nonce":1781944663},{"type":"v1","addr":"172.21.15.191:6815","nonce":1781944663}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6812","nonce":1781944663},{"type":"v1","addr":"172.21.15.191:6813","nonce":1781944663}]},"public_addr":"172.21.15.191:6809/1781944663","cluster_addr":"172.21.15.191:6811/1781944663","heartbeat_back_addr":"172.21.15.191:6815/1781944663","heartbeat_front_addr":"172.21.15.191:6813/1781944663","state":["exists","up"]},{"osd":6,"uuid":"ef65ad00-dfee-4300-9961-dfc8573aab06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6818","nonce":693186773},{"type":"v1","addr":"172.21.15.40:6819","nonce":693186773}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6820","nonce":693186773},{"type":"v1","addr":"172.21.15.40:6821","nonce":693186773}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6824","nonce":693186773},{"type":"v1","addr":"172.21.15.40:6825","nonce":693186773}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6822","nonce":693186773},{"type":"v1","addr":"172.21.15.40:6823","nonce":693186773}]},"public_addr":"172.21.15.40:6819/693186773","cluster_addr":"172.21.15.40:6821/693186773","heartbeat_back_addr":"172.21.15.40:6825/693186773","heartbeat_front_addr":"172.21.15.40:6823/693186773","state":["exists","up"]},{"osd":7,"uuid":"1700fbfa-cbfc-4f76-97be-727a3b100740","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":50,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6816","nonce":2384019528},{"type":"v1","addr":"172.21.15.130:6817","nonce":2384019528}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6818","nonce":2384019528},{"type":"v1","addr":"172.21.15.130:6819","nonce":2384019528}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6822","nonce":2384019528},{"type":"v1","addr":"172.21.15.130:6823","nonce":2384019528}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6820","nonce":2384019528},{"type":"v1","addr":"172.21.15.130:6821","nonce":2384019528}]},"public_addr":"172.21.15.130:6817/2384019528","cluster_addr":"172.21.15.130:6819/2384019528","heartbeat_back_addr":"172.21.15.130:6823/2384019528","heartbeat_front_addr":"172.21.15.130:6821/2384019528","state":["exists","up"]},{"osd":8,"uuid":"b76d4033-c1e2-457b-8571-c8a49fe443f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":57,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6816","nonce":2386818417},{"type":"v1","addr":"172.21.15.191:6817","nonce":2386818417}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6818","nonce":2386818417},{"type":"v1","addr":"172.21.15.191:6819","nonce":2386818417}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6822","nonce":2386818417},{"type":"v1","addr":"172.21.15.191:6823","nonce":2386818417}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6820","nonce":2386818417},{"type":"v1","addr":"172.21.15.191:6821","nonce":2386818417}]},"public_addr":"172.21.15.191:6817/2386818417","cluster_addr":"172.21.15.191:6819/2386818417","heartbeat_back_addr":"172.21.15.191:6823/2386818417","heartbeat_front_addr":"172.21.15.191:6821/2386818417","state":["exists","up"]},{"osd":9,"uuid":"df31bf2f-e30e-41b6-b2b9-0e26a07a88bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":64,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6826","nonce":2228380263},{"type":"v1","addr":"172.21.15.40:6827","nonce":2228380263}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6828","nonce":2228380263},{"type":"v1","addr":"172.21.15.40:6829","nonce":2228380263}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":2228380263},{"type":"v1","addr":"172.21.15.40:6833","nonce":2228380263}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6830","nonce":2228380263},{"type":"v1","addr":"172.21.15.40:6831","nonce":2228380263}]},"public_addr":"172.21.15.40:6827/2228380263","cluster_addr":"172.21.15.40:6829/2228380263","heartbeat_back_addr":"172.21.15.40:6833/2228380263","heartbeat_front_addr":"172.21.15.40:6831/2228380263","state":["exists","up"]},{"osd":10,"uuid":"4a93d621-49ee-4ad7-9e49-750614faf912","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6824","nonce":2417067798},{"type":"v1","addr":"172.21.15.130:6825","nonce":2417067798}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6826","nonce":2417067798},{"type":"v1","addr":"172.21.15.130:6827","nonce":2417067798}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6830","nonce":2417067798},{"type":"v1","addr":"172.21.15.130:6831","nonce":2417067798}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6828","nonce":2417067798},{"type":"v1","addr":"172.21.15.130:6829","nonce":2417067798}]},"public_addr":"172.21.15.130:6825/2417067798","cluster_addr":"172.21.15.130:6827/2417067798","heartbeat_back_addr":"172.21.15.130:6831/2417067798","heartbeat_front_addr":"172.21.15.130:6829/2417067798","state":["exists","up"]},{"osd":11,"uuid":"f43a2a6f-82f3-42ac-b77c-4f1b34da6918","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":76,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6824","nonce":2886059446},{"type":"v1","addr":"172.21.15.191:6825","nonce":2886059446}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6826","nonce":2886059446},{"type":"v1","addr":"172.21.15.191:6827","nonce":2886059446}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6830","nonce":2886059446},{"type":"v1","addr":"172.21.15.191:6831","nonce":2886059446}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6828","nonce":2886059446},{"type":"v1","addr":"172.21.15.191:6829","nonce":2886059446}]},"public_addr":"172.21.15.191:6825/2886059446","cluster_addr":"172.21.15.191:6827/2886059446","heartbeat_back_addr":"172.21.15.191:6831/2886059446","heartbeat_front_addr":"172.21.15.191:6829/2886059446","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:11:18.320495+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:11:39.901199+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:12:00.869042+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:12:23.292742+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:12:44.587019+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:13:06.494893+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:13:28.378374+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:13:50.033491+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:14:10.937062+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:14:32.358498+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:14:53.666584+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:15:14.837566+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"2.b","mappings":[{"from":8,"to":11}]},{"pgid":"2.c","mappings":[{"from":2,"to":4}]},{"pgid":"2.1f","mappings":[{"from":8,"to":1}]},{"pgid":"3.e","mappings":[{"from":4,"to":7}]},{"pgid":"3.17","mappings":[{"from":4,"to":8}]},{"pgid":"3.1a","mappings":[{"from":5,"to":6}]},{"pgid":"3.20","mappings":[{"from":5,"to":9}]},{"pgid":"3.27","mappings":[{"from":3,"to":9}]},{"pgid":"3.2c","mappings":[{"from":4,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.40:6801/3698757264":"2023-07-15T19:09:50.732458+0000","172.21.15.40:6800/3698757264":"2023-07-15T19:09:50.732458+0000","172.21.15.40:6800/2569015667":"2023-07-15T19:09:24.441543+0000","172.21.15.40:0/3653961003":"2023-07-15T19:09:24.441543+0000","172.21.15.40:0/3568861680":"2023-07-15T19:09:24.441543+0000","172.21.15.40:0/3249773172":"2023-07-15T19:09:50.732458+0000","172.21.15.40:0/4084215480":"2023-07-15T19:09:24.441543+0000","172.21.15.40:0/1163941760":"2023-07-15T19:09:50.732458+0000","172.21.15.40:6801/2569015667":"2023-07-15T19:09:24.441543+0000","172.21.15.40:0/2326639960":"2023-07-15T19:09:50.732458+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-07-14T19:16:52.608 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:52.606+0000 7f3272475700 1 -- 172.21.15.40:0/4107412990 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f32540a12a0 msgr2=0x7f32540a3750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:16:52.608 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:52.606+0000 7f3272475700 1 --2- 172.21.15.40:0/4107412990 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f32540a12a0 0x7f32540a3750 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f326c0ffb90 tx=0x7f325c009190 comp rx=0 tx=0).stop 2023-07-14T19:16:52.608 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:52.606+0000 7f3272475700 1 -- 172.21.15.40:0/4107412990 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f326c0ffd30 msgr2=0x7f326c19a820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:16:52.608 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:52.606+0000 7f3272475700 1 --2- 172.21.15.40:0/4107412990 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f326c0ffd30 0x7f326c19a820 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f326000b530 tx=0x7f326000b840 comp rx=0 tx=0).stop 2023-07-14T19:16:52.609 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:52.607+0000 7f3272475700 1 -- 172.21.15.40:0/4107412990 shutdown_connections 2023-07-14T19:16:52.609 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:52.607+0000 7f3272475700 1 --2- 172.21.15.40:0/4107412990 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f32540a12a0 0x7f32540a3750 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:52.609 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:52.607+0000 7f3272475700 1 --2- 172.21.15.40:0/4107412990 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f326c1006c0 0x7f326c19ce60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:52.610 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:52.607+0000 7f3272475700 1 --2- 172.21.15.40:0/4107412990 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f326c0feb40 0x7f326c0692e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:52.610 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:52.607+0000 7f3272475700 1 --2- 172.21.15.40:0/4107412990 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f326c0ffd30 0x7f326c19a820 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:52.610 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:52.607+0000 7f3272475700 1 -- 172.21.15.40:0/4107412990 >> 172.21.15.40:0/4107412990 conn(0x7f326c0fa160 msgr2=0x7f326c103120 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:16:52.610 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:52.607+0000 7f3272475700 1 -- 172.21.15.40:0/4107412990 shutdown_connections 2023-07-14T19:16:52.611 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:52.607+0000 7f3272475700 1 -- 172.21.15.40:0/4107412990 wait complete. 2023-07-14T19:16:52.620 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-07-14T19:12:03.271333+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-07-14T19:15:56.073464+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '80', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-07-14T19:15:57.018726+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '81', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-07-14T19:16:52.621 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-07-14T19:16:52.752 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:52.749+0000 7f8456f62700 1 -- 172.21.15.40:0/3698014465 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8450103010 msgr2=0x7f84501053f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:16:52.753 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:52.749+0000 7f8456f62700 1 --2- 172.21.15.40:0/3698014465 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8450103010 0x7f84501053f0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f843c009990 tx=0x7f843c009ca0 comp rx=0 tx=0).stop 2023-07-14T19:16:52.753 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:52.749+0000 7f8456f62700 1 -- 172.21.15.40:0/3698014465 shutdown_connections 2023-07-14T19:16:52.753 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:52.749+0000 7f8456f62700 1 --2- 172.21.15.40:0/3698014465 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f84501006f0 0x7f8450102ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:52.753 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:52.749+0000 7f8456f62700 1 --2- 172.21.15.40:0/3698014465 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8450105930 0x7f8450107d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:52.753 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:52.749+0000 7f8456f62700 1 --2- 172.21.15.40:0/3698014465 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8450103010 0x7f84501053f0 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:52.754 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:52.749+0000 7f8456f62700 1 -- 172.21.15.40:0/3698014465 >> 172.21.15.40:0/3698014465 conn(0x7f84500fa130 msgr2=0x7f84500fc560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:16:52.754 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:52.749+0000 7f8456f62700 1 -- 172.21.15.40:0/3698014465 shutdown_connections 2023-07-14T19:16:52.754 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:52.749+0000 7f8456f62700 1 -- 172.21.15.40:0/3698014465 wait complete. 2023-07-14T19:16:52.754 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:52.750+0000 7f8456f62700 1 Processor -- start 2023-07-14T19:16:52.754 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:52.750+0000 7f8456f62700 1 -- start start 2023-07-14T19:16:52.754 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:52.750+0000 7f8456f62700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f84501006f0 0x7f8450195990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:16:52.754 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:52.750+0000 7f8456f62700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8450103010 0x7f8450195ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:16:52.755 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:52.750+0000 7f8456f62700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8450105930 0x7f845019cf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:16:52.755 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:52.750+0000 7f8456f62700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f845019d510 con 0x7f8450105930 2023-07-14T19:16:52.755 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:52.750+0000 7f8456f62700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f845019d650 con 0x7f8450103010 2023-07-14T19:16:52.755 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:52.750+0000 7f8456f62700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f845019d7e0 con 0x7f84501006f0 2023-07-14T19:16:52.755 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:52.750+0000 7f84554ff700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8450105930 0x7f845019cf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:16:52.755 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:52.750+0000 7f84554ff700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8450105930 0x7f845019cf40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:33546/0 (socket says 172.21.15.40:33546) 2023-07-14T19:16:52.756 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:52.750+0000 7f84554ff700 1 -- 172.21.15.40:0/2091946707 learned_addr learned my addr 172.21.15.40:0/2091946707 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-07-14T19:16:52.756 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:52.750+0000 7f8454cfe700 1 --2- 172.21.15.40:0/2091946707 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f84501006f0 0x7f8450195990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:16:52.756 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:52.750+0000 7f844ffff700 1 --2- 172.21.15.40:0/2091946707 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8450103010 0x7f8450195ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:16:52.756 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:52.751+0000 7f84554ff700 1 -- 172.21.15.40:0/2091946707 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f84501006f0 msgr2=0x7f8450195990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:16:52.756 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:52.751+0000 7f84554ff700 1 --2- 172.21.15.40:0/2091946707 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f84501006f0 0x7f8450195990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:52.757 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:52.751+0000 7f84554ff700 1 -- 172.21.15.40:0/2091946707 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8450103010 msgr2=0x7f8450195ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:16:52.757 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:52.751+0000 7f84554ff700 1 --2- 172.21.15.40:0/2091946707 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8450103010 0x7f8450195ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:52.757 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:52.751+0000 7f84554ff700 1 -- 172.21.15.40:0/2091946707 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f843c0095f0 con 0x7f8450105930 2023-07-14T19:16:52.757 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:52.751+0000 7f844ffff700 1 --2- 172.21.15.40:0/2091946707 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8450103010 0x7f8450195ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-14T19:16:52.757 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:52.751+0000 7f84554ff700 1 --2- 172.21.15.40:0/2091946707 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8450105930 0x7f845019cf40 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f8448007ee0 tx=0x7f844800d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:16:52.757 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:52.751+0000 7f844dffb700 1 -- 172.21.15.40:0/2091946707 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8448017070 con 0x7f8450105930 2023-07-14T19:16:52.758 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:52.751+0000 7f8456f62700 1 -- 172.21.15.40:0/2091946707 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f845019dac0 con 0x7f8450105930 2023-07-14T19:16:52.758 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:52.751+0000 7f844dffb700 1 -- 172.21.15.40:0/2091946707 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f844800f040 con 0x7f8450105930 2023-07-14T19:16:52.758 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:52.751+0000 7f844dffb700 1 -- 172.21.15.40:0/2091946707 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8448005740 con 0x7f8450105930 2023-07-14T19:16:52.758 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:52.751+0000 7f8456f62700 1 -- 172.21.15.40:0/2091946707 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f845019df80 con 0x7f8450105930 2023-07-14T19:16:52.758 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:52.752+0000 7f8456f62700 1 -- 172.21.15.40:0/2091946707 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f84500fbca0 con 0x7f8450105930 2023-07-14T19:16:52.759 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:52.753+0000 7f844dffb700 1 -- 172.21.15.40:0/2091946707 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7f84480058a0 con 0x7f8450105930 2023-07-14T19:16:52.759 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:52.754+0000 7f844dffb700 1 --2- 172.21.15.40:0/2091946707 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f84400a11e0 0x7f84400a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:16:52.760 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:52.754+0000 7f8454cfe700 1 --2- 172.21.15.40:0/2091946707 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f84400a11e0 0x7f84400a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:16:52.760 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:52.754+0000 7f844dffb700 1 -- 172.21.15.40:0/2091946707 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 8269+0+0 (secure 0 0 0) 0x7f84480cc3b0 con 0x7f8450105930 2023-07-14T19:16:52.760 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:52.754+0000 7f8454cfe700 1 --2- 172.21.15.40:0/2091946707 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f84400a11e0 0x7f84400a3690 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f8444009bf0 tx=0x7f8444009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:16:52.760 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:52.758+0000 7f844dffb700 1 -- 172.21.15.40:0/2091946707 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f8448095920 con 0x7f8450105930 2023-07-14T19:16:52.962 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:52.961+0000 7f8456f62700 1 -- 172.21.15.40:0/2091946707 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f845002cb30 con 0x7f8450105930 2023-07-14T19:16:52.962 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:52.961+0000 7f844dffb700 1 -- 172.21.15.40:0/2091946707 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v83) v1 ==== 93+0+10 (secure 0 0 0) 0x7f8448098f40 con 0x7f8450105930 2023-07-14T19:16:52.963 INFO:teuthology.orchestra.run.smithi040.stdout:pg_num: 1 2023-07-14T19:16:52.964 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:52.963+0000 7f8456f62700 1 -- 172.21.15.40:0/2091946707 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f84400a11e0 msgr2=0x7f84400a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:16:52.964 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:52.963+0000 7f8456f62700 1 --2- 172.21.15.40:0/2091946707 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f84400a11e0 0x7f84400a3690 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f8444009bf0 tx=0x7f8444009260 comp rx=0 tx=0).stop 2023-07-14T19:16:52.965 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:52.963+0000 7f8456f62700 1 -- 172.21.15.40:0/2091946707 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8450105930 msgr2=0x7f845019cf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:16:52.965 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:52.963+0000 7f8456f62700 1 --2- 172.21.15.40:0/2091946707 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8450105930 0x7f845019cf40 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f8448007ee0 tx=0x7f844800d3b0 comp rx=0 tx=0).stop 2023-07-14T19:16:52.965 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:52.963+0000 7f8456f62700 1 -- 172.21.15.40:0/2091946707 shutdown_connections 2023-07-14T19:16:52.965 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:52.963+0000 7f8456f62700 1 --2- 172.21.15.40:0/2091946707 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f84400a11e0 0x7f84400a3690 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:52.965 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:52.963+0000 7f8456f62700 1 --2- 172.21.15.40:0/2091946707 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f84501006f0 0x7f8450195990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:52.965 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:52.963+0000 7f8456f62700 1 --2- 172.21.15.40:0/2091946707 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8450105930 0x7f845019cf40 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:52.965 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:52.963+0000 7f8456f62700 1 --2- 172.21.15.40:0/2091946707 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8450103010 0x7f8450195ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:52.966 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:52.963+0000 7f8456f62700 1 -- 172.21.15.40:0/2091946707 >> 172.21.15.40:0/2091946707 conn(0x7f84500fa130 msgr2=0x7f8450101350 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:16:52.966 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:52.964+0000 7f8456f62700 1 -- 172.21.15.40:0/2091946707 shutdown_connections 2023-07-14T19:16:52.966 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:52.964+0000 7f8456f62700 1 -- 172.21.15.40:0/2091946707 wait complete. 2023-07-14T19:16:52.971 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:16:52 smithi040 ceph-mon[96518]: pgmap v281: 97 pgs: 97 active+clean; 638 KiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 10 op/s 2023-07-14T19:16:52.971 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:16:52 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.a", "id": [2, 11]}]': finished 2023-07-14T19:16:52.971 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:16:52 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.b", "id": [8, 11]}]': finished 2023-07-14T19:16:52.971 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:16:52 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.c", "id": [2, 4]}]': finished 2023-07-14T19:16:52.971 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:16:52 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1f", "id": [8, 1]}]': finished 2023-07-14T19:16:52.971 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:16:52 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.e", "id": [4, 7]}]': finished 2023-07-14T19:16:52.972 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:16:52 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.17", "id": [4, 8]}]': finished 2023-07-14T19:16:52.972 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:16:52 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1a", "id": [5, 6]}]': finished 2023-07-14T19:16:52.972 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:16:52 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.20", "id": [5, 9]}]': finished 2023-07-14T19:16:52.972 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:16:52 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.27", "id": [3, 9]}]': finished 2023-07-14T19:16:52.972 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:16:52 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2c", "id": [4, 9]}]': finished 2023-07-14T19:16:52.972 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:16:52 smithi040 ceph-mon[96518]: osdmap e82: 12 total, 12 up, 12 in 2023-07-14T19:16:52.972 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:16:52 smithi040 ceph-mon[96518]: from='client.? 172.21.15.40:0/1021997328' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2023-07-14T19:16:52.973 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:16:52 smithi040 ceph-mon[96518]: from='client.? 172.21.15.40:0/4107412990' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T19:16:52.977 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-07-14T19:16:53.109 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:53.107+0000 7faa24071700 1 -- 172.21.15.40:0/3462437874 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7faa1c105910 msgr2=0x7faa1c107d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:16:53.109 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:53.107+0000 7faa24071700 1 --2- 172.21.15.40:0/3462437874 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7faa1c105910 0x7faa1c107d30 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7faa1400b0a0 tx=0x7faa1400b3b0 comp rx=0 tx=0).stop 2023-07-14T19:16:53.109 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:53.108+0000 7faa24071700 1 -- 172.21.15.40:0/3462437874 shutdown_connections 2023-07-14T19:16:53.109 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:53.108+0000 7faa24071700 1 --2- 172.21.15.40:0/3462437874 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7faa1c105910 0x7faa1c107d30 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:53.109 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:53.108+0000 7faa24071700 1 --2- 172.21.15.40:0/3462437874 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7faa1c1006d0 0x7faa1c102ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:53.110 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:53.108+0000 7faa24071700 1 --2- 172.21.15.40:0/3462437874 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7faa1c102ff0 0x7faa1c1053d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:53.110 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:53.108+0000 7faa24071700 1 -- 172.21.15.40:0/3462437874 >> 172.21.15.40:0/3462437874 conn(0x7faa1c0fa130 msgr2=0x7faa1c0fc560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:16:53.110 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:53.108+0000 7faa24071700 1 -- 172.21.15.40:0/3462437874 shutdown_connections 2023-07-14T19:16:53.110 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:53.108+0000 7faa24071700 1 -- 172.21.15.40:0/3462437874 wait complete. 2023-07-14T19:16:53.110 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:53.108+0000 7faa24071700 1 Processor -- start 2023-07-14T19:16:53.110 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:53.108+0000 7faa24071700 1 -- start start 2023-07-14T19:16:53.112 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:53.108+0000 7faa24071700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7faa1c1006d0 0x7faa1c100040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:16:53.112 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:53.108+0000 7faa24071700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7faa1c102ff0 0x7faa1c0fe680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:16:53.112 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:53.108+0000 7faa24071700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7faa1c105910 0x7faa1c0febc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:16:53.112 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:53.108+0000 7faa24071700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa1c0ff360 con 0x7faa1c105910 2023-07-14T19:16:53.112 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:53.108+0000 7faa24071700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa1c0ff4d0 con 0x7faa1c102ff0 2023-07-14T19:16:53.113 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:53.108+0000 7faa24071700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa1c19faa0 con 0x7faa1c1006d0 2023-07-14T19:16:53.113 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:53.109+0000 7faa2260e700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7faa1c105910 0x7faa1c0febc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:16:53.113 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:53.109+0000 7faa2260e700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7faa1c105910 0x7faa1c0febc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:33560/0 (socket says 172.21.15.40:33560) 2023-07-14T19:16:53.113 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:53.109+0000 7faa2260e700 1 -- 172.21.15.40:0/1783321818 learned_addr learned my addr 172.21.15.40:0/1783321818 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-07-14T19:16:53.113 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:53.109+0000 7faa21e0d700 1 --2- 172.21.15.40:0/1783321818 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7faa1c1006d0 0x7faa1c100040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:16:53.113 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:53.109+0000 7faa2160c700 1 --2- 172.21.15.40:0/1783321818 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7faa1c102ff0 0x7faa1c0fe680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:16:53.113 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:53.109+0000 7faa2260e700 1 -- 172.21.15.40:0/1783321818 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7faa1c1006d0 msgr2=0x7faa1c100040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:16:53.114 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:53.109+0000 7faa2260e700 1 --2- 172.21.15.40:0/1783321818 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7faa1c1006d0 0x7faa1c100040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:53.114 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:53.109+0000 7faa2260e700 1 -- 172.21.15.40:0/1783321818 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7faa1c102ff0 msgr2=0x7faa1c0fe680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:16:53.114 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:53.109+0000 7faa2260e700 1 --2- 172.21.15.40:0/1783321818 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7faa1c102ff0 0x7faa1c0fe680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:53.114 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:53.109+0000 7faa2260e700 1 -- 172.21.15.40:0/1783321818 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa14009d00 con 0x7faa1c105910 2023-07-14T19:16:53.114 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:53.109+0000 7faa2260e700 1 --2- 172.21.15.40:0/1783321818 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7faa1c105910 0x7faa1c0febc0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7faa140062a0 tx=0x7faa14004d00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:16:53.114 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:53.109+0000 7faa12ffd700 1 -- 172.21.15.40:0/1783321818 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa1400e040 con 0x7faa1c105910 2023-07-14T19:16:53.114 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:53.109+0000 7faa12ffd700 1 -- 172.21.15.40:0/1783321818 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7faa1400cec0 con 0x7faa1c105910 2023-07-14T19:16:53.114 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:53.109+0000 7faa24071700 1 -- 172.21.15.40:0/1783321818 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faa1c19fd20 con 0x7faa1c105910 2023-07-14T19:16:53.115 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:53.109+0000 7faa12ffd700 1 -- 172.21.15.40:0/1783321818 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa1401baf0 con 0x7faa1c105910 2023-07-14T19:16:53.115 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:53.109+0000 7faa24071700 1 -- 172.21.15.40:0/1783321818 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faa1c1a0150 con 0x7faa1c105910 2023-07-14T19:16:53.115 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:53.110+0000 7faa24071700 1 -- 172.21.15.40:0/1783321818 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faa1c0fbca0 con 0x7faa1c105910 2023-07-14T19:16:53.115 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:53.111+0000 7faa12ffd700 1 -- 172.21.15.40:0/1783321818 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7faa1401bc50 con 0x7faa1c105910 2023-07-14T19:16:53.116 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:53.112+0000 7faa12ffd700 1 --2- 172.21.15.40:0/1783321818 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7faa080a11e0 0x7faa080a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:16:53.116 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:53.112+0000 7faa12ffd700 1 -- 172.21.15.40:0/1783321818 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 8269+0+0 (secure 0 0 0) 0x7faa140ce730 con 0x7faa1c105910 2023-07-14T19:16:53.116 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:53.112+0000 7faa21e0d700 1 --2- 172.21.15.40:0/1783321818 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7faa080a11e0 0x7faa080a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:16:53.116 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:53.112+0000 7faa21e0d700 1 --2- 172.21.15.40:0/1783321818 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7faa080a11e0 0x7faa080a3690 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7faa18007c60 tx=0x7faa180073d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:16:53.116 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:53.116+0000 7faa12ffd700 1 -- 172.21.15.40:0/1783321818 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7faa14097d90 con 0x7faa1c105910 2023-07-14T19:16:53.222 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:16:52 smithi191 ceph-mon[98311]: pgmap v281: 97 pgs: 97 active+clean; 638 KiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 10 op/s 2023-07-14T19:16:53.222 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:16:52 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.a", "id": [2, 11]}]': finished 2023-07-14T19:16:53.222 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:16:52 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.b", "id": [8, 11]}]': finished 2023-07-14T19:16:53.222 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:16:52 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.c", "id": [2, 4]}]': finished 2023-07-14T19:16:53.223 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:16:52 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1f", "id": [8, 1]}]': finished 2023-07-14T19:16:53.223 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:16:52 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.e", "id": [4, 7]}]': finished 2023-07-14T19:16:53.223 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:16:52 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.17", "id": [4, 8]}]': finished 2023-07-14T19:16:53.223 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:16:52 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1a", "id": [5, 6]}]': finished 2023-07-14T19:16:53.223 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:16:52 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.20", "id": [5, 9]}]': finished 2023-07-14T19:16:53.223 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:16:52 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.27", "id": [3, 9]}]': finished 2023-07-14T19:16:53.223 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:16:52 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2c", "id": [4, 9]}]': finished 2023-07-14T19:16:53.223 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:16:52 smithi191 ceph-mon[98311]: osdmap e82: 12 total, 12 up, 12 in 2023-07-14T19:16:53.224 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:16:52 smithi191 ceph-mon[98311]: from='client.? 172.21.15.40:0/1021997328' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2023-07-14T19:16:53.224 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:16:52 smithi191 ceph-mon[98311]: from='client.? 172.21.15.40:0/4107412990' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T19:16:53.242 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:16:52 smithi130 ceph-mon[97832]: pgmap v281: 97 pgs: 97 active+clean; 638 KiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 10 op/s 2023-07-14T19:16:53.243 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:16:52 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.a", "id": [2, 11]}]': finished 2023-07-14T19:16:53.243 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:16:52 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.b", "id": [8, 11]}]': finished 2023-07-14T19:16:53.243 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:16:52 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.c", "id": [2, 4]}]': finished 2023-07-14T19:16:53.243 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:16:52 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1f", "id": [8, 1]}]': finished 2023-07-14T19:16:53.243 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:16:52 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.e", "id": [4, 7]}]': finished 2023-07-14T19:16:53.243 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:16:52 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.17", "id": [4, 8]}]': finished 2023-07-14T19:16:53.244 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:16:52 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1a", "id": [5, 6]}]': finished 2023-07-14T19:16:53.244 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:16:52 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.20", "id": [5, 9]}]': finished 2023-07-14T19:16:53.244 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:16:52 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.27", "id": [3, 9]}]': finished 2023-07-14T19:16:53.244 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:16:52 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2c", "id": [4, 9]}]': finished 2023-07-14T19:16:53.244 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:16:52 smithi130 ceph-mon[97832]: osdmap e82: 12 total, 12 up, 12 in 2023-07-14T19:16:53.244 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:16:52 smithi130 ceph-mon[97832]: from='client.? 172.21.15.40:0/1021997328' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2023-07-14T19:16:53.244 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:16:52 smithi130 ceph-mon[97832]: from='client.? 172.21.15.40:0/4107412990' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T19:16:53.318 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:53.318+0000 7faa24071700 1 -- 172.21.15.40:0/1783321818 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7faa1c02cb00 con 0x7faa1c105910 2023-07-14T19:16:53.319 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:53.318+0000 7faa12ffd700 1 -- 172.21.15.40:0/1783321818 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v83) v1 ==== 104+0+11 (secure 0 0 0) 0x7faa1409b3b0 con 0x7faa1c105910 2023-07-14T19:16:53.320 INFO:teuthology.orchestra.run.smithi040.stdout:pg_num: 32 2023-07-14T19:16:53.322 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:53.320+0000 7faa24071700 1 -- 172.21.15.40:0/1783321818 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7faa080a11e0 msgr2=0x7faa080a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:16:53.322 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:53.320+0000 7faa24071700 1 --2- 172.21.15.40:0/1783321818 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7faa080a11e0 0x7faa080a3690 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7faa18007c60 tx=0x7faa180073d0 comp rx=0 tx=0).stop 2023-07-14T19:16:53.322 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:53.320+0000 7faa24071700 1 -- 172.21.15.40:0/1783321818 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7faa1c105910 msgr2=0x7faa1c0febc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:16:53.323 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:53.320+0000 7faa24071700 1 --2- 172.21.15.40:0/1783321818 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7faa1c105910 0x7faa1c0febc0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7faa140062a0 tx=0x7faa14004d00 comp rx=0 tx=0).stop 2023-07-14T19:16:53.323 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:53.320+0000 7faa24071700 1 -- 172.21.15.40:0/1783321818 shutdown_connections 2023-07-14T19:16:53.323 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:53.320+0000 7faa24071700 1 --2- 172.21.15.40:0/1783321818 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7faa080a11e0 0x7faa080a3690 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:53.323 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:53.320+0000 7faa24071700 1 --2- 172.21.15.40:0/1783321818 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7faa1c1006d0 0x7faa1c100040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:53.324 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:53.320+0000 7faa24071700 1 --2- 172.21.15.40:0/1783321818 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7faa1c105910 0x7faa1c0febc0 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:53.324 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:53.320+0000 7faa24071700 1 --2- 172.21.15.40:0/1783321818 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7faa1c102ff0 0x7faa1c0fe680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:53.324 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:53.320+0000 7faa24071700 1 -- 172.21.15.40:0/1783321818 >> 172.21.15.40:0/1783321818 conn(0x7faa1c0fa130 msgr2=0x7faa1c101330 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:16:53.325 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:53.320+0000 7faa24071700 1 -- 172.21.15.40:0/1783321818 shutdown_connections 2023-07-14T19:16:53.325 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:53.320+0000 7faa24071700 1 -- 172.21.15.40:0/1783321818 wait complete. 2023-07-14T19:16:53.334 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-07-14T19:16:53.462 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:53.462+0000 7f41a580f700 1 -- 172.21.15.40:0/880790662 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f41a00feb10 msgr2=0x7f41a00fef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:16:53.463 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:53.462+0000 7f41a580f700 1 --2- 172.21.15.40:0/880790662 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f41a00feb10 0x7f41a00fef20 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f4194009990 tx=0x7f4194009ca0 comp rx=0 tx=0).stop 2023-07-14T19:16:53.463 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:53.462+0000 7f41a580f700 1 -- 172.21.15.40:0/880790662 shutdown_connections 2023-07-14T19:16:53.463 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:53.462+0000 7f41a580f700 1 --2- 172.21.15.40:0/880790662 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f41a0100690 0x7f41a0106f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:53.463 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:53.462+0000 7f41a580f700 1 --2- 172.21.15.40:0/880790662 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f41a00feb10 0x7f41a00fef20 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:53.464 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:53.462+0000 7f41a580f700 1 --2- 172.21.15.40:0/880790662 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f41a00ffd00 0x7f41a0100150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:53.464 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:53.462+0000 7f41a580f700 1 -- 172.21.15.40:0/880790662 >> 172.21.15.40:0/880790662 conn(0x7f41a00fa130 msgr2=0x7f41a00fc560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:16:53.464 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:53.462+0000 7f41a580f700 1 -- 172.21.15.40:0/880790662 shutdown_connections 2023-07-14T19:16:53.464 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:53.462+0000 7f41a580f700 1 -- 172.21.15.40:0/880790662 wait complete. 2023-07-14T19:16:53.466 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:53.462+0000 7f41a580f700 1 Processor -- start 2023-07-14T19:16:53.466 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:53.462+0000 7f41a580f700 1 -- start start 2023-07-14T19:16:53.466 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:53.463+0000 7f41a580f700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f41a00feb10 0x7f41a0194810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:16:53.467 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:53.463+0000 7f41a580f700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f41a00ffd00 0x7f41a0194d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:16:53.467 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:53.463+0000 7f41a580f700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f41a0100690 0x7f41a019bdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:16:53.467 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:53.463+0000 7f41a580f700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41a019c390 con 0x7f41a00ffd00 2023-07-14T19:16:53.467 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:53.463+0000 7f41a580f700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41a019c4d0 con 0x7f41a0100690 2023-07-14T19:16:53.467 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:53.463+0000 7f41a580f700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41a019c660 con 0x7f41a00feb10 2023-07-14T19:16:53.467 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:53.463+0000 7f419e7fc700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f41a00ffd00 0x7f41a0194d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:16:53.467 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:53.463+0000 7f419e7fc700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f41a00ffd00 0x7f41a0194d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:33572/0 (socket says 172.21.15.40:33572) 2023-07-14T19:16:53.468 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:53.463+0000 7f419e7fc700 1 -- 172.21.15.40:0/362624095 learned_addr learned my addr 172.21.15.40:0/362624095 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-07-14T19:16:53.468 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:53.463+0000 7f419effd700 1 --2- 172.21.15.40:0/362624095 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f41a00feb10 0x7f41a0194810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:16:53.468 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:53.463+0000 7f419f7fe700 1 --2- 172.21.15.40:0/362624095 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f41a0100690 0x7f41a019bdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:16:53.468 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:53.463+0000 7f419e7fc700 1 -- 172.21.15.40:0/362624095 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f41a00feb10 msgr2=0x7f41a0194810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:16:53.468 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:53.463+0000 7f419e7fc700 1 --2- 172.21.15.40:0/362624095 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f41a00feb10 0x7f41a0194810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:53.468 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:53.463+0000 7f419e7fc700 1 -- 172.21.15.40:0/362624095 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f41a0100690 msgr2=0x7f41a019bdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:16:53.468 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:53.463+0000 7f419e7fc700 1 --2- 172.21.15.40:0/362624095 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f41a0100690 0x7f41a019bdc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:53.469 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:53.463+0000 7f419e7fc700 1 -- 172.21.15.40:0/362624095 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41940095f0 con 0x7f41a00ffd00 2023-07-14T19:16:53.469 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:53.463+0000 7f419effd700 1 --2- 172.21.15.40:0/362624095 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f41a00feb10 0x7f41a0194810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T19:16:53.469 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:53.463+0000 7f419e7fc700 1 --2- 172.21.15.40:0/362624095 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f41a00ffd00 0x7f41a0194d50 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f419000c9b0 tx=0x7f419000ccc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:16:53.469 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:53.463+0000 7f41a480d700 1 -- 172.21.15.40:0/362624095 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4190007b00 con 0x7f41a00ffd00 2023-07-14T19:16:53.469 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:53.463+0000 7f41a580f700 1 -- 172.21.15.40:0/362624095 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f41a019c940 con 0x7f41a00ffd00 2023-07-14T19:16:53.469 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:53.463+0000 7f41a480d700 1 -- 172.21.15.40:0/362624095 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f4190007c60 con 0x7f41a00ffd00 2023-07-14T19:16:53.469 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:53.463+0000 7f41a480d700 1 -- 172.21.15.40:0/362624095 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4190018660 con 0x7f41a00ffd00 2023-07-14T19:16:53.469 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:53.464+0000 7f41a580f700 1 -- 172.21.15.40:0/362624095 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f41a019ce50 con 0x7f41a00ffd00 2023-07-14T19:16:53.470 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:53.464+0000 7f41a580f700 1 -- 172.21.15.40:0/362624095 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f41a010a2e0 con 0x7f41a00ffd00 2023-07-14T19:16:53.470 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:53.465+0000 7f41a480d700 1 -- 172.21.15.40:0/362624095 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7f419000f450 con 0x7f41a00ffd00 2023-07-14T19:16:53.470 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:53.466+0000 7f41a480d700 1 --2- 172.21.15.40:0/362624095 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f41880a11e0 0x7f41880a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:16:53.470 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:53.466+0000 7f419effd700 1 --2- 172.21.15.40:0/362624095 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f41880a11e0 0x7f41880a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:16:53.471 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:53.466+0000 7f41a480d700 1 -- 172.21.15.40:0/362624095 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 8269+0+0 (secure 0 0 0) 0x7f41900ccab0 con 0x7f41a00ffd00 2023-07-14T19:16:53.471 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:53.467+0000 7f419effd700 1 --2- 172.21.15.40:0/362624095 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f41880a11e0 0x7f41880a3690 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f4194006010 tx=0x7f419400b540 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:16:53.471 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:53.470+0000 7f41a480d700 1 -- 172.21.15.40:0/362624095 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f4190096080 con 0x7f41a00ffd00 2023-07-14T19:16:53.670 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:53.670+0000 7f41a580f700 1 -- 172.21.15.40:0/362624095 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f41a01059c0 con 0x7f41a00ffd00 2023-07-14T19:16:53.671 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:53.670+0000 7f41a480d700 1 -- 172.21.15.40:0/362624095 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v83) v1 ==== 100+0+11 (secure 0 0 0) 0x7f41900996a0 con 0x7f41a00ffd00 2023-07-14T19:16:53.671 INFO:teuthology.orchestra.run.smithi040.stdout:pg_num: 64 2023-07-14T19:16:53.673 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:53.672+0000 7f41a580f700 1 -- 172.21.15.40:0/362624095 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f41880a11e0 msgr2=0x7f41880a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:16:53.673 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:53.672+0000 7f41a580f700 1 --2- 172.21.15.40:0/362624095 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f41880a11e0 0x7f41880a3690 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f4194006010 tx=0x7f419400b540 comp rx=0 tx=0).stop 2023-07-14T19:16:53.673 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:53.672+0000 7f41a580f700 1 -- 172.21.15.40:0/362624095 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f41a00ffd00 msgr2=0x7f41a0194d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:16:53.673 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:53.672+0000 7f41a580f700 1 --2- 172.21.15.40:0/362624095 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f41a00ffd00 0x7f41a0194d50 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f419000c9b0 tx=0x7f419000ccc0 comp rx=0 tx=0).stop 2023-07-14T19:16:53.674 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:53.672+0000 7f41a580f700 1 -- 172.21.15.40:0/362624095 shutdown_connections 2023-07-14T19:16:53.674 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:53.672+0000 7f41a580f700 1 --2- 172.21.15.40:0/362624095 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f41880a11e0 0x7f41880a3690 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:53.674 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:53.672+0000 7f41a580f700 1 --2- 172.21.15.40:0/362624095 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f41a00feb10 0x7f41a0194810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:53.674 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:53.672+0000 7f41a580f700 1 --2- 172.21.15.40:0/362624095 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f41a0100690 0x7f41a019bdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:53.675 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:53.672+0000 7f41a580f700 1 --2- 172.21.15.40:0/362624095 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f41a00ffd00 0x7f41a0194d50 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:53.675 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:53.672+0000 7f41a580f700 1 -- 172.21.15.40:0/362624095 >> 172.21.15.40:0/362624095 conn(0x7f41a00fa130 msgr2=0x7f41a00fc390 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:16:53.675 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:53.672+0000 7f41a580f700 1 -- 172.21.15.40:0/362624095 shutdown_connections 2023-07-14T19:16:53.675 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:53.672+0000 7f41a580f700 1 -- 172.21.15.40:0/362624095 wait complete. 2023-07-14T19:16:53.685 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-07-14T19:16:53.814 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:53.811+0000 7fce035d5700 1 -- 172.21.15.40:0/1063463857 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcdfc105900 msgr2=0x7fcdfc107d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:16:53.814 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:53.811+0000 7fce035d5700 1 --2- 172.21.15.40:0/1063463857 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcdfc105900 0x7fcdfc107d60 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7fcdf400b0a0 tx=0x7fcdf400b3b0 comp rx=0 tx=0).stop 2023-07-14T19:16:53.815 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:53.811+0000 7fce035d5700 1 -- 172.21.15.40:0/1063463857 shutdown_connections 2023-07-14T19:16:53.815 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:53.811+0000 7fce035d5700 1 --2- 172.21.15.40:0/1063463857 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcdfc105900 0x7fcdfc107d60 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:53.815 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:53.811+0000 7fce035d5700 1 --2- 172.21.15.40:0/1063463857 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fcdfc1006c0 0x7fcdfc102aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:53.815 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:53.811+0000 7fce035d5700 1 --2- 172.21.15.40:0/1063463857 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fcdfc102fe0 0x7fcdfc1053c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:53.815 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:53.811+0000 7fce035d5700 1 -- 172.21.15.40:0/1063463857 >> 172.21.15.40:0/1063463857 conn(0x7fcdfc0fa160 msgr2=0x7fcdfc0fc590 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:16:53.815 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:53.812+0000 7fce035d5700 1 -- 172.21.15.40:0/1063463857 shutdown_connections 2023-07-14T19:16:53.816 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:53.812+0000 7fce035d5700 1 -- 172.21.15.40:0/1063463857 wait complete. 2023-07-14T19:16:53.816 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:53.812+0000 7fce035d5700 1 Processor -- start 2023-07-14T19:16:53.816 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:53.812+0000 7fce035d5700 1 -- start start 2023-07-14T19:16:53.816 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:53.812+0000 7fce035d5700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fcdfc1006c0 0x7fcdfc195990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:16:53.816 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:53.812+0000 7fce035d5700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcdfc102fe0 0x7fcdfc195ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:16:53.817 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:53.812+0000 7fce035d5700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fcdfc105900 0x7fcdfc19cf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:16:53.817 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:53.812+0000 7fce035d5700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcdfc19d510 con 0x7fcdfc102fe0 2023-07-14T19:16:53.817 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:53.812+0000 7fce035d5700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcdfc19d650 con 0x7fcdfc1006c0 2023-07-14T19:16:53.817 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:53.812+0000 7fce035d5700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcdfc19d7e0 con 0x7fcdfc105900 2023-07-14T19:16:53.817 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:53.812+0000 7fce00b70700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcdfc102fe0 0x7fcdfc195ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:16:53.817 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:53.813+0000 7fce00b70700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcdfc102fe0 0x7fcdfc195ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:33586/0 (socket says 172.21.15.40:33586) 2023-07-14T19:16:53.817 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:53.813+0000 7fce00b70700 1 -- 172.21.15.40:0/249206260 learned_addr learned my addr 172.21.15.40:0/249206260 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-07-14T19:16:53.818 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:53.813+0000 7fce01b72700 1 --2- 172.21.15.40:0/249206260 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fcdfc105900 0x7fcdfc19cf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:16:53.818 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:53.813+0000 7fce01371700 1 --2- 172.21.15.40:0/249206260 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fcdfc1006c0 0x7fcdfc195990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:16:53.818 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:53.813+0000 7fce00b70700 1 -- 172.21.15.40:0/249206260 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fcdfc105900 msgr2=0x7fcdfc19cf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:16:53.818 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:53.813+0000 7fce00b70700 1 --2- 172.21.15.40:0/249206260 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fcdfc105900 0x7fcdfc19cf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:53.818 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:53.813+0000 7fce00b70700 1 -- 172.21.15.40:0/249206260 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fcdfc1006c0 msgr2=0x7fcdfc195990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:16:53.818 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:53.813+0000 7fce00b70700 1 --2- 172.21.15.40:0/249206260 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fcdfc1006c0 0x7fcdfc195990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:53.819 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:53.813+0000 7fce00b70700 1 -- 172.21.15.40:0/249206260 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcdf4009d00 con 0x7fcdfc102fe0 2023-07-14T19:16:53.819 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:53.813+0000 7fce01371700 1 --2- 172.21.15.40:0/249206260 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fcdfc1006c0 0x7fcdfc195990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T19:16:53.819 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:53.813+0000 7fce00b70700 1 --2- 172.21.15.40:0/249206260 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcdfc102fe0 0x7fcdfc195ed0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7fcdec00d950 tx=0x7fcdec00dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:16:53.819 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:53.813+0000 7fcdf27fc700 1 -- 172.21.15.40:0/249206260 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcdec009940 con 0x7fcdfc102fe0 2023-07-14T19:16:53.819 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:53.813+0000 7fcdf27fc700 1 -- 172.21.15.40:0/249206260 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fcdec010460 con 0x7fcdfc102fe0 2023-07-14T19:16:53.819 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:53.813+0000 7fce035d5700 1 -- 172.21.15.40:0/249206260 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcdfc19dac0 con 0x7fcdfc102fe0 2023-07-14T19:16:53.820 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:53.813+0000 7fcdf27fc700 1 -- 172.21.15.40:0/249206260 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcdec00f620 con 0x7fcdfc102fe0 2023-07-14T19:16:53.820 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:53.813+0000 7fce035d5700 1 -- 172.21.15.40:0/249206260 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcdfc19dfd0 con 0x7fcdfc102fe0 2023-07-14T19:16:53.821 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:53.814+0000 7fce035d5700 1 -- 172.21.15.40:0/249206260 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcdfc0fbcd0 con 0x7fcdfc102fe0 2023-07-14T19:16:53.821 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:53.815+0000 7fcdf27fc700 1 -- 172.21.15.40:0/249206260 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7fcdec0108d0 con 0x7fcdfc102fe0 2023-07-14T19:16:53.821 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:53.816+0000 7fcdf27fc700 1 --2- 172.21.15.40:0/249206260 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7fcde80a1220 0x7fcde80a36d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:16:53.822 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:53.816+0000 7fcdf27fc700 1 -- 172.21.15.40:0/249206260 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 8269+0+0 (secure 0 0 0) 0x7fcdec0cc950 con 0x7fcdfc102fe0 2023-07-14T19:16:53.822 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:53.816+0000 7fce01371700 1 --2- 172.21.15.40:0/249206260 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7fcde80a1220 0x7fcde80a36d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:16:53.822 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:53.816+0000 7fce01371700 1 --2- 172.21.15.40:0/249206260 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7fcde80a1220 0x7fcde80a36d0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fcdf8005950 tx=0x7fcdf80058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:16:53.822 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:53.819+0000 7fcdf27fc700 1 -- 172.21.15.40:0/249206260 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fcdec095f20 con 0x7fcdfc102fe0 2023-07-14T19:16:53.971 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:16:53 smithi040 ceph-mon[96518]: osdmap e83: 12 total, 12 up, 12 in 2023-07-14T19:16:53.971 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:16:53 smithi040 ceph-mon[96518]: from='client.? 172.21.15.40:0/2091946707' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-07-14T19:16:53.971 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:16:53 smithi040 ceph-mon[96518]: from='client.? 172.21.15.40:0/1783321818' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-07-14T19:16:53.971 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:16:53 smithi040 ceph-mon[96518]: from='client.? 172.21.15.40:0/362624095' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-07-14T19:16:54.064 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:54.063+0000 7fce035d5700 1 -- 172.21.15.40:0/249206260 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcdfc02c8e0 con 0x7fcdfc102fe0 2023-07-14T19:16:54.065 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:54.064+0000 7fcdf27fc700 1 -- 172.21.15.40:0/249206260 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) v1 ==== 94+0+10877 (secure 0 0 0) 0x7fcdec016090 con 0x7fcdfc102fe0 2023-07-14T19:16:54.066 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-07-14T19:16:54.066 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14523,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6835/501143718","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6834","nonce":501143718},{"type":"v1","addr":"172.21.15.40:6835","nonce":501143718}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14535,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6837/2634476117","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6836","nonce":2634476117},{"type":"v1","addr":"172.21.15.40:6837","nonce":2634476117}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":26,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2023-07-14T19:15:57.650710+0000","modified":"2023-07-14T19:16:05.624568+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24539,"mds_1":24508,"mds_2":24506,"mds_3":24490,"mds_4":24482},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14586":{"gid":14586,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.40:6839/959937545","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6838","nonce":959937545},{"type":"v1","addr":"172.21.15.40:6839","nonce":959937545}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_14598":{"gid":14598,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.40:6841/1590486077","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6840","nonce":1590486077},{"type":"v1","addr":"172.21.15.40:6841","nonce":1590486077}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24433":{"gid":24433,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.191:6833/31509862","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":31509862},{"type":"v1","addr":"172.21.15.191:6833","nonce":31509862}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24452":{"gid":24452,"name":"k","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.130:6833/3677452688","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6832","nonce":3677452688},{"type":"v1","addr":"172.21.15.130:6833","nonce":3677452688}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24478":{"gid":24478,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.191:6835/3712674634","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":3712674634},{"type":"v1","addr":"172.21.15.191:6835","nonce":3712674634}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24482":{"gid":24482,"name":"e","rank":4,"incarnation":24,"state":"up:active","state_seq":8,"addr":"172.21.15.130:6835/3088490808","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6834","nonce":3088490808},{"type":"v1","addr":"172.21.15.130:6835","nonce":3088490808}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24490":{"gid":24490,"name":"c","rank":3,"incarnation":22,"state":"up:active","state_seq":7,"addr":"172.21.15.191:6837/2406892975","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6836","nonce":2406892975},{"type":"v1","addr":"172.21.15.191:6837","nonce":2406892975}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24506":{"gid":24506,"name":"h","rank":2,"incarnation":20,"state":"up:active","state_seq":6,"addr":"172.21.15.130:6837/2604019369","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6836","nonce":2604019369},{"type":"v1","addr":"172.21.15.130:6837","nonce":2604019369}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24508":{"gid":24508,"name":"i","rank":1,"incarnation":18,"state":"up:active","state_seq":5,"addr":"172.21.15.191:6839/1977507302","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":1977507302},{"type":"v1","addr":"172.21.15.191:6839","nonce":1977507302}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24539":{"gid":24539,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.130:6839/2145479535","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6838","nonce":2145479535},{"type":"v1","addr":"172.21.15.130:6839","nonce":2145479535}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-07-14T19:16:54.068 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:54.066+0000 7fce035d5700 1 -- 172.21.15.40:0/249206260 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7fcde80a1220 msgr2=0x7fcde80a36d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:16:54.068 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:54.066+0000 7fce035d5700 1 --2- 172.21.15.40:0/249206260 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7fcde80a1220 0x7fcde80a36d0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fcdf8005950 tx=0x7fcdf80058e0 comp rx=0 tx=0).stop 2023-07-14T19:16:54.069 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:54.066+0000 7fce035d5700 1 -- 172.21.15.40:0/249206260 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcdfc102fe0 msgr2=0x7fcdfc195ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:16:54.069 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:54.066+0000 7fce035d5700 1 --2- 172.21.15.40:0/249206260 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcdfc102fe0 0x7fcdfc195ed0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7fcdec00d950 tx=0x7fcdec00dc60 comp rx=0 tx=0).stop 2023-07-14T19:16:54.069 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:54.066+0000 7fce035d5700 1 -- 172.21.15.40:0/249206260 shutdown_connections 2023-07-14T19:16:54.069 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:54.066+0000 7fce035d5700 1 --2- 172.21.15.40:0/249206260 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7fcde80a1220 0x7fcde80a36d0 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:54.069 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:54.066+0000 7fce035d5700 1 --2- 172.21.15.40:0/249206260 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fcdfc105900 0x7fcdfc19cf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:54.069 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:54.066+0000 7fce035d5700 1 --2- 172.21.15.40:0/249206260 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fcdfc1006c0 0x7fcdfc195990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:54.070 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:54.066+0000 7fce035d5700 1 --2- 172.21.15.40:0/249206260 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcdfc102fe0 0x7fcdfc195ed0 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:54.070 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:54.066+0000 7fce035d5700 1 -- 172.21.15.40:0/249206260 >> 172.21.15.40:0/249206260 conn(0x7fcdfc0fa160 msgr2=0x7fcdfc101320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:16:54.070 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:54.066+0000 7fce035d5700 1 -- 172.21.15.40:0/249206260 shutdown_connections 2023-07-14T19:16:54.070 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:54.066+0000 7fce035d5700 1 -- 172.21.15.40:0/249206260 wait complete. 2023-07-14T19:16:54.070 INFO:teuthology.orchestra.run.smithi040.stderr:dumped fsmap epoch 26 2023-07-14T19:16:54.080 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-07-14T19:16:54.208 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:54.205+0000 7fa87afd4700 1 -- 172.21.15.40:0/235820645 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa874100710 msgr2=0x7fa874102af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:16:54.208 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:54.205+0000 7fa87afd4700 1 --2- 172.21.15.40:0/235820645 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa874100710 0x7fa874102af0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7fa868009990 tx=0x7fa868009ca0 comp rx=0 tx=0).stop 2023-07-14T19:16:54.208 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:54.206+0000 7fa87afd4700 1 -- 172.21.15.40:0/235820645 shutdown_connections 2023-07-14T19:16:54.208 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:54.206+0000 7fa87afd4700 1 --2- 172.21.15.40:0/235820645 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa874105950 0x7fa874107d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:54.209 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:54.206+0000 7fa87afd4700 1 --2- 172.21.15.40:0/235820645 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa874100710 0x7fa874102af0 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:54.209 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:54.206+0000 7fa87afd4700 1 --2- 172.21.15.40:0/235820645 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa874103030 0x7fa874105410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:54.209 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:54.206+0000 7fa87afd4700 1 -- 172.21.15.40:0/235820645 >> 172.21.15.40:0/235820645 conn(0x7fa8740fa150 msgr2=0x7fa8740fc580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:16:54.209 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:54.206+0000 7fa87afd4700 1 -- 172.21.15.40:0/235820645 shutdown_connections 2023-07-14T19:16:54.209 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:54.206+0000 7fa87afd4700 1 -- 172.21.15.40:0/235820645 wait complete. 2023-07-14T19:16:54.209 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:54.206+0000 7fa87afd4700 1 Processor -- start 2023-07-14T19:16:54.209 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:54.206+0000 7fa87afd4700 1 -- start start 2023-07-14T19:16:54.210 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:54.206+0000 7fa87afd4700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa874100710 0x7fa8741958f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:16:54.210 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:54.206+0000 7fa87afd4700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa874103030 0x7fa874195e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:16:54.210 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:54.206+0000 7fa87afd4700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa874105950 0x7fa87419cea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:16:54.210 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:54.206+0000 7fa87afd4700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa87419d470 con 0x7fa874103030 2023-07-14T19:16:54.210 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:54.206+0000 7fa87afd4700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa87419d5b0 con 0x7fa874100710 2023-07-14T19:16:54.210 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:54.206+0000 7fa87afd4700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa87419d740 con 0x7fa874105950 2023-07-14T19:16:54.210 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:54.206+0000 7fa873fff700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa874103030 0x7fa874195e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:16:54.211 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:54.206+0000 7fa873fff700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa874103030 0x7fa874195e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:33600/0 (socket says 172.21.15.40:33600) 2023-07-14T19:16:54.211 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:54.207+0000 7fa873fff700 1 -- 172.21.15.40:0/2006297146 learned_addr learned my addr 172.21.15.40:0/2006297146 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-07-14T19:16:54.211 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:54.207+0000 7fa878d70700 1 --2- 172.21.15.40:0/2006297146 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa874100710 0x7fa8741958f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:16:54.211 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:54.207+0000 7fa879571700 1 --2- 172.21.15.40:0/2006297146 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa874105950 0x7fa87419cea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:16:54.211 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:54.207+0000 7fa873fff700 1 -- 172.21.15.40:0/2006297146 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa874105950 msgr2=0x7fa87419cea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:16:54.211 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:54.207+0000 7fa873fff700 1 --2- 172.21.15.40:0/2006297146 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa874105950 0x7fa87419cea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:54.211 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:54.207+0000 7fa873fff700 1 -- 172.21.15.40:0/2006297146 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa874100710 msgr2=0x7fa8741958f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:16:54.211 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:54.207+0000 7fa873fff700 1 --2- 172.21.15.40:0/2006297146 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa874100710 0x7fa8741958f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:54.212 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:54.207+0000 7fa873fff700 1 -- 172.21.15.40:0/2006297146 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8680095f0 con 0x7fa874103030 2023-07-14T19:16:54.212 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:54.207+0000 7fa873fff700 1 --2- 172.21.15.40:0/2006297146 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa874103030 0x7fa874195e30 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7fa86000e9a0 tx=0x7fa86000ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:16:54.212 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:54.207+0000 7fa871ffb700 1 -- 172.21.15.40:0/2006297146 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa860009790 con 0x7fa874103030 2023-07-14T19:16:54.212 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:54.207+0000 7fa87afd4700 1 -- 172.21.15.40:0/2006297146 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa87419da20 con 0x7fa874103030 2023-07-14T19:16:54.212 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:54.207+0000 7fa871ffb700 1 -- 172.21.15.40:0/2006297146 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa86000cbd0 con 0x7fa874103030 2023-07-14T19:16:54.212 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:54.207+0000 7fa871ffb700 1 -- 172.21.15.40:0/2006297146 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa860018680 con 0x7fa874103030 2023-07-14T19:16:54.212 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:54.207+0000 7fa87afd4700 1 -- 172.21.15.40:0/2006297146 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa87419df80 con 0x7fa874103030 2023-07-14T19:16:54.213 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:54.208+0000 7fa87afd4700 1 -- 172.21.15.40:0/2006297146 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa8740fbcc0 con 0x7fa874103030 2023-07-14T19:16:54.213 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:54.209+0000 7fa871ffb700 1 -- 172.21.15.40:0/2006297146 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7fa86000cd40 con 0x7fa874103030 2023-07-14T19:16:54.214 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:54.210+0000 7fa871ffb700 1 --2- 172.21.15.40:0/2006297146 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7fa8640a1220 0x7fa8640a36d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:16:54.214 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:54.210+0000 7fa878d70700 1 --2- 172.21.15.40:0/2006297146 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7fa8640a1220 0x7fa8640a36d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:16:54.214 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:54.210+0000 7fa871ffb700 1 -- 172.21.15.40:0/2006297146 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 8269+0+0 (secure 0 0 0) 0x7fa860014070 con 0x7fa874103030 2023-07-14T19:16:54.214 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:54.211+0000 7fa878d70700 1 --2- 172.21.15.40:0/2006297146 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7fa8640a1220 0x7fa8640a36d0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fa86800b2e0 tx=0x7fa868005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:16:54.214 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:54.214+0000 7fa871ffb700 1 -- 172.21.15.40:0/2006297146 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fa8600965c0 con 0x7fa874103030 2023-07-14T19:16:54.221 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:16:53 smithi191 ceph-mon[98311]: osdmap e83: 12 total, 12 up, 12 in 2023-07-14T19:16:54.222 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:16:53 smithi191 ceph-mon[98311]: from='client.? 172.21.15.40:0/2091946707' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-07-14T19:16:54.222 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:16:53 smithi191 ceph-mon[98311]: from='client.? 172.21.15.40:0/1783321818' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-07-14T19:16:54.222 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:16:53 smithi191 ceph-mon[98311]: from='client.? 172.21.15.40:0/362624095' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-07-14T19:16:54.242 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:16:53 smithi130 ceph-mon[97832]: osdmap e83: 12 total, 12 up, 12 in 2023-07-14T19:16:54.242 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:16:53 smithi130 ceph-mon[97832]: from='client.? 172.21.15.40:0/2091946707' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-07-14T19:16:54.242 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:16:53 smithi130 ceph-mon[97832]: from='client.? 172.21.15.40:0/1783321818' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-07-14T19:16:54.242 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:16:53 smithi130 ceph-mon[97832]: from='client.? 172.21.15.40:0/362624095' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-07-14T19:16:54.416 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:54.415+0000 7fa87afd4700 1 -- 172.21.15.40:0/2006297146 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa87402caf0 con 0x7fa874103030 2023-07-14T19:16:54.418 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:54.417+0000 7fa871ffb700 1 -- 172.21.15.40:0/2006297146 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v83) v1 ==== 74+0+22647 (secure 0 0 0) 0x7fa860099be0 con 0x7fa874103030 2023-07-14T19:16:54.418 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-07-14T19:16:54.419 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":83,"fsid":"adc52f04-2279-11ee-9b33-001a4aab830c","created":"2023-07-14T19:08:56.893730+0000","modified":"2023-07-14T19:16:52.763512+0000","last_up_change":"2023-07-14T19:15:16.632979+0000","last_in_change":"2023-07-14T19:14:58.961702+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-07-14T19:12:03.271333+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-07-14T19:15:56.073464+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-07-14T19:15:57.018726+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"0d79356a-95d2-49ad-857b-2092ae9f7e13","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":3599703125},{"type":"v1","addr":"172.21.15.40:6803","nonce":3599703125}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":3599703125},{"type":"v1","addr":"172.21.15.40:6805","nonce":3599703125}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":3599703125},{"type":"v1","addr":"172.21.15.40:6809","nonce":3599703125}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":3599703125},{"type":"v1","addr":"172.21.15.40:6807","nonce":3599703125}]},"public_addr":"172.21.15.40:6803/3599703125","cluster_addr":"172.21.15.40:6805/3599703125","heartbeat_back_addr":"172.21.15.40:6809/3599703125","heartbeat_front_addr":"172.21.15.40:6807/3599703125","state":["exists","up"]},{"osd":1,"uuid":"40f8a3d1-a5e4-473c-b9f2-7600ec22be0d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6800","nonce":2512760702},{"type":"v1","addr":"172.21.15.130:6801","nonce":2512760702}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6802","nonce":2512760702},{"type":"v1","addr":"172.21.15.130:6803","nonce":2512760702}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6806","nonce":2512760702},{"type":"v1","addr":"172.21.15.130:6807","nonce":2512760702}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6804","nonce":2512760702},{"type":"v1","addr":"172.21.15.130:6805","nonce":2512760702}]},"public_addr":"172.21.15.130:6801/2512760702","cluster_addr":"172.21.15.130:6803/2512760702","heartbeat_back_addr":"172.21.15.130:6807/2512760702","heartbeat_front_addr":"172.21.15.130:6805/2512760702","state":["exists","up"]},{"osd":2,"uuid":"5a157891-cb7a-4ea6-9239-f48a7c67c003","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6800","nonce":589569135},{"type":"v1","addr":"172.21.15.191:6801","nonce":589569135}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6802","nonce":589569135},{"type":"v1","addr":"172.21.15.191:6803","nonce":589569135}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6806","nonce":589569135},{"type":"v1","addr":"172.21.15.191:6807","nonce":589569135}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6804","nonce":589569135},{"type":"v1","addr":"172.21.15.191:6805","nonce":589569135}]},"public_addr":"172.21.15.191:6801/589569135","cluster_addr":"172.21.15.191:6803/589569135","heartbeat_back_addr":"172.21.15.191:6807/589569135","heartbeat_front_addr":"172.21.15.191:6805/589569135","state":["exists","up"]},{"osd":3,"uuid":"8020e7dc-ac40-487f-857a-8da40bccb459","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":996287860},{"type":"v1","addr":"172.21.15.40:6811","nonce":996287860}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6812","nonce":996287860},{"type":"v1","addr":"172.21.15.40:6813","nonce":996287860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6816","nonce":996287860},{"type":"v1","addr":"172.21.15.40:6817","nonce":996287860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":996287860},{"type":"v1","addr":"172.21.15.40:6815","nonce":996287860}]},"public_addr":"172.21.15.40:6811/996287860","cluster_addr":"172.21.15.40:6813/996287860","heartbeat_back_addr":"172.21.15.40:6817/996287860","heartbeat_front_addr":"172.21.15.40:6815/996287860","state":["exists","up"]},{"osd":4,"uuid":"c5501b35-f62d-482d-8d22-115c88f0298e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6808","nonce":3507448720},{"type":"v1","addr":"172.21.15.130:6809","nonce":3507448720}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6810","nonce":3507448720},{"type":"v1","addr":"172.21.15.130:6811","nonce":3507448720}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6814","nonce":3507448720},{"type":"v1","addr":"172.21.15.130:6815","nonce":3507448720}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6812","nonce":3507448720},{"type":"v1","addr":"172.21.15.130:6813","nonce":3507448720}]},"public_addr":"172.21.15.130:6809/3507448720","cluster_addr":"172.21.15.130:6811/3507448720","heartbeat_back_addr":"172.21.15.130:6815/3507448720","heartbeat_front_addr":"172.21.15.130:6813/3507448720","state":["exists","up"]},{"osd":5,"uuid":"7770b6d6-5874-493a-986a-84db482cee42","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6808","nonce":1781944663},{"type":"v1","addr":"172.21.15.191:6809","nonce":1781944663}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6810","nonce":1781944663},{"type":"v1","addr":"172.21.15.191:6811","nonce":1781944663}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6814","nonce":1781944663},{"type":"v1","addr":"172.21.15.191:6815","nonce":1781944663}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6812","nonce":1781944663},{"type":"v1","addr":"172.21.15.191:6813","nonce":1781944663}]},"public_addr":"172.21.15.191:6809/1781944663","cluster_addr":"172.21.15.191:6811/1781944663","heartbeat_back_addr":"172.21.15.191:6815/1781944663","heartbeat_front_addr":"172.21.15.191:6813/1781944663","state":["exists","up"]},{"osd":6,"uuid":"ef65ad00-dfee-4300-9961-dfc8573aab06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6818","nonce":693186773},{"type":"v1","addr":"172.21.15.40:6819","nonce":693186773}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6820","nonce":693186773},{"type":"v1","addr":"172.21.15.40:6821","nonce":693186773}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6824","nonce":693186773},{"type":"v1","addr":"172.21.15.40:6825","nonce":693186773}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6822","nonce":693186773},{"type":"v1","addr":"172.21.15.40:6823","nonce":693186773}]},"public_addr":"172.21.15.40:6819/693186773","cluster_addr":"172.21.15.40:6821/693186773","heartbeat_back_addr":"172.21.15.40:6825/693186773","heartbeat_front_addr":"172.21.15.40:6823/693186773","state":["exists","up"]},{"osd":7,"uuid":"1700fbfa-cbfc-4f76-97be-727a3b100740","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":50,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6816","nonce":2384019528},{"type":"v1","addr":"172.21.15.130:6817","nonce":2384019528}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6818","nonce":2384019528},{"type":"v1","addr":"172.21.15.130:6819","nonce":2384019528}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6822","nonce":2384019528},{"type":"v1","addr":"172.21.15.130:6823","nonce":2384019528}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6820","nonce":2384019528},{"type":"v1","addr":"172.21.15.130:6821","nonce":2384019528}]},"public_addr":"172.21.15.130:6817/2384019528","cluster_addr":"172.21.15.130:6819/2384019528","heartbeat_back_addr":"172.21.15.130:6823/2384019528","heartbeat_front_addr":"172.21.15.130:6821/2384019528","state":["exists","up"]},{"osd":8,"uuid":"b76d4033-c1e2-457b-8571-c8a49fe443f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":57,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6816","nonce":2386818417},{"type":"v1","addr":"172.21.15.191:6817","nonce":2386818417}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6818","nonce":2386818417},{"type":"v1","addr":"172.21.15.191:6819","nonce":2386818417}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6822","nonce":2386818417},{"type":"v1","addr":"172.21.15.191:6823","nonce":2386818417}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6820","nonce":2386818417},{"type":"v1","addr":"172.21.15.191:6821","nonce":2386818417}]},"public_addr":"172.21.15.191:6817/2386818417","cluster_addr":"172.21.15.191:6819/2386818417","heartbeat_back_addr":"172.21.15.191:6823/2386818417","heartbeat_front_addr":"172.21.15.191:6821/2386818417","state":["exists","up"]},{"osd":9,"uuid":"df31bf2f-e30e-41b6-b2b9-0e26a07a88bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":64,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6826","nonce":2228380263},{"type":"v1","addr":"172.21.15.40:6827","nonce":2228380263}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6828","nonce":2228380263},{"type":"v1","addr":"172.21.15.40:6829","nonce":2228380263}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":2228380263},{"type":"v1","addr":"172.21.15.40:6833","nonce":2228380263}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6830","nonce":2228380263},{"type":"v1","addr":"172.21.15.40:6831","nonce":2228380263}]},"public_addr":"172.21.15.40:6827/2228380263","cluster_addr":"172.21.15.40:6829/2228380263","heartbeat_back_addr":"172.21.15.40:6833/2228380263","heartbeat_front_addr":"172.21.15.40:6831/2228380263","state":["exists","up"]},{"osd":10,"uuid":"4a93d621-49ee-4ad7-9e49-750614faf912","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6824","nonce":2417067798},{"type":"v1","addr":"172.21.15.130:6825","nonce":2417067798}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6826","nonce":2417067798},{"type":"v1","addr":"172.21.15.130:6827","nonce":2417067798}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6830","nonce":2417067798},{"type":"v1","addr":"172.21.15.130:6831","nonce":2417067798}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6828","nonce":2417067798},{"type":"v1","addr":"172.21.15.130:6829","nonce":2417067798}]},"public_addr":"172.21.15.130:6825/2417067798","cluster_addr":"172.21.15.130:6827/2417067798","heartbeat_back_addr":"172.21.15.130:6831/2417067798","heartbeat_front_addr":"172.21.15.130:6829/2417067798","state":["exists","up"]},{"osd":11,"uuid":"f43a2a6f-82f3-42ac-b77c-4f1b34da6918","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":76,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6824","nonce":2886059446},{"type":"v1","addr":"172.21.15.191:6825","nonce":2886059446}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6826","nonce":2886059446},{"type":"v1","addr":"172.21.15.191:6827","nonce":2886059446}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6830","nonce":2886059446},{"type":"v1","addr":"172.21.15.191:6831","nonce":2886059446}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6828","nonce":2886059446},{"type":"v1","addr":"172.21.15.191:6829","nonce":2886059446}]},"public_addr":"172.21.15.191:6825/2886059446","cluster_addr":"172.21.15.191:6827/2886059446","heartbeat_back_addr":"172.21.15.191:6831/2886059446","heartbeat_front_addr":"172.21.15.191:6829/2886059446","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:11:18.320495+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:11:39.901199+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:12:00.869042+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:12:23.292742+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:12:44.587019+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:13:06.494893+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:13:28.378374+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:13:50.033491+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:14:10.937062+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:14:32.358498+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:14:53.666584+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:15:14.837566+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"2.b","mappings":[{"from":8,"to":11}]},{"pgid":"2.c","mappings":[{"from":2,"to":4}]},{"pgid":"2.1f","mappings":[{"from":8,"to":1}]},{"pgid":"3.e","mappings":[{"from":4,"to":7}]},{"pgid":"3.17","mappings":[{"from":4,"to":8}]},{"pgid":"3.1a","mappings":[{"from":5,"to":6}]},{"pgid":"3.20","mappings":[{"from":5,"to":9}]},{"pgid":"3.27","mappings":[{"from":3,"to":9}]},{"pgid":"3.2c","mappings":[{"from":4,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.40:6801/3698757264":"2023-07-15T19:09:50.732458+0000","172.21.15.40:6800/3698757264":"2023-07-15T19:09:50.732458+0000","172.21.15.40:6800/2569015667":"2023-07-15T19:09:24.441543+0000","172.21.15.40:0/3653961003":"2023-07-15T19:09:24.441543+0000","172.21.15.40:0/3568861680":"2023-07-15T19:09:24.441543+0000","172.21.15.40:0/3249773172":"2023-07-15T19:09:50.732458+0000","172.21.15.40:0/4084215480":"2023-07-15T19:09:24.441543+0000","172.21.15.40:0/1163941760":"2023-07-15T19:09:50.732458+0000","172.21.15.40:6801/2569015667":"2023-07-15T19:09:24.441543+0000","172.21.15.40:0/2326639960":"2023-07-15T19:09:50.732458+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-07-14T19:16:54.420 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:54.419+0000 7fa87afd4700 1 -- 172.21.15.40:0/2006297146 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7fa8640a1220 msgr2=0x7fa8640a36d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:16:54.420 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:54.419+0000 7fa87afd4700 1 --2- 172.21.15.40:0/2006297146 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7fa8640a1220 0x7fa8640a36d0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fa86800b2e0 tx=0x7fa868005fb0 comp rx=0 tx=0).stop 2023-07-14T19:16:54.420 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:54.419+0000 7fa87afd4700 1 -- 172.21.15.40:0/2006297146 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa874103030 msgr2=0x7fa874195e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:16:54.421 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:54.419+0000 7fa87afd4700 1 --2- 172.21.15.40:0/2006297146 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa874103030 0x7fa874195e30 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7fa86000e9a0 tx=0x7fa86000ecb0 comp rx=0 tx=0).stop 2023-07-14T19:16:54.421 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:54.419+0000 7fa87afd4700 1 -- 172.21.15.40:0/2006297146 shutdown_connections 2023-07-14T19:16:54.421 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:54.419+0000 7fa87afd4700 1 --2- 172.21.15.40:0/2006297146 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7fa8640a1220 0x7fa8640a36d0 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:54.421 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:54.419+0000 7fa87afd4700 1 --2- 172.21.15.40:0/2006297146 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa874105950 0x7fa87419cea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:54.421 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:54.419+0000 7fa87afd4700 1 --2- 172.21.15.40:0/2006297146 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa874100710 0x7fa8741958f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:54.421 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:54.419+0000 7fa87afd4700 1 --2- 172.21.15.40:0/2006297146 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa874103030 0x7fa874195e30 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:54.422 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:54.419+0000 7fa87afd4700 1 -- 172.21.15.40:0/2006297146 >> 172.21.15.40:0/2006297146 conn(0x7fa8740fa150 msgr2=0x7fa874101370 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:16:54.422 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:54.419+0000 7fa87afd4700 1 -- 172.21.15.40:0/2006297146 shutdown_connections 2023-07-14T19:16:54.422 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:54.419+0000 7fa87afd4700 1 -- 172.21.15.40:0/2006297146 wait complete. 2023-07-14T19:16:54.433 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-07-14T19:12:03.271333+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-07-14T19:15:56.073464+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '80', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-07-14T19:15:57.018726+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '81', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-07-14T19:16:54.434 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-07-14T19:16:54.563 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:54.562+0000 7f5d1ea25700 1 -- 172.21.15.40:0/846133896 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5d180feb80 msgr2=0x7f5d180fef90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:16:54.563 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:54.562+0000 7f5d1ea25700 1 --2- 172.21.15.40:0/846133896 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5d180feb80 0x7f5d180fef90 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f5d08009990 tx=0x7f5d08009ca0 comp rx=0 tx=0).stop 2023-07-14T19:16:54.563 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:54.562+0000 7f5d1ea25700 1 -- 172.21.15.40:0/846133896 shutdown_connections 2023-07-14T19:16:54.563 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:54.562+0000 7f5d1ea25700 1 --2- 172.21.15.40:0/846133896 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5d18100700 0x7f5d18106fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:54.564 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:54.562+0000 7f5d1ea25700 1 --2- 172.21.15.40:0/846133896 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5d180feb80 0x7f5d180fef90 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:54.564 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:54.562+0000 7f5d1ea25700 1 --2- 172.21.15.40:0/846133896 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5d180ffd70 0x7f5d181001c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:54.564 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:54.562+0000 7f5d1ea25700 1 -- 172.21.15.40:0/846133896 >> 172.21.15.40:0/846133896 conn(0x7f5d180fa1a0 msgr2=0x7f5d180fc5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:16:54.564 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:54.562+0000 7f5d1ea25700 1 -- 172.21.15.40:0/846133896 shutdown_connections 2023-07-14T19:16:54.564 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:54.562+0000 7f5d1ea25700 1 -- 172.21.15.40:0/846133896 wait complete. 2023-07-14T19:16:54.564 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:54.562+0000 7f5d1ea25700 1 Processor -- start 2023-07-14T19:16:54.565 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:54.562+0000 7f5d1ea25700 1 -- start start 2023-07-14T19:16:54.566 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:54.563+0000 7f5d1ea25700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5d180feb80 0x7f5d180694e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:16:54.566 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:54.563+0000 7f5d1ea25700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5d180ffd70 0x7f5d1819a640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:16:54.567 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:54.563+0000 7f5d1ea25700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5d18100700 0x7f5d1819cc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:16:54.567 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:54.563+0000 7f5d1ea25700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d1819d460 con 0x7f5d180ffd70 2023-07-14T19:16:54.567 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:54.563+0000 7f5d1ea25700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d1819d5d0 con 0x7f5d18100700 2023-07-14T19:16:54.567 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:54.563+0000 7f5d1ea25700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d1819d790 con 0x7f5d180feb80 2023-07-14T19:16:54.567 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:54.563+0000 7f5d177fe700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5d180ffd70 0x7f5d1819a640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:16:54.567 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:54.563+0000 7f5d177fe700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5d180ffd70 0x7f5d1819a640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:33614/0 (socket says 172.21.15.40:33614) 2023-07-14T19:16:54.567 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:54.563+0000 7f5d177fe700 1 -- 172.21.15.40:0/1279917645 learned_addr learned my addr 172.21.15.40:0/1279917645 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-07-14T19:16:54.567 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:54.563+0000 7f5d17fff700 1 --2- 172.21.15.40:0/1279917645 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5d180feb80 0x7f5d180694e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:16:54.568 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:54.563+0000 7f5d1cfc2700 1 --2- 172.21.15.40:0/1279917645 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5d18100700 0x7f5d1819cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:16:54.568 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:54.563+0000 7f5d177fe700 1 -- 172.21.15.40:0/1279917645 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5d180feb80 msgr2=0x7f5d180694e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:16:54.568 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:54.563+0000 7f5d177fe700 1 --2- 172.21.15.40:0/1279917645 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5d180feb80 0x7f5d180694e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:54.568 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:54.563+0000 7f5d177fe700 1 -- 172.21.15.40:0/1279917645 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5d18100700 msgr2=0x7f5d1819cc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:16:54.568 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:54.563+0000 7f5d177fe700 1 --2- 172.21.15.40:0/1279917645 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5d18100700 0x7f5d1819cc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:54.568 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:54.563+0000 7f5d177fe700 1 -- 172.21.15.40:0/1279917645 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d080095f0 con 0x7f5d180ffd70 2023-07-14T19:16:54.568 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:54.563+0000 7f5d17fff700 1 --2- 172.21.15.40:0/1279917645 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5d180feb80 0x7f5d180694e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T19:16:54.569 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:54.563+0000 7f5d177fe700 1 --2- 172.21.15.40:0/1279917645 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5d180ffd70 0x7f5d1819a640 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f5d0c00c9b0 tx=0x7f5d0c00cd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:16:54.569 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:54.563+0000 7f5d157fa700 1 -- 172.21.15.40:0/1279917645 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d0c007b00 con 0x7f5d180ffd70 2023-07-14T19:16:54.569 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:54.564+0000 7f5d157fa700 1 -- 172.21.15.40:0/1279917645 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f5d0c007c60 con 0x7f5d180ffd70 2023-07-14T19:16:54.569 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:54.564+0000 7f5d157fa700 1 -- 172.21.15.40:0/1279917645 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d0c018660 con 0x7f5d180ffd70 2023-07-14T19:16:54.569 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:54.564+0000 7f5d1ea25700 1 -- 172.21.15.40:0/1279917645 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5d1819da70 con 0x7f5d180ffd70 2023-07-14T19:16:54.569 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:54.564+0000 7f5d1ea25700 1 -- 172.21.15.40:0/1279917645 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5d1819dfb0 con 0x7f5d180ffd70 2023-07-14T19:16:54.569 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:54.565+0000 7f5d1ea25700 1 -- 172.21.15.40:0/1279917645 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5d1810a360 con 0x7f5d180ffd70 2023-07-14T19:16:54.570 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:54.566+0000 7f5d157fa700 1 -- 172.21.15.40:0/1279917645 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7f5d0c018880 con 0x7f5d180ffd70 2023-07-14T19:16:54.570 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:54.566+0000 7f5d157fa700 1 --2- 172.21.15.40:0/1279917645 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f5d000a11e0 0x7f5d000a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:16:54.571 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:54.566+0000 7f5d157fa700 1 -- 172.21.15.40:0/1279917645 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 8269+0+0 (secure 0 0 0) 0x7f5d0c0ccdf0 con 0x7f5d180ffd70 2023-07-14T19:16:54.571 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:54.566+0000 7f5d17fff700 1 --2- 172.21.15.40:0/1279917645 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f5d000a11e0 0x7f5d000a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:16:54.571 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:54.567+0000 7f5d17fff700 1 --2- 172.21.15.40:0/1279917645 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f5d000a11e0 0x7f5d000a3690 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f5d08006010 tx=0x7f5d0800b540 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:16:54.571 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:54.570+0000 7f5d157fa700 1 -- 172.21.15.40:0/1279917645 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f5d0c096450 con 0x7f5d180ffd70 2023-07-14T19:16:54.768 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:54.768+0000 7f5d1ea25700 1 -- 172.21.15.40:0/1279917645 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f5d18105a00 con 0x7f5d180ffd70 2023-07-14T19:16:54.769 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:54.768+0000 7f5d157fa700 1 -- 172.21.15.40:0/1279917645 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v83) v1 ==== 93+0+10 (secure 0 0 0) 0x7f5d0c099a70 con 0x7f5d180ffd70 2023-07-14T19:16:54.769 INFO:teuthology.orchestra.run.smithi040.stdout:pg_num: 1 2023-07-14T19:16:54.771 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:54.770+0000 7f5d1ea25700 1 -- 172.21.15.40:0/1279917645 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f5d000a11e0 msgr2=0x7f5d000a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:16:54.771 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:54.770+0000 7f5d1ea25700 1 --2- 172.21.15.40:0/1279917645 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f5d000a11e0 0x7f5d000a3690 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f5d08006010 tx=0x7f5d0800b540 comp rx=0 tx=0).stop 2023-07-14T19:16:54.771 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:54.770+0000 7f5d1ea25700 1 -- 172.21.15.40:0/1279917645 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5d180ffd70 msgr2=0x7f5d1819a640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:16:54.772 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:54.770+0000 7f5d1ea25700 1 --2- 172.21.15.40:0/1279917645 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5d180ffd70 0x7f5d1819a640 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f5d0c00c9b0 tx=0x7f5d0c00cd70 comp rx=0 tx=0).stop 2023-07-14T19:16:54.772 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:54.770+0000 7f5d1ea25700 1 -- 172.21.15.40:0/1279917645 shutdown_connections 2023-07-14T19:16:54.772 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:54.770+0000 7f5d1ea25700 1 --2- 172.21.15.40:0/1279917645 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f5d000a11e0 0x7f5d000a3690 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:54.772 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:54.770+0000 7f5d1ea25700 1 --2- 172.21.15.40:0/1279917645 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5d180feb80 0x7f5d180694e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:54.772 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:54.770+0000 7f5d1ea25700 1 --2- 172.21.15.40:0/1279917645 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5d18100700 0x7f5d1819cc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:54.772 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:54.770+0000 7f5d1ea25700 1 --2- 172.21.15.40:0/1279917645 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5d180ffd70 0x7f5d1819a640 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:54.773 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:54.770+0000 7f5d1ea25700 1 -- 172.21.15.40:0/1279917645 >> 172.21.15.40:0/1279917645 conn(0x7f5d180fa1a0 msgr2=0x7f5d180fc380 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:16:54.773 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:54.770+0000 7f5d1ea25700 1 -- 172.21.15.40:0/1279917645 shutdown_connections 2023-07-14T19:16:54.773 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:54.770+0000 7f5d1ea25700 1 -- 172.21.15.40:0/1279917645 wait complete. 2023-07-14T19:16:54.784 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-07-14T19:16:54.919 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:54.914+0000 7f957e568700 1 -- 172.21.15.40:0/395512280 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9578103060 msgr2=0x7f9578105440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:16:54.920 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:54.914+0000 7f957e568700 1 --2- 172.21.15.40:0/395512280 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9578103060 0x7f9578105440 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f956c009990 tx=0x7f956c009ca0 comp rx=0 tx=0).stop 2023-07-14T19:16:54.920 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:54.914+0000 7f957e568700 1 -- 172.21.15.40:0/395512280 shutdown_connections 2023-07-14T19:16:54.920 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:54.914+0000 7f957e568700 1 --2- 172.21.15.40:0/395512280 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9578100740 0x7f9578102b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:54.922 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:54.914+0000 7f957e568700 1 --2- 172.21.15.40:0/395512280 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9578105980 0x7f9578107d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:54.922 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:54.914+0000 7f957e568700 1 --2- 172.21.15.40:0/395512280 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9578103060 0x7f9578105440 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:54.922 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:54.914+0000 7f957e568700 1 -- 172.21.15.40:0/395512280 >> 172.21.15.40:0/395512280 conn(0x7f95780fa160 msgr2=0x7f95780fc570 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:16:54.922 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:54.914+0000 7f957e568700 1 -- 172.21.15.40:0/395512280 shutdown_connections 2023-07-14T19:16:54.923 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:54.914+0000 7f957e568700 1 -- 172.21.15.40:0/395512280 wait complete. 2023-07-14T19:16:54.923 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:54.914+0000 7f957e568700 1 Processor -- start 2023-07-14T19:16:54.923 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:54.914+0000 7f957e568700 1 -- start start 2023-07-14T19:16:54.923 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:54.915+0000 7f957e568700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9578100740 0x7f9578197b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:16:54.923 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:54.915+0000 7f957e568700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9578103060 0x7f9578198060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:16:54.923 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:54.915+0000 7f957e568700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9578105980 0x7f957819f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:16:54.923 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:54.915+0000 7f957e568700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f957819f6a0 con 0x7f9578103060 2023-07-14T19:16:54.924 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:54.915+0000 7f957e568700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f957819f7e0 con 0x7f9578105980 2023-07-14T19:16:54.924 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:54.915+0000 7f957e568700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f957819f970 con 0x7f9578100740 2023-07-14T19:16:54.924 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:54.915+0000 7f95777fe700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9578103060 0x7f9578198060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:16:54.925 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:54.915+0000 7f95777fe700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9578103060 0x7f9578198060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:33628/0 (socket says 172.21.15.40:33628) 2023-07-14T19:16:54.925 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:54.915+0000 7f95777fe700 1 -- 172.21.15.40:0/3113494074 learned_addr learned my addr 172.21.15.40:0/3113494074 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-07-14T19:16:54.925 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:54.915+0000 7f9577fff700 1 --2- 172.21.15.40:0/3113494074 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9578100740 0x7f9578197b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:16:54.926 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:54.915+0000 7f957cb05700 1 --2- 172.21.15.40:0/3113494074 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9578105980 0x7f957819f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:16:54.926 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:54.915+0000 7f95777fe700 1 -- 172.21.15.40:0/3113494074 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9578100740 msgr2=0x7f9578197b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:16:54.926 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:54.915+0000 7f95777fe700 1 --2- 172.21.15.40:0/3113494074 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9578100740 0x7f9578197b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:54.926 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:54.915+0000 7f95777fe700 1 -- 172.21.15.40:0/3113494074 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9578105980 msgr2=0x7f957819f0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:16:54.927 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:54.915+0000 7f95777fe700 1 --2- 172.21.15.40:0/3113494074 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9578105980 0x7f957819f0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:54.927 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:54.915+0000 7f95777fe700 1 -- 172.21.15.40:0/3113494074 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f956c0095f0 con 0x7f9578103060 2023-07-14T19:16:54.927 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:54.915+0000 7f957cb05700 1 --2- 172.21.15.40:0/3113494074 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9578105980 0x7f957819f0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-14T19:16:54.928 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:54.915+0000 7f95777fe700 1 --2- 172.21.15.40:0/3113494074 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9578103060 0x7f9578198060 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f956c000c00 tx=0x7f956c00f710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:16:54.928 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:54.915+0000 7f95757fa700 1 -- 172.21.15.40:0/3113494074 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f956c01d070 con 0x7f9578103060 2023-07-14T19:16:54.929 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:54.915+0000 7f95757fa700 1 -- 172.21.15.40:0/3113494074 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f956c00fb50 con 0x7f9578103060 2023-07-14T19:16:54.929 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:54.915+0000 7f957e568700 1 -- 172.21.15.40:0/3113494074 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f957819fbf0 con 0x7f9578103060 2023-07-14T19:16:54.929 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:54.915+0000 7f95757fa700 1 -- 172.21.15.40:0/3113494074 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f956c017650 con 0x7f9578103060 2023-07-14T19:16:54.929 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:54.916+0000 7f957e568700 1 -- 172.21.15.40:0/3113494074 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f95781a00f0 con 0x7f9578103060 2023-07-14T19:16:54.929 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:54.916+0000 7f957e568700 1 -- 172.21.15.40:0/3113494074 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f95781074f0 con 0x7f9578103060 2023-07-14T19:16:54.929 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:54.918+0000 7f95757fa700 1 -- 172.21.15.40:0/3113494074 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7f956c00fcc0 con 0x7f9578103060 2023-07-14T19:16:54.929 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:54.918+0000 7f95757fa700 1 --2- 172.21.15.40:0/3113494074 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f95600a1260 0x7f95600a3710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:16:54.929 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:54.918+0000 7f9577fff700 1 --2- 172.21.15.40:0/3113494074 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f95600a1260 0x7f95600a3710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:16:54.930 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:54.918+0000 7f95757fa700 1 -- 172.21.15.40:0/3113494074 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 8269+0+0 (secure 0 0 0) 0x7f956c0ce100 con 0x7f9578103060 2023-07-14T19:16:54.930 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:54.919+0000 7f9577fff700 1 --2- 172.21.15.40:0/3113494074 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f95600a1260 0x7f95600a3710 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f9568005950 tx=0x7f956800b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:16:54.930 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:54.921+0000 7f95757fa700 1 -- 172.21.15.40:0/3113494074 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f956c097760 con 0x7f9578103060 2023-07-14T19:16:55.123 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:55.122+0000 7f957e568700 1 -- 172.21.15.40:0/3113494074 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f957802cd70 con 0x7f9578103060 2023-07-14T19:16:55.123 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:55.122+0000 7f95757fa700 1 -- 172.21.15.40:0/3113494074 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v83) v1 ==== 104+0+11 (secure 0 0 0) 0x7f956c09ad80 con 0x7f9578103060 2023-07-14T19:16:55.124 INFO:teuthology.orchestra.run.smithi040.stdout:pg_num: 32 2023-07-14T19:16:55.125 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:55.125+0000 7f957e568700 1 -- 172.21.15.40:0/3113494074 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f95600a1260 msgr2=0x7f95600a3710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:16:55.126 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:55.125+0000 7f957e568700 1 --2- 172.21.15.40:0/3113494074 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f95600a1260 0x7f95600a3710 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f9568005950 tx=0x7f956800b410 comp rx=0 tx=0).stop 2023-07-14T19:16:55.126 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:55.125+0000 7f957e568700 1 -- 172.21.15.40:0/3113494074 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9578103060 msgr2=0x7f9578198060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:16:55.126 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:55.125+0000 7f957e568700 1 --2- 172.21.15.40:0/3113494074 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9578103060 0x7f9578198060 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f956c000c00 tx=0x7f956c00f710 comp rx=0 tx=0).stop 2023-07-14T19:16:55.126 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:55.125+0000 7f957e568700 1 -- 172.21.15.40:0/3113494074 shutdown_connections 2023-07-14T19:16:55.126 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:55.125+0000 7f957e568700 1 --2- 172.21.15.40:0/3113494074 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f95600a1260 0x7f95600a3710 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:55.126 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:55.125+0000 7f957e568700 1 --2- 172.21.15.40:0/3113494074 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9578100740 0x7f9578197b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:55.127 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:55.125+0000 7f957e568700 1 --2- 172.21.15.40:0/3113494074 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9578105980 0x7f957819f0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:55.127 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:55.125+0000 7f957e568700 1 --2- 172.21.15.40:0/3113494074 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9578103060 0x7f9578198060 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:55.127 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:55.125+0000 7f957e568700 1 -- 172.21.15.40:0/3113494074 >> 172.21.15.40:0/3113494074 conn(0x7f95780fa160 msgr2=0x7f95780fedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:16:55.127 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:55.125+0000 7f957e568700 1 -- 172.21.15.40:0/3113494074 shutdown_connections 2023-07-14T19:16:55.127 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:55.125+0000 7f957e568700 1 -- 172.21.15.40:0/3113494074 wait complete. 2023-07-14T19:16:55.137 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:16:54 smithi040 ceph-mon[96518]: pgmap v284: 97 pgs: 6 peering, 91 active+clean; 638 KiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 4 op/s 2023-07-14T19:16:55.137 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:16:54 smithi040 ceph-mon[96518]: from='client.? 172.21.15.40:0/249206260' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T19:16:55.137 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:16:54 smithi040 ceph-mon[96518]: from='client.? 172.21.15.40:0/2006297146' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T19:16:55.137 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:16:54 smithi040 ceph-mon[96518]: from='client.? 172.21.15.40:0/1279917645' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-07-14T19:16:55.138 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-07-14T19:16:55.222 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:16:54 smithi191 ceph-mon[98311]: pgmap v284: 97 pgs: 6 peering, 91 active+clean; 638 KiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 4 op/s 2023-07-14T19:16:55.222 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:16:54 smithi191 ceph-mon[98311]: from='client.? 172.21.15.40:0/249206260' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T19:16:55.222 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:16:54 smithi191 ceph-mon[98311]: from='client.? 172.21.15.40:0/2006297146' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T19:16:55.222 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:16:54 smithi191 ceph-mon[98311]: from='client.? 172.21.15.40:0/1279917645' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-07-14T19:16:55.242 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:16:54 smithi130 ceph-mon[97832]: pgmap v284: 97 pgs: 6 peering, 91 active+clean; 638 KiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 4 op/s 2023-07-14T19:16:55.242 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:16:54 smithi130 ceph-mon[97832]: from='client.? 172.21.15.40:0/249206260' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T19:16:55.243 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:16:54 smithi130 ceph-mon[97832]: from='client.? 172.21.15.40:0/2006297146' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T19:16:55.243 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:16:54 smithi130 ceph-mon[97832]: from='client.? 172.21.15.40:0/1279917645' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-07-14T19:16:55.272 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:55.271+0000 7f7dd89a1700 1 -- 172.21.15.40:0/257581379 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7dd40ffc10 msgr2=0x7f7dd4100020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:16:55.272 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:55.271+0000 7f7dd89a1700 1 --2- 172.21.15.40:0/257581379 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7dd40ffc10 0x7f7dd4100020 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f7dc4009990 tx=0x7f7dc4009ca0 comp rx=0 tx=0).stop 2023-07-14T19:16:55.272 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:55.271+0000 7f7dd89a1700 1 -- 172.21.15.40:0/257581379 shutdown_connections 2023-07-14T19:16:55.273 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:55.271+0000 7f7dd89a1700 1 --2- 172.21.15.40:0/257581379 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7dd4101790 0x7f7dd4108050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:55.273 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:55.271+0000 7f7dd89a1700 1 --2- 172.21.15.40:0/257581379 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7dd40ffc10 0x7f7dd4100020 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:55.273 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:55.271+0000 7f7dd89a1700 1 --2- 172.21.15.40:0/257581379 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7dd4100e00 0x7f7dd4101250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:55.273 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:55.271+0000 7f7dd89a1700 1 -- 172.21.15.40:0/257581379 >> 172.21.15.40:0/257581379 conn(0x7f7dd40fb250 msgr2=0x7f7dd40fd660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:16:55.273 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:55.271+0000 7f7dd89a1700 1 -- 172.21.15.40:0/257581379 shutdown_connections 2023-07-14T19:16:55.273 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:55.271+0000 7f7dd89a1700 1 -- 172.21.15.40:0/257581379 wait complete. 2023-07-14T19:16:55.275 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:55.272+0000 7f7dd89a1700 1 Processor -- start 2023-07-14T19:16:55.275 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:55.272+0000 7f7dd89a1700 1 -- start start 2023-07-14T19:16:55.276 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:55.272+0000 7f7dd89a1700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7dd40ffc10 0x7f7dd4195970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:16:55.276 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:55.272+0000 7f7dd89a1700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7dd4100e00 0x7f7dd4195eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:16:55.276 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:55.272+0000 7f7dd89a1700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7dd4101790 0x7f7dd419cf20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:16:55.276 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:55.272+0000 7f7dd89a1700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7dd419d4f0 con 0x7f7dd4101790 2023-07-14T19:16:55.276 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:55.272+0000 7f7dd89a1700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7dd419d630 con 0x7f7dd4100e00 2023-07-14T19:16:55.276 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:55.272+0000 7f7dd89a1700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7dd419d7c0 con 0x7f7dd40ffc10 2023-07-14T19:16:55.276 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:55.272+0000 7f7dd2d9d700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7dd4101790 0x7f7dd419cf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:16:55.277 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:55.272+0000 7f7dd2d9d700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7dd4101790 0x7f7dd419cf20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:33644/0 (socket says 172.21.15.40:33644) 2023-07-14T19:16:55.277 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:55.272+0000 7f7dd2d9d700 1 -- 172.21.15.40:0/627626701 learned_addr learned my addr 172.21.15.40:0/627626701 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-07-14T19:16:55.277 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:55.272+0000 7f7dd259c700 1 --2- 172.21.15.40:0/627626701 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7dd40ffc10 0x7f7dd4195970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:16:55.277 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:55.273+0000 7f7dd1d9b700 1 --2- 172.21.15.40:0/627626701 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7dd4100e00 0x7f7dd4195eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:16:55.278 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:55.273+0000 7f7dd2d9d700 1 -- 172.21.15.40:0/627626701 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7dd40ffc10 msgr2=0x7f7dd4195970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:16:55.278 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:55.273+0000 7f7dd2d9d700 1 --2- 172.21.15.40:0/627626701 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7dd40ffc10 0x7f7dd4195970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:55.278 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:55.273+0000 7f7dd2d9d700 1 -- 172.21.15.40:0/627626701 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7dd4100e00 msgr2=0x7f7dd4195eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:16:55.278 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:55.273+0000 7f7dd2d9d700 1 --2- 172.21.15.40:0/627626701 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7dd4100e00 0x7f7dd4195eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:55.278 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:55.273+0000 7f7dd2d9d700 1 -- 172.21.15.40:0/627626701 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7dc40095f0 con 0x7f7dd4101790 2023-07-14T19:16:55.278 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:55.273+0000 7f7dd2d9d700 1 --2- 172.21.15.40:0/627626701 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7dd4101790 0x7f7dd419cf20 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f7dcc00ef30 tx=0x7f7dcc00c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:16:55.278 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:55.273+0000 7f7dc37fe700 1 -- 172.21.15.40:0/627626701 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7dcc018070 con 0x7f7dd4101790 2023-07-14T19:16:55.279 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:55.273+0000 7f7dc37fe700 1 -- 172.21.15.40:0/627626701 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f7dcc00f040 con 0x7f7dd4101790 2023-07-14T19:16:55.279 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:55.273+0000 7f7dd89a1700 1 -- 172.21.15.40:0/627626701 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7dd419da40 con 0x7f7dd4101790 2023-07-14T19:16:55.279 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:55.273+0000 7f7dc37fe700 1 -- 172.21.15.40:0/627626701 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7dcc00cd00 con 0x7f7dd4101790 2023-07-14T19:16:55.279 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:55.273+0000 7f7dd89a1700 1 -- 172.21.15.40:0/627626701 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7dd419dfa0 con 0x7f7dd4101790 2023-07-14T19:16:55.279 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:55.274+0000 7f7dd89a1700 1 -- 172.21.15.40:0/627626701 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7dd4106b90 con 0x7f7dd4101790 2023-07-14T19:16:55.280 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:55.275+0000 7f7dc37fe700 1 -- 172.21.15.40:0/627626701 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7f7dcc003a00 con 0x7f7dd4101790 2023-07-14T19:16:55.280 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:55.276+0000 7f7dc37fe700 1 --2- 172.21.15.40:0/627626701 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f7dbc0a1260 0x7f7dbc0a3710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:16:55.280 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:55.276+0000 7f7dc37fe700 1 -- 172.21.15.40:0/627626701 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 8269+0+0 (secure 0 0 0) 0x7f7dcc0ccf20 con 0x7f7dd4101790 2023-07-14T19:16:55.280 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:55.276+0000 7f7dd259c700 1 --2- 172.21.15.40:0/627626701 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f7dbc0a1260 0x7f7dbc0a3710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:16:55.281 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:55.276+0000 7f7dd259c700 1 --2- 172.21.15.40:0/627626701 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f7dbc0a1260 0x7f7dbc0a3710 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f7dc4006010 tx=0x7f7dc4005960 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:16:55.281 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:55.279+0000 7f7dc37fe700 1 -- 172.21.15.40:0/627626701 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f7dcc096520 con 0x7f7dd4101790 2023-07-14T19:16:55.482 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:55.481+0000 7f7dd89a1700 1 -- 172.21.15.40:0/627626701 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f7dd402cb30 con 0x7f7dd4101790 2023-07-14T19:16:55.482 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:55.481+0000 7f7dc37fe700 1 -- 172.21.15.40:0/627626701 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v83) v1 ==== 100+0+11 (secure 0 0 0) 0x7f7dcc099b40 con 0x7f7dd4101790 2023-07-14T19:16:55.483 INFO:teuthology.orchestra.run.smithi040.stdout:pg_num: 64 2023-07-14T19:16:55.484 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:55.484+0000 7f7dd89a1700 1 -- 172.21.15.40:0/627626701 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f7dbc0a1260 msgr2=0x7f7dbc0a3710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:16:55.485 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:55.484+0000 7f7dd89a1700 1 --2- 172.21.15.40:0/627626701 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f7dbc0a1260 0x7f7dbc0a3710 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f7dc4006010 tx=0x7f7dc4005960 comp rx=0 tx=0).stop 2023-07-14T19:16:55.485 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:55.484+0000 7f7dd89a1700 1 -- 172.21.15.40:0/627626701 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7dd4101790 msgr2=0x7f7dd419cf20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:16:55.485 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:55.484+0000 7f7dd89a1700 1 --2- 172.21.15.40:0/627626701 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7dd4101790 0x7f7dd419cf20 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f7dcc00ef30 tx=0x7f7dcc00c3a0 comp rx=0 tx=0).stop 2023-07-14T19:16:55.485 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:55.484+0000 7f7dd89a1700 1 -- 172.21.15.40:0/627626701 shutdown_connections 2023-07-14T19:16:55.485 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:55.484+0000 7f7dd89a1700 1 --2- 172.21.15.40:0/627626701 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f7dbc0a1260 0x7f7dbc0a3710 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:55.485 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:55.484+0000 7f7dd89a1700 1 --2- 172.21.15.40:0/627626701 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7dd40ffc10 0x7f7dd4195970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:55.485 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:55.484+0000 7f7dd89a1700 1 --2- 172.21.15.40:0/627626701 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7dd4101790 0x7f7dd419cf20 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:55.486 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:55.484+0000 7f7dd89a1700 1 --2- 172.21.15.40:0/627626701 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7dd4100e00 0x7f7dd4195eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:55.486 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:55.484+0000 7f7dd89a1700 1 -- 172.21.15.40:0/627626701 >> 172.21.15.40:0/627626701 conn(0x7f7dd40fb250 msgr2=0x7f7dd40fd530 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:16:55.486 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:55.484+0000 7f7dd89a1700 1 -- 172.21.15.40:0/627626701 shutdown_connections 2023-07-14T19:16:55.486 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:55.484+0000 7f7dd89a1700 1 -- 172.21.15.40:0/627626701 wait complete. 2023-07-14T19:16:55.497 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-07-14T19:16:55.622 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:55.621+0000 7f84b5062700 1 -- 172.21.15.40:0/442659605 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f84b0100680 msgr2=0x7f84b0106f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:16:55.623 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:55.621+0000 7f84b5062700 1 --2- 172.21.15.40:0/442659605 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f84b0100680 0x7f84b0106f40 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f84a800b0a0 tx=0x7f84a800b3b0 comp rx=0 tx=0).stop 2023-07-14T19:16:55.623 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:55.622+0000 7f84b5062700 1 -- 172.21.15.40:0/442659605 shutdown_connections 2023-07-14T19:16:55.623 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:55.622+0000 7f84b5062700 1 --2- 172.21.15.40:0/442659605 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f84b00feb00 0x7f84b00fef10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:55.623 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:55.622+0000 7f84b5062700 1 --2- 172.21.15.40:0/442659605 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f84b0100680 0x7f84b0106f40 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:55.623 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:55.622+0000 7f84b5062700 1 --2- 172.21.15.40:0/442659605 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f84b00ffcf0 0x7f84b0100140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:55.624 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:55.622+0000 7f84b5062700 1 -- 172.21.15.40:0/442659605 >> 172.21.15.40:0/442659605 conn(0x7f84b00fa100 msgr2=0x7f84b00fc550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:16:55.624 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:55.622+0000 7f84b5062700 1 -- 172.21.15.40:0/442659605 shutdown_connections 2023-07-14T19:16:55.624 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:55.622+0000 7f84b5062700 1 -- 172.21.15.40:0/442659605 wait complete. 2023-07-14T19:16:55.626 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:55.622+0000 7f84b5062700 1 Processor -- start 2023-07-14T19:16:55.626 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:55.623+0000 7f84b5062700 1 -- start start 2023-07-14T19:16:55.626 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:55.623+0000 7f84b5062700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f84b00feb00 0x7f84b0068370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:16:55.626 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:55.623+0000 7f84b5062700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f84b00ffcf0 0x7f84b019a700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:16:55.627 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:55.623+0000 7f84b5062700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f84b0100680 0x7f84b019cd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:16:55.627 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:55.623+0000 7f84b5062700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84b019d530 con 0x7f84b0100680 2023-07-14T19:16:55.627 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:55.623+0000 7f84b5062700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84b019d6a0 con 0x7f84b00ffcf0 2023-07-14T19:16:55.627 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:55.623+0000 7f84b5062700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84b019d860 con 0x7f84b00feb00 2023-07-14T19:16:55.627 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:55.623+0000 7f84af59e700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f84b0100680 0x7f84b019cd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:16:55.627 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:55.623+0000 7f84af59e700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f84b0100680 0x7f84b019cd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:33658/0 (socket says 172.21.15.40:33658) 2023-07-14T19:16:55.628 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:55.623+0000 7f84aed9d700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f84b00feb00 0x7f84b0068370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:16:55.628 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:55.623+0000 7f84af59e700 1 -- 172.21.15.40:0/3609328448 learned_addr learned my addr 172.21.15.40:0/3609328448 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-07-14T19:16:55.628 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:55.623+0000 7f84ae59c700 1 --2- 172.21.15.40:0/3609328448 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f84b00ffcf0 0x7f84b019a700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:16:55.628 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:55.623+0000 7f84af59e700 1 -- 172.21.15.40:0/3609328448 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f84b00feb00 msgr2=0x7f84b0068370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:16:55.628 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:55.623+0000 7f84af59e700 1 --2- 172.21.15.40:0/3609328448 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f84b00feb00 0x7f84b0068370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:55.629 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:55.623+0000 7f84af59e700 1 -- 172.21.15.40:0/3609328448 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f84b00ffcf0 msgr2=0x7f84b019a700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:16:55.629 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:55.623+0000 7f84af59e700 1 --2- 172.21.15.40:0/3609328448 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f84b00ffcf0 0x7f84b019a700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:55.629 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:55.623+0000 7f84af59e700 1 -- 172.21.15.40:0/3609328448 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84a8009d00 con 0x7f84b0100680 2023-07-14T19:16:55.629 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:55.623+0000 7f84aed9d700 1 --2- 172.21.15.40:0/3609328448 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f84b00feb00 0x7f84b0068370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-14T19:16:55.629 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:55.623+0000 7f84af59e700 1 --2- 172.21.15.40:0/3609328448 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f84b0100680 0x7f84b019cd90 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f84a800b780 tx=0x7f84a800bd00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:16:55.629 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:55.624+0000 7f8497fff700 1 -- 172.21.15.40:0/3609328448 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84a800e040 con 0x7f84b0100680 2023-07-14T19:16:55.629 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:55.624+0000 7f8497fff700 1 -- 172.21.15.40:0/3609328448 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f84a8004c60 con 0x7f84b0100680 2023-07-14T19:16:55.630 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:55.624+0000 7f84b5062700 1 -- 172.21.15.40:0/3609328448 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f84b019dae0 con 0x7f84b0100680 2023-07-14T19:16:55.630 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:55.624+0000 7f8497fff700 1 -- 172.21.15.40:0/3609328448 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84a801b6b0 con 0x7f84b0100680 2023-07-14T19:16:55.630 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:55.624+0000 7f84b5062700 1 -- 172.21.15.40:0/3609328448 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f84b019df60 con 0x7f84b0100680 2023-07-14T19:16:55.630 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:55.625+0000 7f84b5062700 1 -- 172.21.15.40:0/3609328448 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f84b010a300 con 0x7f84b0100680 2023-07-14T19:16:55.631 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:55.626+0000 7f8497fff700 1 -- 172.21.15.40:0/3609328448 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7f84a801d480 con 0x7f84b0100680 2023-07-14T19:16:55.631 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:55.626+0000 7f8497fff700 1 --2- 172.21.15.40:0/3609328448 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f84980a1220 0x7f84980a36d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:16:55.631 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:55.627+0000 7f84aed9d700 1 --2- 172.21.15.40:0/3609328448 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f84980a1220 0x7f84980a36d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:16:55.631 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:55.627+0000 7f8497fff700 1 -- 172.21.15.40:0/3609328448 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 8269+0+0 (secure 0 0 0) 0x7f84a80ce420 con 0x7f84b0100680 2023-07-14T19:16:55.631 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:55.627+0000 7f84aed9d700 1 --2- 172.21.15.40:0/3609328448 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f84980a1220 0x7f84980a36d0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f84b00ffb50 tx=0x7f84a0009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:16:55.632 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:55.631+0000 7f8497fff700 1 -- 172.21.15.40:0/3609328448 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f84a8097a20 con 0x7f84b0100680 2023-07-14T19:16:55.878 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:55.877+0000 7f84b5062700 1 -- 172.21.15.40:0/3609328448 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f84b010a4f0 con 0x7f84b0100680 2023-07-14T19:16:55.878 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:55.878+0000 7f8497fff700 1 -- 172.21.15.40:0/3609328448 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) v1 ==== 94+0+10877 (secure 0 0 0) 0x7f84a809b040 con 0x7f84b0100680 2023-07-14T19:16:55.879 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-07-14T19:16:55.879 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14523,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6835/501143718","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6834","nonce":501143718},{"type":"v1","addr":"172.21.15.40:6835","nonce":501143718}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14535,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6837/2634476117","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6836","nonce":2634476117},{"type":"v1","addr":"172.21.15.40:6837","nonce":2634476117}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":26,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2023-07-14T19:15:57.650710+0000","modified":"2023-07-14T19:16:05.624568+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24539,"mds_1":24508,"mds_2":24506,"mds_3":24490,"mds_4":24482},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14586":{"gid":14586,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.40:6839/959937545","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6838","nonce":959937545},{"type":"v1","addr":"172.21.15.40:6839","nonce":959937545}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_14598":{"gid":14598,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.40:6841/1590486077","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6840","nonce":1590486077},{"type":"v1","addr":"172.21.15.40:6841","nonce":1590486077}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24433":{"gid":24433,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.191:6833/31509862","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":31509862},{"type":"v1","addr":"172.21.15.191:6833","nonce":31509862}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24452":{"gid":24452,"name":"k","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.130:6833/3677452688","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6832","nonce":3677452688},{"type":"v1","addr":"172.21.15.130:6833","nonce":3677452688}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24478":{"gid":24478,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.191:6835/3712674634","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":3712674634},{"type":"v1","addr":"172.21.15.191:6835","nonce":3712674634}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24482":{"gid":24482,"name":"e","rank":4,"incarnation":24,"state":"up:active","state_seq":8,"addr":"172.21.15.130:6835/3088490808","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6834","nonce":3088490808},{"type":"v1","addr":"172.21.15.130:6835","nonce":3088490808}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24490":{"gid":24490,"name":"c","rank":3,"incarnation":22,"state":"up:active","state_seq":7,"addr":"172.21.15.191:6837/2406892975","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6836","nonce":2406892975},{"type":"v1","addr":"172.21.15.191:6837","nonce":2406892975}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24506":{"gid":24506,"name":"h","rank":2,"incarnation":20,"state":"up:active","state_seq":6,"addr":"172.21.15.130:6837/2604019369","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6836","nonce":2604019369},{"type":"v1","addr":"172.21.15.130:6837","nonce":2604019369}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24508":{"gid":24508,"name":"i","rank":1,"incarnation":18,"state":"up:active","state_seq":5,"addr":"172.21.15.191:6839/1977507302","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":1977507302},{"type":"v1","addr":"172.21.15.191:6839","nonce":1977507302}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24539":{"gid":24539,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.130:6839/2145479535","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6838","nonce":2145479535},{"type":"v1","addr":"172.21.15.130:6839","nonce":2145479535}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-07-14T19:16:55.880 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:55.880+0000 7f84b5062700 1 -- 172.21.15.40:0/3609328448 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f84980a1220 msgr2=0x7f84980a36d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:16:55.881 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:55.880+0000 7f84b5062700 1 --2- 172.21.15.40:0/3609328448 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f84980a1220 0x7f84980a36d0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f84b00ffb50 tx=0x7f84a0009260 comp rx=0 tx=0).stop 2023-07-14T19:16:55.881 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:55.880+0000 7f84b5062700 1 -- 172.21.15.40:0/3609328448 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f84b0100680 msgr2=0x7f84b019cd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:16:55.881 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:55.880+0000 7f84b5062700 1 --2- 172.21.15.40:0/3609328448 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f84b0100680 0x7f84b019cd90 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f84a800b780 tx=0x7f84a800bd00 comp rx=0 tx=0).stop 2023-07-14T19:16:55.881 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:55.880+0000 7f84b5062700 1 -- 172.21.15.40:0/3609328448 shutdown_connections 2023-07-14T19:16:55.881 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:55.880+0000 7f84b5062700 1 --2- 172.21.15.40:0/3609328448 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f84980a1220 0x7f84980a36d0 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:55.881 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:55.880+0000 7f84b5062700 1 --2- 172.21.15.40:0/3609328448 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f84b00feb00 0x7f84b0068370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:55.881 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:55.880+0000 7f84b5062700 1 --2- 172.21.15.40:0/3609328448 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f84b0100680 0x7f84b019cd90 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:55.882 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:55.880+0000 7f84b5062700 1 --2- 172.21.15.40:0/3609328448 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f84b00ffcf0 0x7f84b019a700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:55.882 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:55.880+0000 7f84b5062700 1 -- 172.21.15.40:0/3609328448 >> 172.21.15.40:0/3609328448 conn(0x7f84b00fa100 msgr2=0x7f84b00fc2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:16:55.882 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:55.880+0000 7f84b5062700 1 -- 172.21.15.40:0/3609328448 shutdown_connections 2023-07-14T19:16:55.882 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:55.880+0000 7f84b5062700 1 -- 172.21.15.40:0/3609328448 wait complete. 2023-07-14T19:16:55.882 INFO:teuthology.orchestra.run.smithi040.stderr:dumped fsmap epoch 26 2023-07-14T19:16:55.893 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-07-14T19:16:56.019 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.018+0000 7f5da2b55700 1 -- 172.21.15.40:0/375949459 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5d9c1058e0 msgr2=0x7f5d9c107d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:16:56.020 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.018+0000 7f5da2b55700 1 --2- 172.21.15.40:0/375949459 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5d9c1058e0 0x7f5d9c107d40 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f5d9400b0a0 tx=0x7f5d9400b3b0 comp rx=0 tx=0).stop 2023-07-14T19:16:56.020 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.018+0000 7f5da2b55700 1 -- 172.21.15.40:0/375949459 shutdown_connections 2023-07-14T19:16:56.020 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.018+0000 7f5da2b55700 1 --2- 172.21.15.40:0/375949459 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5d9c1006a0 0x7f5d9c102a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:56.020 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.018+0000 7f5da2b55700 1 --2- 172.21.15.40:0/375949459 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5d9c1058e0 0x7f5d9c107d40 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:56.020 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.018+0000 7f5da2b55700 1 --2- 172.21.15.40:0/375949459 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5d9c102fc0 0x7f5d9c1053a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:56.021 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.018+0000 7f5da2b55700 1 -- 172.21.15.40:0/375949459 >> 172.21.15.40:0/375949459 conn(0x7f5d9c0fa100 msgr2=0x7f5d9c0fc570 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:16:56.021 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.018+0000 7f5da2b55700 1 -- 172.21.15.40:0/375949459 shutdown_connections 2023-07-14T19:16:56.021 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.018+0000 7f5da2b55700 1 -- 172.21.15.40:0/375949459 wait complete. 2023-07-14T19:16:56.021 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.019+0000 7f5da2b55700 1 Processor -- start 2023-07-14T19:16:56.021 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.019+0000 7f5da2b55700 1 -- start start 2023-07-14T19:16:56.023 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.019+0000 7f5da2b55700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5d9c1006a0 0x7f5d9c1959a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:16:56.023 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.019+0000 7f5da2b55700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5d9c102fc0 0x7f5d9c195ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:16:56.023 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.019+0000 7f5da2b55700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5d9c1058e0 0x7f5d9c19cf50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:16:56.023 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.019+0000 7f5da2b55700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d9c19d520 con 0x7f5d9c1058e0 2023-07-14T19:16:56.024 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.019+0000 7f5da2b55700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d9c19d660 con 0x7f5d9c1006a0 2023-07-14T19:16:56.024 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.019+0000 7f5da2b55700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d9c19d7f0 con 0x7f5d9c102fc0 2023-07-14T19:16:56.024 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.019+0000 7f5da10f2700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5d9c1058e0 0x7f5d9c19cf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:16:56.024 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.019+0000 7f5da10f2700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5d9c1058e0 0x7f5d9c19cf50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:33672/0 (socket says 172.21.15.40:33672) 2023-07-14T19:16:56.024 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.019+0000 7f5da10f2700 1 -- 172.21.15.40:0/311948507 learned_addr learned my addr 172.21.15.40:0/311948507 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-07-14T19:16:56.024 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.019+0000 7f5d9bfff700 1 --2- 172.21.15.40:0/311948507 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5d9c102fc0 0x7f5d9c195ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:16:56.024 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.020+0000 7f5da08f1700 1 --2- 172.21.15.40:0/311948507 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5d9c1006a0 0x7f5d9c1959a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:16:56.025 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.020+0000 7f5da10f2700 1 -- 172.21.15.40:0/311948507 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5d9c102fc0 msgr2=0x7f5d9c195ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:16:56.025 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.020+0000 7f5da10f2700 1 --2- 172.21.15.40:0/311948507 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5d9c102fc0 0x7f5d9c195ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:56.025 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.020+0000 7f5da10f2700 1 -- 172.21.15.40:0/311948507 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5d9c1006a0 msgr2=0x7f5d9c1959a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:16:56.025 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.020+0000 7f5da10f2700 1 --2- 172.21.15.40:0/311948507 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5d9c1006a0 0x7f5d9c1959a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:56.025 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.020+0000 7f5da10f2700 1 -- 172.21.15.40:0/311948507 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d94009d00 con 0x7f5d9c1058e0 2023-07-14T19:16:56.025 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.020+0000 7f5da10f2700 1 --2- 172.21.15.40:0/311948507 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5d9c1058e0 0x7f5d9c19cf50 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f5d940062a0 tx=0x7f5d94004d00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:16:56.025 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.020+0000 7f5d99ffb700 1 -- 172.21.15.40:0/311948507 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d9400e040 con 0x7f5d9c1058e0 2023-07-14T19:16:56.026 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.020+0000 7f5d99ffb700 1 -- 172.21.15.40:0/311948507 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f5d9400cec0 con 0x7f5d9c1058e0 2023-07-14T19:16:56.026 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.020+0000 7f5da2b55700 1 -- 172.21.15.40:0/311948507 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5d9c19da70 con 0x7f5d9c1058e0 2023-07-14T19:16:56.026 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.020+0000 7f5d99ffb700 1 -- 172.21.15.40:0/311948507 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d9401baf0 con 0x7f5d9c1058e0 2023-07-14T19:16:56.026 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.020+0000 7f5da2b55700 1 -- 172.21.15.40:0/311948507 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5d9c19def0 con 0x7f5d9c1058e0 2023-07-14T19:16:56.026 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.021+0000 7f5da2b55700 1 -- 172.21.15.40:0/311948507 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5d9c0fbc70 con 0x7f5d9c1058e0 2023-07-14T19:16:56.026 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.022+0000 7f5d99ffb700 1 -- 172.21.15.40:0/311948507 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7f5d9401bc50 con 0x7f5d9c1058e0 2023-07-14T19:16:56.027 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.023+0000 7f5d99ffb700 1 --2- 172.21.15.40:0/311948507 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f5d8c0a1220 0x7f5d8c0a36d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:16:56.027 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.023+0000 7f5da08f1700 1 --2- 172.21.15.40:0/311948507 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f5d8c0a1220 0x7f5d8c0a36d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:16:56.027 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.023+0000 7f5d99ffb700 1 -- 172.21.15.40:0/311948507 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 8269+0+0 (secure 0 0 0) 0x7f5d940ce630 con 0x7f5d9c1058e0 2023-07-14T19:16:56.027 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.024+0000 7f5da08f1700 1 --2- 172.21.15.40:0/311948507 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f5d8c0a1220 0x7f5d8c0a36d0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f5d90005950 tx=0x7f5d9000b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:16:56.030 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.028+0000 7f5d99ffb700 1 -- 172.21.15.40:0/311948507 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f5d94097c30 con 0x7f5d9c1058e0 2023-07-14T19:16:56.221 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:16:55 smithi040 ceph-mon[96518]: from='client.? 172.21.15.40:0/3113494074' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-07-14T19:16:56.221 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:16:55 smithi040 ceph-mon[96518]: from='client.? 172.21.15.40:0/627626701' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-07-14T19:16:56.222 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:16:55 smithi191 ceph-mon[98311]: from='client.? 172.21.15.40:0/3113494074' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-07-14T19:16:56.222 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:16:55 smithi191 ceph-mon[98311]: from='client.? 172.21.15.40:0/627626701' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-07-14T19:16:56.231 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.231+0000 7f5da2b55700 1 -- 172.21.15.40:0/311948507 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5d9c02caf0 con 0x7f5d9c1058e0 2023-07-14T19:16:56.233 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.232+0000 7f5d99ffb700 1 -- 172.21.15.40:0/311948507 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v83) v1 ==== 74+0+22647 (secure 0 0 0) 0x7f5d9409b250 con 0x7f5d9c1058e0 2023-07-14T19:16:56.233 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-07-14T19:16:56.233 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":83,"fsid":"adc52f04-2279-11ee-9b33-001a4aab830c","created":"2023-07-14T19:08:56.893730+0000","modified":"2023-07-14T19:16:52.763512+0000","last_up_change":"2023-07-14T19:15:16.632979+0000","last_in_change":"2023-07-14T19:14:58.961702+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-07-14T19:12:03.271333+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-07-14T19:15:56.073464+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-07-14T19:15:57.018726+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"0d79356a-95d2-49ad-857b-2092ae9f7e13","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":3599703125},{"type":"v1","addr":"172.21.15.40:6803","nonce":3599703125}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":3599703125},{"type":"v1","addr":"172.21.15.40:6805","nonce":3599703125}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":3599703125},{"type":"v1","addr":"172.21.15.40:6809","nonce":3599703125}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":3599703125},{"type":"v1","addr":"172.21.15.40:6807","nonce":3599703125}]},"public_addr":"172.21.15.40:6803/3599703125","cluster_addr":"172.21.15.40:6805/3599703125","heartbeat_back_addr":"172.21.15.40:6809/3599703125","heartbeat_front_addr":"172.21.15.40:6807/3599703125","state":["exists","up"]},{"osd":1,"uuid":"40f8a3d1-a5e4-473c-b9f2-7600ec22be0d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6800","nonce":2512760702},{"type":"v1","addr":"172.21.15.130:6801","nonce":2512760702}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6802","nonce":2512760702},{"type":"v1","addr":"172.21.15.130:6803","nonce":2512760702}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6806","nonce":2512760702},{"type":"v1","addr":"172.21.15.130:6807","nonce":2512760702}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6804","nonce":2512760702},{"type":"v1","addr":"172.21.15.130:6805","nonce":2512760702}]},"public_addr":"172.21.15.130:6801/2512760702","cluster_addr":"172.21.15.130:6803/2512760702","heartbeat_back_addr":"172.21.15.130:6807/2512760702","heartbeat_front_addr":"172.21.15.130:6805/2512760702","state":["exists","up"]},{"osd":2,"uuid":"5a157891-cb7a-4ea6-9239-f48a7c67c003","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6800","nonce":589569135},{"type":"v1","addr":"172.21.15.191:6801","nonce":589569135}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6802","nonce":589569135},{"type":"v1","addr":"172.21.15.191:6803","nonce":589569135}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6806","nonce":589569135},{"type":"v1","addr":"172.21.15.191:6807","nonce":589569135}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6804","nonce":589569135},{"type":"v1","addr":"172.21.15.191:6805","nonce":589569135}]},"public_addr":"172.21.15.191:6801/589569135","cluster_addr":"172.21.15.191:6803/589569135","heartbeat_back_addr":"172.21.15.191:6807/589569135","heartbeat_front_addr":"172.21.15.191:6805/589569135","state":["exists","up"]},{"osd":3,"uuid":"8020e7dc-ac40-487f-857a-8da40bccb459","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":996287860},{"type":"v1","addr":"172.21.15.40:6811","nonce":996287860}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6812","nonce":996287860},{"type":"v1","addr":"172.21.15.40:6813","nonce":996287860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6816","nonce":996287860},{"type":"v1","addr":"172.21.15.40:6817","nonce":996287860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":996287860},{"type":"v1","addr":"172.21.15.40:6815","nonce":996287860}]},"public_addr":"172.21.15.40:6811/996287860","cluster_addr":"172.21.15.40:6813/996287860","heartbeat_back_addr":"172.21.15.40:6817/996287860","heartbeat_front_addr":"172.21.15.40:6815/996287860","state":["exists","up"]},{"osd":4,"uuid":"c5501b35-f62d-482d-8d22-115c88f0298e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6808","nonce":3507448720},{"type":"v1","addr":"172.21.15.130:6809","nonce":3507448720}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6810","nonce":3507448720},{"type":"v1","addr":"172.21.15.130:6811","nonce":3507448720}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6814","nonce":3507448720},{"type":"v1","addr":"172.21.15.130:6815","nonce":3507448720}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6812","nonce":3507448720},{"type":"v1","addr":"172.21.15.130:6813","nonce":3507448720}]},"public_addr":"172.21.15.130:6809/3507448720","cluster_addr":"172.21.15.130:6811/3507448720","heartbeat_back_addr":"172.21.15.130:6815/3507448720","heartbeat_front_addr":"172.21.15.130:6813/3507448720","state":["exists","up"]},{"osd":5,"uuid":"7770b6d6-5874-493a-986a-84db482cee42","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6808","nonce":1781944663},{"type":"v1","addr":"172.21.15.191:6809","nonce":1781944663}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6810","nonce":1781944663},{"type":"v1","addr":"172.21.15.191:6811","nonce":1781944663}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6814","nonce":1781944663},{"type":"v1","addr":"172.21.15.191:6815","nonce":1781944663}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6812","nonce":1781944663},{"type":"v1","addr":"172.21.15.191:6813","nonce":1781944663}]},"public_addr":"172.21.15.191:6809/1781944663","cluster_addr":"172.21.15.191:6811/1781944663","heartbeat_back_addr":"172.21.15.191:6815/1781944663","heartbeat_front_addr":"172.21.15.191:6813/1781944663","state":["exists","up"]},{"osd":6,"uuid":"ef65ad00-dfee-4300-9961-dfc8573aab06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6818","nonce":693186773},{"type":"v1","addr":"172.21.15.40:6819","nonce":693186773}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6820","nonce":693186773},{"type":"v1","addr":"172.21.15.40:6821","nonce":693186773}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6824","nonce":693186773},{"type":"v1","addr":"172.21.15.40:6825","nonce":693186773}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6822","nonce":693186773},{"type":"v1","addr":"172.21.15.40:6823","nonce":693186773}]},"public_addr":"172.21.15.40:6819/693186773","cluster_addr":"172.21.15.40:6821/693186773","heartbeat_back_addr":"172.21.15.40:6825/693186773","heartbeat_front_addr":"172.21.15.40:6823/693186773","state":["exists","up"]},{"osd":7,"uuid":"1700fbfa-cbfc-4f76-97be-727a3b100740","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":50,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6816","nonce":2384019528},{"type":"v1","addr":"172.21.15.130:6817","nonce":2384019528}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6818","nonce":2384019528},{"type":"v1","addr":"172.21.15.130:6819","nonce":2384019528}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6822","nonce":2384019528},{"type":"v1","addr":"172.21.15.130:6823","nonce":2384019528}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6820","nonce":2384019528},{"type":"v1","addr":"172.21.15.130:6821","nonce":2384019528}]},"public_addr":"172.21.15.130:6817/2384019528","cluster_addr":"172.21.15.130:6819/2384019528","heartbeat_back_addr":"172.21.15.130:6823/2384019528","heartbeat_front_addr":"172.21.15.130:6821/2384019528","state":["exists","up"]},{"osd":8,"uuid":"b76d4033-c1e2-457b-8571-c8a49fe443f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":57,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6816","nonce":2386818417},{"type":"v1","addr":"172.21.15.191:6817","nonce":2386818417}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6818","nonce":2386818417},{"type":"v1","addr":"172.21.15.191:6819","nonce":2386818417}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6822","nonce":2386818417},{"type":"v1","addr":"172.21.15.191:6823","nonce":2386818417}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6820","nonce":2386818417},{"type":"v1","addr":"172.21.15.191:6821","nonce":2386818417}]},"public_addr":"172.21.15.191:6817/2386818417","cluster_addr":"172.21.15.191:6819/2386818417","heartbeat_back_addr":"172.21.15.191:6823/2386818417","heartbeat_front_addr":"172.21.15.191:6821/2386818417","state":["exists","up"]},{"osd":9,"uuid":"df31bf2f-e30e-41b6-b2b9-0e26a07a88bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":64,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6826","nonce":2228380263},{"type":"v1","addr":"172.21.15.40:6827","nonce":2228380263}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6828","nonce":2228380263},{"type":"v1","addr":"172.21.15.40:6829","nonce":2228380263}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":2228380263},{"type":"v1","addr":"172.21.15.40:6833","nonce":2228380263}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6830","nonce":2228380263},{"type":"v1","addr":"172.21.15.40:6831","nonce":2228380263}]},"public_addr":"172.21.15.40:6827/2228380263","cluster_addr":"172.21.15.40:6829/2228380263","heartbeat_back_addr":"172.21.15.40:6833/2228380263","heartbeat_front_addr":"172.21.15.40:6831/2228380263","state":["exists","up"]},{"osd":10,"uuid":"4a93d621-49ee-4ad7-9e49-750614faf912","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6824","nonce":2417067798},{"type":"v1","addr":"172.21.15.130:6825","nonce":2417067798}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6826","nonce":2417067798},{"type":"v1","addr":"172.21.15.130:6827","nonce":2417067798}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6830","nonce":2417067798},{"type":"v1","addr":"172.21.15.130:6831","nonce":2417067798}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6828","nonce":2417067798},{"type":"v1","addr":"172.21.15.130:6829","nonce":2417067798}]},"public_addr":"172.21.15.130:6825/2417067798","cluster_addr":"172.21.15.130:6827/2417067798","heartbeat_back_addr":"172.21.15.130:6831/2417067798","heartbeat_front_addr":"172.21.15.130:6829/2417067798","state":["exists","up"]},{"osd":11,"uuid":"f43a2a6f-82f3-42ac-b77c-4f1b34da6918","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":76,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6824","nonce":2886059446},{"type":"v1","addr":"172.21.15.191:6825","nonce":2886059446}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6826","nonce":2886059446},{"type":"v1","addr":"172.21.15.191:6827","nonce":2886059446}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6830","nonce":2886059446},{"type":"v1","addr":"172.21.15.191:6831","nonce":2886059446}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6828","nonce":2886059446},{"type":"v1","addr":"172.21.15.191:6829","nonce":2886059446}]},"public_addr":"172.21.15.191:6825/2886059446","cluster_addr":"172.21.15.191:6827/2886059446","heartbeat_back_addr":"172.21.15.191:6831/2886059446","heartbeat_front_addr":"172.21.15.191:6829/2886059446","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:11:18.320495+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:11:39.901199+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:12:00.869042+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:12:23.292742+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:12:44.587019+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:13:06.494893+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:13:28.378374+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:13:50.033491+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:14:10.937062+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:14:32.358498+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:14:53.666584+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:15:14.837566+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"2.b","mappings":[{"from":8,"to":11}]},{"pgid":"2.c","mappings":[{"from":2,"to":4}]},{"pgid":"2.1f","mappings":[{"from":8,"to":1}]},{"pgid":"3.e","mappings":[{"from":4,"to":7}]},{"pgid":"3.17","mappings":[{"from":4,"to":8}]},{"pgid":"3.1a","mappings":[{"from":5,"to":6}]},{"pgid":"3.20","mappings":[{"from":5,"to":9}]},{"pgid":"3.27","mappings":[{"from":3,"to":9}]},{"pgid":"3.2c","mappings":[{"from":4,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.40:6801/3698757264":"2023-07-15T19:09:50.732458+0000","172.21.15.40:6800/3698757264":"2023-07-15T19:09:50.732458+0000","172.21.15.40:6800/2569015667":"2023-07-15T19:09:24.441543+0000","172.21.15.40:0/3653961003":"2023-07-15T19:09:24.441543+0000","172.21.15.40:0/3568861680":"2023-07-15T19:09:24.441543+0000","172.21.15.40:0/3249773172":"2023-07-15T19:09:50.732458+0000","172.21.15.40:0/4084215480":"2023-07-15T19:09:24.441543+0000","172.21.15.40:0/1163941760":"2023-07-15T19:09:50.732458+0000","172.21.15.40:6801/2569015667":"2023-07-15T19:09:24.441543+0000","172.21.15.40:0/2326639960":"2023-07-15T19:09:50.732458+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-07-14T19:16:56.235 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.234+0000 7f5da2b55700 1 -- 172.21.15.40:0/311948507 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f5d8c0a1220 msgr2=0x7f5d8c0a36d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:16:56.235 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.234+0000 7f5da2b55700 1 --2- 172.21.15.40:0/311948507 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f5d8c0a1220 0x7f5d8c0a36d0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f5d90005950 tx=0x7f5d9000b410 comp rx=0 tx=0).stop 2023-07-14T19:16:56.235 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.234+0000 7f5da2b55700 1 -- 172.21.15.40:0/311948507 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5d9c1058e0 msgr2=0x7f5d9c19cf50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:16:56.236 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.234+0000 7f5da2b55700 1 --2- 172.21.15.40:0/311948507 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5d9c1058e0 0x7f5d9c19cf50 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f5d940062a0 tx=0x7f5d94004d00 comp rx=0 tx=0).stop 2023-07-14T19:16:56.236 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.234+0000 7f5da2b55700 1 -- 172.21.15.40:0/311948507 shutdown_connections 2023-07-14T19:16:56.236 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.234+0000 7f5da2b55700 1 --2- 172.21.15.40:0/311948507 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f5d8c0a1220 0x7f5d8c0a36d0 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:56.236 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.234+0000 7f5da2b55700 1 --2- 172.21.15.40:0/311948507 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5d9c1058e0 0x7f5d9c19cf50 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:56.236 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.234+0000 7f5da2b55700 1 --2- 172.21.15.40:0/311948507 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5d9c1006a0 0x7f5d9c1959a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:56.236 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.234+0000 7f5da2b55700 1 --2- 172.21.15.40:0/311948507 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5d9c102fc0 0x7f5d9c195ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:56.236 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.234+0000 7f5da2b55700 1 -- 172.21.15.40:0/311948507 >> 172.21.15.40:0/311948507 conn(0x7f5d9c0fa100 msgr2=0x7f5d9c101300 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:16:56.237 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.234+0000 7f5da2b55700 1 -- 172.21.15.40:0/311948507 shutdown_connections 2023-07-14T19:16:56.237 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.235+0000 7f5da2b55700 1 -- 172.21.15.40:0/311948507 wait complete. 2023-07-14T19:16:56.242 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:16:55 smithi130 ceph-mon[97832]: from='client.? 172.21.15.40:0/3113494074' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-07-14T19:16:56.242 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:16:55 smithi130 ceph-mon[97832]: from='client.? 172.21.15.40:0/627626701' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-07-14T19:16:56.248 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-07-14T19:16:56.374 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.373+0000 7f71ed32f700 1 -- 172.21.15.40:0/3848120973 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f71e80feb20 msgr2=0x7f71e80fef30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:16:56.374 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.373+0000 7f71ed32f700 1 --2- 172.21.15.40:0/3848120973 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f71e80feb20 0x7f71e80fef30 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f71d0009990 tx=0x7f71d0009ca0 comp rx=0 tx=0).stop 2023-07-14T19:16:56.374 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.373+0000 7f71ed32f700 1 -- 172.21.15.40:0/3848120973 shutdown_connections 2023-07-14T19:16:56.375 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.373+0000 7f71ed32f700 1 --2- 172.21.15.40:0/3848120973 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f71e81006a0 0x7f71e8106f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:56.375 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.373+0000 7f71ed32f700 1 --2- 172.21.15.40:0/3848120973 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f71e80feb20 0x7f71e80fef30 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:56.375 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.373+0000 7f71ed32f700 1 --2- 172.21.15.40:0/3848120973 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f71e80ffd10 0x7f71e8100160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:56.375 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.373+0000 7f71ed32f700 1 -- 172.21.15.40:0/3848120973 >> 172.21.15.40:0/3848120973 conn(0x7f71e80fa100 msgr2=0x7f71e80fc570 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:16:56.375 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.373+0000 7f71ed32f700 1 -- 172.21.15.40:0/3848120973 shutdown_connections 2023-07-14T19:16:56.375 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.373+0000 7f71ed32f700 1 -- 172.21.15.40:0/3848120973 wait complete. 2023-07-14T19:16:56.375 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.373+0000 7f71ed32f700 1 Processor -- start 2023-07-14T19:16:56.377 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.374+0000 7f71ed32f700 1 -- start start 2023-07-14T19:16:56.377 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.374+0000 7f71ed32f700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f71e80feb20 0x7f71e8194770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:16:56.377 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.374+0000 7f71ed32f700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f71e80ffd10 0x7f71e8194cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:16:56.378 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.374+0000 7f71ed32f700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f71e81006a0 0x7f71e819bd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:16:56.378 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.374+0000 7f71ed32f700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71e819c2f0 con 0x7f71e80feb20 2023-07-14T19:16:56.378 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.374+0000 7f71ed32f700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71e819c430 con 0x7f71e80ffd10 2023-07-14T19:16:56.378 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.374+0000 7f71ed32f700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71e819c5c0 con 0x7f71e81006a0 2023-07-14T19:16:56.378 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.374+0000 7f71e6ffd700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f71e80feb20 0x7f71e8194770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:16:56.378 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.374+0000 7f71e6ffd700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f71e80feb20 0x7f71e8194770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:33682/0 (socket says 172.21.15.40:33682) 2023-07-14T19:16:56.378 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.374+0000 7f71e6ffd700 1 -- 172.21.15.40:0/2968496269 learned_addr learned my addr 172.21.15.40:0/2968496269 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-07-14T19:16:56.379 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.374+0000 7f71e77fe700 1 --2- 172.21.15.40:0/2968496269 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f71e81006a0 0x7f71e819bd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:16:56.379 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.374+0000 7f71e67fc700 1 --2- 172.21.15.40:0/2968496269 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f71e80ffd10 0x7f71e8194cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:16:56.379 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.374+0000 7f71e6ffd700 1 -- 172.21.15.40:0/2968496269 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f71e81006a0 msgr2=0x7f71e819bd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:16:56.379 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.374+0000 7f71e6ffd700 1 --2- 172.21.15.40:0/2968496269 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f71e81006a0 0x7f71e819bd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:56.379 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.374+0000 7f71e6ffd700 1 -- 172.21.15.40:0/2968496269 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f71e80ffd10 msgr2=0x7f71e8194cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:16:56.379 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.374+0000 7f71e6ffd700 1 --2- 172.21.15.40:0/2968496269 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f71e80ffd10 0x7f71e8194cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:56.379 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.374+0000 7f71e6ffd700 1 -- 172.21.15.40:0/2968496269 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71d00095f0 con 0x7f71e80feb20 2023-07-14T19:16:56.380 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.374+0000 7f71e6ffd700 1 --2- 172.21.15.40:0/2968496269 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f71e80feb20 0x7f71e8194770 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f71d00059d0 tx=0x7f71d0005740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:16:56.380 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.375+0000 7f71dffff700 1 -- 172.21.15.40:0/2968496269 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71d001d070 con 0x7f71e80feb20 2023-07-14T19:16:56.380 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.375+0000 7f71dffff700 1 -- 172.21.15.40:0/2968496269 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f71d00041a0 con 0x7f71e80feb20 2023-07-14T19:16:56.380 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.375+0000 7f71ed32f700 1 -- 172.21.15.40:0/2968496269 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f71e819c840 con 0x7f71e80feb20 2023-07-14T19:16:56.380 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.375+0000 7f71dffff700 1 -- 172.21.15.40:0/2968496269 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71d000f460 con 0x7f71e80feb20 2023-07-14T19:16:56.380 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.375+0000 7f71ed32f700 1 -- 172.21.15.40:0/2968496269 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f71e819cc60 con 0x7f71e80feb20 2023-07-14T19:16:56.380 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.376+0000 7f71ed32f700 1 -- 172.21.15.40:0/2968496269 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f71e810a310 con 0x7f71e80feb20 2023-07-14T19:16:56.381 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.377+0000 7f71dffff700 1 -- 172.21.15.40:0/2968496269 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7f71d0022600 con 0x7f71e80feb20 2023-07-14T19:16:56.381 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.378+0000 7f71dffff700 1 --2- 172.21.15.40:0/2968496269 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f71d40a1220 0x7f71d40a36d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:16:56.382 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.378+0000 7f71e67fc700 1 --2- 172.21.15.40:0/2968496269 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f71d40a1220 0x7f71d40a36d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:16:56.382 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.378+0000 7f71dffff700 1 -- 172.21.15.40:0/2968496269 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 8269+0+0 (secure 0 0 0) 0x7f71d00ce190 con 0x7f71e80feb20 2023-07-14T19:16:56.382 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.378+0000 7f71e67fc700 1 --2- 172.21.15.40:0/2968496269 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f71d40a1220 0x7f71d40a36d0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f71d800ba60 tx=0x7f71d8005d50 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:16:56.382 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.382+0000 7f71dffff700 1 -- 172.21.15.40:0/2968496269 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f71d00977f0 con 0x7f71e80feb20 2023-07-14T19:16:56.629 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.629+0000 7f71ed32f700 1 -- 172.21.15.40:0/2968496269 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f71e810a500 con 0x7f71e80feb20 2023-07-14T19:16:56.630 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.629+0000 7f71dffff700 1 -- 172.21.15.40:0/2968496269 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) v1 ==== 94+0+10877 (secure 0 0 0) 0x7f71d009ae10 con 0x7f71e80feb20 2023-07-14T19:16:56.630 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-07-14T19:16:56.630 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14523,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6835/501143718","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6834","nonce":501143718},{"type":"v1","addr":"172.21.15.40:6835","nonce":501143718}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14535,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6837/2634476117","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6836","nonce":2634476117},{"type":"v1","addr":"172.21.15.40:6837","nonce":2634476117}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":26,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2023-07-14T19:15:57.650710+0000","modified":"2023-07-14T19:16:05.624568+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24539,"mds_1":24508,"mds_2":24506,"mds_3":24490,"mds_4":24482},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14586":{"gid":14586,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.40:6839/959937545","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6838","nonce":959937545},{"type":"v1","addr":"172.21.15.40:6839","nonce":959937545}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_14598":{"gid":14598,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.40:6841/1590486077","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6840","nonce":1590486077},{"type":"v1","addr":"172.21.15.40:6841","nonce":1590486077}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24433":{"gid":24433,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.191:6833/31509862","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":31509862},{"type":"v1","addr":"172.21.15.191:6833","nonce":31509862}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24452":{"gid":24452,"name":"k","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.130:6833/3677452688","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6832","nonce":3677452688},{"type":"v1","addr":"172.21.15.130:6833","nonce":3677452688}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24478":{"gid":24478,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.191:6835/3712674634","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":3712674634},{"type":"v1","addr":"172.21.15.191:6835","nonce":3712674634}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24482":{"gid":24482,"name":"e","rank":4,"incarnation":24,"state":"up:active","state_seq":8,"addr":"172.21.15.130:6835/3088490808","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6834","nonce":3088490808},{"type":"v1","addr":"172.21.15.130:6835","nonce":3088490808}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24490":{"gid":24490,"name":"c","rank":3,"incarnation":22,"state":"up:active","state_seq":7,"addr":"172.21.15.191:6837/2406892975","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6836","nonce":2406892975},{"type":"v1","addr":"172.21.15.191:6837","nonce":2406892975}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24506":{"gid":24506,"name":"h","rank":2,"incarnation":20,"state":"up:active","state_seq":6,"addr":"172.21.15.130:6837/2604019369","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6836","nonce":2604019369},{"type":"v1","addr":"172.21.15.130:6837","nonce":2604019369}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24508":{"gid":24508,"name":"i","rank":1,"incarnation":18,"state":"up:active","state_seq":5,"addr":"172.21.15.191:6839/1977507302","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":1977507302},{"type":"v1","addr":"172.21.15.191:6839","nonce":1977507302}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24539":{"gid":24539,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.130:6839/2145479535","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6838","nonce":2145479535},{"type":"v1","addr":"172.21.15.130:6839","nonce":2145479535}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-07-14T19:16:56.632 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.632+0000 7f71ed32f700 1 -- 172.21.15.40:0/2968496269 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f71d40a1220 msgr2=0x7f71d40a36d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:16:56.633 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.632+0000 7f71ed32f700 1 --2- 172.21.15.40:0/2968496269 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f71d40a1220 0x7f71d40a36d0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f71d800ba60 tx=0x7f71d8005d50 comp rx=0 tx=0).stop 2023-07-14T19:16:56.633 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.632+0000 7f71ed32f700 1 -- 172.21.15.40:0/2968496269 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f71e80feb20 msgr2=0x7f71e8194770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:16:56.633 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.632+0000 7f71ed32f700 1 --2- 172.21.15.40:0/2968496269 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f71e80feb20 0x7f71e8194770 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f71d00059d0 tx=0x7f71d0005740 comp rx=0 tx=0).stop 2023-07-14T19:16:56.633 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.632+0000 7f71ed32f700 1 -- 172.21.15.40:0/2968496269 shutdown_connections 2023-07-14T19:16:56.633 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.632+0000 7f71ed32f700 1 --2- 172.21.15.40:0/2968496269 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f71d40a1220 0x7f71d40a36d0 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:56.633 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.632+0000 7f71ed32f700 1 --2- 172.21.15.40:0/2968496269 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f71e81006a0 0x7f71e819bd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:56.633 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.632+0000 7f71ed32f700 1 --2- 172.21.15.40:0/2968496269 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f71e80feb20 0x7f71e8194770 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:56.634 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.632+0000 7f71ed32f700 1 --2- 172.21.15.40:0/2968496269 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f71e80ffd10 0x7f71e8194cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:56.634 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.632+0000 7f71ed32f700 1 -- 172.21.15.40:0/2968496269 >> 172.21.15.40:0/2968496269 conn(0x7f71e80fa100 msgr2=0x7f71e80fc2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:16:56.634 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.632+0000 7f71ed32f700 1 -- 172.21.15.40:0/2968496269 shutdown_connections 2023-07-14T19:16:56.634 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.632+0000 7f71ed32f700 1 -- 172.21.15.40:0/2968496269 wait complete. 2023-07-14T19:16:56.634 INFO:teuthology.orchestra.run.smithi040.stderr:dumped fsmap epoch 26 2023-07-14T19:16:56.645 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 26, 'flags': 50, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2023-07-14T19:15:57.650710+0000', 'modified': '2023-07-14T19:16:05.624568+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 24539, 'mds_1': 24508, 'mds_2': 24506, 'mds_3': 24490, 'mds_4': 24482}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14586': {'gid': 14586, 'name': 'g', 'rank': 4, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.40:6839/959937545', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.40:6838', 'nonce': 959937545}, {'type': 'v1', 'addr': '172.21.15.40:6839', 'nonce': 959937545}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}, 'gid_14598': {'gid': 14598, 'name': 'a', 'rank': 3, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.40:6841/1590486077', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.40:6840', 'nonce': 1590486077}, {'type': 'v1', 'addr': '172.21.15.40:6841', 'nonce': 1590486077}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}, 'gid_24433': {'gid': 24433, 'name': 'l', 'rank': 2, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.191:6833/31509862', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.191:6832', 'nonce': 31509862}, {'type': 'v1', 'addr': '172.21.15.191:6833', 'nonce': 31509862}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}, 'gid_24452': {'gid': 24452, 'name': 'k', 'rank': 1, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.130:6833/3677452688', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.130:6832', 'nonce': 3677452688}, {'type': 'v1', 'addr': '172.21.15.130:6833', 'nonce': 3677452688}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}, 'gid_24478': {'gid': 24478, 'name': 'f', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.191:6835/3712674634', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.191:6834', 'nonce': 3712674634}, {'type': 'v1', 'addr': '172.21.15.191:6835', 'nonce': 3712674634}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}, 'gid_24482': {'gid': 24482, 'name': 'e', 'rank': 4, 'incarnation': 24, 'state': 'up:active', 'state_seq': 8, 'addr': '172.21.15.130:6835/3088490808', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.130:6834', 'nonce': 3088490808}, {'type': 'v1', 'addr': '172.21.15.130:6835', 'nonce': 3088490808}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}, 'gid_24490': {'gid': 24490, 'name': 'c', 'rank': 3, 'incarnation': 22, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.191:6837/2406892975', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.191:6836', 'nonce': 2406892975}, {'type': 'v1', 'addr': '172.21.15.191:6837', 'nonce': 2406892975}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}, 'gid_24506': {'gid': 24506, 'name': 'h', 'rank': 2, 'incarnation': 20, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.130:6837/2604019369', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.130:6836', 'nonce': 2604019369}, {'type': 'v1', 'addr': '172.21.15.130:6837', 'nonce': 2604019369}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}, 'gid_24508': {'gid': 24508, 'name': 'i', 'rank': 1, 'incarnation': 18, 'state': 'up:active', 'state_seq': 5, 'addr': '172.21.15.191:6839/1977507302', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.191:6838', 'nonce': 1977507302}, {'type': 'v1', 'addr': '172.21.15.191:6839', 'nonce': 1977507302}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}, 'gid_24539': {'gid': 24539, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.130:6839/2145479535', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.130:6838', 'nonce': 2145479535}, {'type': 'v1', 'addr': '172.21.15.130:6839', 'nonce': 2145479535}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2023-07-14T19:16:56.646 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 5/5 2023-07-14T19:16:56.646 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.e status 2023-07-14T19:16:56.775 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.772+0000 7fbc0b983700 1 -- 172.21.15.40:0/3467097335 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbc04100690 msgr2=0x7fbc04106f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:16:56.776 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.772+0000 7fbc0b983700 1 --2- 172.21.15.40:0/3467097335 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbc04100690 0x7fbc04106f50 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7fbbfc00b0a0 tx=0x7fbbfc00b3b0 comp rx=0 tx=0).stop 2023-07-14T19:16:56.776 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.773+0000 7fbc0b983700 1 -- 172.21.15.40:0/3467097335 shutdown_connections 2023-07-14T19:16:56.776 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.773+0000 7fbc0b983700 1 --2- 172.21.15.40:0/3467097335 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbc040feb10 0x7fbc040fef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:56.777 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.773+0000 7fbc0b983700 1 --2- 172.21.15.40:0/3467097335 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbc04100690 0x7fbc04106f50 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:56.777 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.773+0000 7fbc0b983700 1 --2- 172.21.15.40:0/3467097335 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fbc040ffd00 0x7fbc04100150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:56.777 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.773+0000 7fbc0b983700 1 -- 172.21.15.40:0/3467097335 >> 172.21.15.40:0/3467097335 conn(0x7fbc040fa130 msgr2=0x7fbc040fc560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:16:56.777 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.773+0000 7fbc0b983700 1 -- 172.21.15.40:0/3467097335 shutdown_connections 2023-07-14T19:16:56.777 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.773+0000 7fbc0b983700 1 -- 172.21.15.40:0/3467097335 wait complete. 2023-07-14T19:16:56.777 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.773+0000 7fbc0b983700 1 Processor -- start 2023-07-14T19:16:56.777 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.773+0000 7fbc0b983700 1 -- start start 2023-07-14T19:16:56.778 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.773+0000 7fbc0b983700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbc040feb10 0x7fbc04195a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:16:56.778 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.773+0000 7fbc0b983700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbc040ffd00 0x7fbc04195f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:16:56.778 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.774+0000 7fbc0b983700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fbc04100690 0x7fbc0419cfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:16:56.778 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.774+0000 7fbc0b983700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc0419d5a0 con 0x7fbc040feb10 2023-07-14T19:16:56.778 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.774+0000 7fbc0b983700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc0419d6e0 con 0x7fbc04100690 2023-07-14T19:16:56.778 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.774+0000 7fbc0b983700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc0419d870 con 0x7fbc040ffd00 2023-07-14T19:16:56.778 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.774+0000 7fbc0971f700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbc040feb10 0x7fbc04195a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:16:56.779 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.774+0000 7fbc0971f700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbc040feb10 0x7fbc04195a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:33696/0 (socket says 172.21.15.40:33696) 2023-07-14T19:16:56.779 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.774+0000 7fbc0971f700 1 -- 172.21.15.40:0/3446665732 learned_addr learned my addr 172.21.15.40:0/3446665732 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-07-14T19:16:56.779 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.774+0000 7fbc09f20700 1 --2- 172.21.15.40:0/3446665732 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fbc04100690 0x7fbc0419cfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:16:56.779 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.774+0000 7fbc08f1e700 1 --2- 172.21.15.40:0/3446665732 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbc040ffd00 0x7fbc04195f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:16:56.779 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.774+0000 7fbc0971f700 1 -- 172.21.15.40:0/3446665732 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbc040ffd00 msgr2=0x7fbc04195f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:16:56.779 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.774+0000 7fbc0971f700 1 --2- 172.21.15.40:0/3446665732 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbc040ffd00 0x7fbc04195f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:56.780 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.774+0000 7fbc0971f700 1 -- 172.21.15.40:0/3446665732 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fbc04100690 msgr2=0x7fbc0419cfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:16:56.780 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.774+0000 7fbc0971f700 1 --2- 172.21.15.40:0/3446665732 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fbc04100690 0x7fbc0419cfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:56.780 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.774+0000 7fbc0971f700 1 -- 172.21.15.40:0/3446665732 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbbfc009d00 con 0x7fbc040feb10 2023-07-14T19:16:56.780 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.774+0000 7fbc08f1e700 1 --2- 172.21.15.40:0/3446665732 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbc040ffd00 0x7fbc04195f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T19:16:56.780 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.774+0000 7fbc0971f700 1 --2- 172.21.15.40:0/3446665732 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbc040feb10 0x7fbc04195a20 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7fbc0000c970 tx=0x7fbc0000cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:16:56.780 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.774+0000 7fbbfa7fc700 1 -- 172.21.15.40:0/3446665732 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc00004220 con 0x7fbc040feb10 2023-07-14T19:16:56.780 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.774+0000 7fbbfa7fc700 1 -- 172.21.15.40:0/3446665732 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fbc00007d60 con 0x7fbc040feb10 2023-07-14T19:16:56.781 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.774+0000 7fbc0b983700 1 -- 172.21.15.40:0/3446665732 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbc0419db50 con 0x7fbc040feb10 2023-07-14T19:16:56.781 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.774+0000 7fbbfa7fc700 1 -- 172.21.15.40:0/3446665732 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc00003c90 con 0x7fbc040feb10 2023-07-14T19:16:56.781 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.774+0000 7fbc0b983700 1 -- 172.21.15.40:0/3446665732 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbc0419e0b0 con 0x7fbc040feb10 2023-07-14T19:16:56.787 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.777+0000 7fbbfa7fc700 1 -- 172.21.15.40:0/3446665732 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7fbc00004380 con 0x7fbc040feb10 2023-07-14T19:16:56.787 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.777+0000 7fbbfa7fc700 1 --2- 172.21.15.40:0/3446665732 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7fbbf00a1260 0x7fbbf00a3710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:16:56.787 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.777+0000 7fbc08f1e700 1 --2- 172.21.15.40:0/3446665732 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7fbbf00a1260 0x7fbbf00a3710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:16:56.787 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.777+0000 7fbbfa7fc700 1 -- 172.21.15.40:0/3446665732 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 8269+0+0 (secure 0 0 0) 0x7fbc000cc580 con 0x7fbc040feb10 2023-07-14T19:16:56.788 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.778+0000 7fbc08f1e700 1 --2- 172.21.15.40:0/3446665732 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7fbbf00a1260 0x7fbbf00a3710 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fbbf4005fd0 tx=0x7fbbf4005e30 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:16:56.788 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.778+0000 7fbc17b1bb80 1 Processor -- start 2023-07-14T19:16:56.788 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.778+0000 7fbc17b1bb80 1 -- start start 2023-07-14T19:16:56.788 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.778+0000 7fbc17b1bb80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5615c222ff10 0x5615c2230320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:16:56.788 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.778+0000 7fbc17b1bb80 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5615c2230860 0x5615c2230e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:16:56.788 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.778+0000 7fbc17b1bb80 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5615c2231460 0x5615c22318d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:16:56.788 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.778+0000 7fbc17b1bb80 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x5615c2134c60 con 0x5615c2231460 2023-07-14T19:16:56.789 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.778+0000 7fbc17b1bb80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5615c21879f0 con 0x5615c222ff10 2023-07-14T19:16:56.789 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.778+0000 7fbc17b1bb80 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x5615c2180e80 con 0x5615c2230860 2023-07-14T19:16:56.789 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.779+0000 7fbc09f20700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5615c2231460 0x5615c22318d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:16:56.789 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.779+0000 7fbc09f20700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5615c2231460 0x5615c22318d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:33708/0 (socket says 172.21.15.40:33708) 2023-07-14T19:16:56.789 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.779+0000 7fbc09f20700 1 -- 172.21.15.40:0/2713030003 learned_addr learned my addr 172.21.15.40:0/2713030003 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-07-14T19:16:56.789 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.779+0000 7fbc0971f700 1 --2- 172.21.15.40:0/2713030003 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5615c222ff10 0x5615c2230320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:16:56.789 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.779+0000 7fbc08f1e700 1 --2- 172.21.15.40:0/2713030003 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5615c2230860 0x5615c2230e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:16:56.789 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.779+0000 7fbc09f20700 1 -- 172.21.15.40:0/2713030003 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5615c2230860 msgr2=0x5615c2230e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T19:16:56.790 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.779+0000 7fbc09f20700 1 --2- 172.21.15.40:0/2713030003 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5615c2230860 0x5615c2230e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:56.790 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.779+0000 7fbc09f20700 1 -- 172.21.15.40:0/2713030003 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5615c222ff10 msgr2=0x5615c2230320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T19:16:56.790 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.779+0000 7fbc09f20700 1 --2- 172.21.15.40:0/2713030003 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5615c222ff10 0x5615c2230320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:56.790 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.779+0000 7fbc09f20700 1 -- 172.21.15.40:0/2713030003 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5615c1faed90 con 0x5615c2231460 2023-07-14T19:16:56.790 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.779+0000 7fbc0971f700 1 --2- 172.21.15.40:0/2713030003 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5615c222ff10 0x5615c2230320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T19:16:56.790 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.779+0000 7fbc09f20700 1 --2- 172.21.15.40:0/2713030003 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5615c2231460 0x5615c22318d0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7fbbfc007a60 tx=0x7fbbfc0075d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=38171fec8c05fcb2 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:16:56.790 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.779+0000 7fbbef7fe700 1 -- 172.21.15.40:0/2713030003 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbfc00e040 con 0x5615c2231460 2023-07-14T19:16:56.791 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.779+0000 7fbbef7fe700 1 -- 172.21.15.40:0/2713030003 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fbbfc009b80 con 0x5615c2231460 2023-07-14T19:16:56.791 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.779+0000 7fbbef7fe700 1 -- 172.21.15.40:0/2713030003 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbfc01cb40 con 0x5615c2231460 2023-07-14T19:16:56.791 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.780+0000 7fbc17b1bb80 1 -- 172.21.15.40:0/2713030003 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5615c2231460 msgr2=0x5615c22318d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:16:56.791 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.780+0000 7fbc17b1bb80 1 --2- 172.21.15.40:0/2713030003 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5615c2231460 0x5615c22318d0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7fbbfc007a60 tx=0x7fbbfc0075d0 comp rx=0 tx=0).stop 2023-07-14T19:16:56.791 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.780+0000 7fbc17b1bb80 1 -- 172.21.15.40:0/2713030003 shutdown_connections 2023-07-14T19:16:56.791 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.780+0000 7fbc17b1bb80 1 --2- 172.21.15.40:0/2713030003 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5615c2231460 0x5615c22318d0 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:56.791 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.780+0000 7fbc17b1bb80 1 --2- 172.21.15.40:0/2713030003 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5615c222ff10 0x5615c2230320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:56.791 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.780+0000 7fbc17b1bb80 1 --2- 172.21.15.40:0/2713030003 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5615c2230860 0x5615c2230e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:56.792 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.780+0000 7fbc17b1bb80 1 -- 172.21.15.40:0/2713030003 >> 172.21.15.40:0/2713030003 conn(0x5615c2225630 msgr2=0x5615c222a980 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:16:56.792 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.780+0000 7fbc17b1bb80 1 -- 172.21.15.40:0/2713030003 shutdown_connections 2023-07-14T19:16:56.792 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.780+0000 7fbc17b1bb80 1 -- 172.21.15.40:0/2713030003 wait complete. 2023-07-14T19:16:56.792 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.780+0000 7fbc17b1bb80 1 -- start start 2023-07-14T19:16:56.792 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.781+0000 7fbc17b1bb80 1 Processor -- start 2023-07-14T19:16:56.792 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.781+0000 7fbc17b1bb80 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5615c222ff10 0x5615c2239890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:16:56.792 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.781+0000 7fbc17b1bb80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5615c2230860 0x5615c2239dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:16:56.793 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.781+0000 7fbc17b1bb80 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5615c2231460 0x5615c2240e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:16:56.793 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.781+0000 7fbc17b1bb80 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x5615c2120a70 con 0x5615c2231460 2023-07-14T19:16:56.793 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.781+0000 7fbc17b1bb80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5615c2120770 con 0x5615c2230860 2023-07-14T19:16:56.793 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.781+0000 7fbc17b1bb80 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x5615c20048d0 con 0x5615c222ff10 2023-07-14T19:16:56.793 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.781+0000 7fbc17b1bb80 10 client.0 _ll_register_callbacks cb 0x5615c211e710 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-14T19:16:56.793 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.781+0000 7fbc09f20700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5615c2231460 0x5615c2240e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:16:56.793 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.781+0000 7fbc09f20700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5615c2231460 0x5615c2240e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:33714/0 (socket says 172.21.15.40:33714) 2023-07-14T19:16:56.794 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.781+0000 7fbc09f20700 1 -- 172.21.15.40:0/17126259 learned_addr learned my addr 172.21.15.40:0/17126259 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-07-14T19:16:56.794 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.781+0000 7fbc0971f700 1 --2- 172.21.15.40:0/17126259 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5615c222ff10 0x5615c2239890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:16:56.794 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.781+0000 7fbc08f1e700 1 --2- 172.21.15.40:0/17126259 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5615c2230860 0x5615c2239dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:16:56.794 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.781+0000 7fbc09f20700 1 -- 172.21.15.40:0/17126259 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5615c222ff10 msgr2=0x5615c2239890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T19:16:56.794 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.781+0000 7fbc09f20700 1 --2- 172.21.15.40:0/17126259 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5615c222ff10 0x5615c2239890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:56.794 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.781+0000 7fbc09f20700 1 -- 172.21.15.40:0/17126259 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5615c2230860 msgr2=0x5615c2239dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T19:16:56.795 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.781+0000 7fbc09f20700 1 --2- 172.21.15.40:0/17126259 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5615c2230860 0x5615c2239dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:56.795 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.781+0000 7fbc09f20700 1 -- 172.21.15.40:0/17126259 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbc04107880 con 0x5615c2231460 2023-07-14T19:16:56.795 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.781+0000 7fbc0971f700 1 --2- 172.21.15.40:0/17126259 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5615c222ff10 0x5615c2239890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T19:16:56.795 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.781+0000 7fbc08f1e700 1 --2- 172.21.15.40:0/17126259 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5615c2230860 0x5615c2239dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-14T19:16:56.795 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.781+0000 7fbc09f20700 1 --2- 172.21.15.40:0/17126259 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5615c2231460 0x5615c2240e10 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7fbbfc0062a0 tx=0x7fbbfc003ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=491a407a109662d9 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:16:56.795 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.781+0000 7fbbecff9700 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2023-07-14T19:16:56.795 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.782+0000 7fbbecff9700 1 -- 172.21.15.40:0/17126259 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbfc00e040 con 0x5615c2231460 2023-07-14T19:16:56.796 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.782+0000 7fbbecff9700 1 -- 172.21.15.40:0/17126259 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(83..83 src has 1..83) v4 ==== 8269+0+0 (secure 0 0 0) 0x7fbbfc01cad0 con 0x5615c2231460 2023-07-14T19:16:56.796 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.782+0000 7fbc17b1bb80 1 -- 172.21.15.40:0/17126259 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5615c21fed50 con 0x5615c2231460 2023-07-14T19:16:56.796 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.782+0000 7fbbecff9700 1 -- 172.21.15.40:0/17126259 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7fbbfc01c4e0 con 0x5615c2231460 2023-07-14T19:16:56.796 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.782+0000 7fbc17b1bb80 10 client.15006 fetch_fsmap learned FSMap version 26 2023-07-14T19:16:56.796 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.782+0000 7fbc17b1bb80 1 -- 172.21.15.40:0/17126259 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) v3 -- 0x5615c1f97c10 con 0x5615c2231460 2023-07-14T19:16:56.796 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.782+0000 7fbbecff9700 1 -- 172.21.15.40:0/17126259 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x5615c21fed50 con 0x5615c2231460 2023-07-14T19:16:56.796 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.782+0000 7fbbecff9700 1 -- 172.21.15.40:0/17126259 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 26) v1 ==== 7462+0+0 (secure 0 0 0) 0x7fbbfc019070 con 0x5615c2231460 2023-07-14T19:16:56.797 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.782+0000 7fbc17b1bb80 10 client.15006 fetch_fsmap finished waiting for FSMap version 26 2023-07-14T19:16:56.797 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.782+0000 7fbc17b1bb80 10 client.15006 resolve_mds: resolved name 'e' to daemon mds.e 2023-07-14T19:16:56.797 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.782+0000 7fbc17b1bb80 20 client.15006 populate_metadata read hostname 'smithi040' 2023-07-14T19:16:56.797 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.782+0000 7fbc17b1bb80 1 --2- 172.21.15.40:0/17126259 >> [v2:172.21.15.130:6834/3088490808,v1:172.21.15.130:6835/3088490808] conn(0x5615c22447f0 0x5615c2243d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:16:56.797 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.782+0000 7fbc17b1bb80 4 client.15006 mds_command: new command op to 24482 tid=0[{"prefix": "get_command_descriptions"}] 2023-07-14T19:16:56.797 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.782+0000 7fbc17b1bb80 1 -- 172.21.15.40:0/17126259 --> [v2:172.21.15.130:6834/3088490808,v1:172.21.15.130:6835/3088490808] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5615c2179e60 con 0x5615c22447f0 2023-07-14T19:16:56.797 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.783+0000 7fbc0971f700 1 --2- 172.21.15.40:0/17126259 >> [v2:172.21.15.130:6834/3088490808,v1:172.21.15.130:6835/3088490808] conn(0x5615c22447f0 0x5615c2243d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:16:56.797 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.783+0000 7fbc0971f700 1 --2- 172.21.15.40:0/17126259 >> [v2:172.21.15.130:6834/3088490808,v1:172.21.15.130:6835/3088490808] conn(0x5615c22447f0 0x5615c2243d40 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=c267b4ce43830572 server_cookie=7506242af84a35b6 in_seq=0 out_seq=0 2023-07-14T19:16:56.798 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.783+0000 7fbbecff9700 10 client.15006 ms_handle_connect on v2:172.21.15.130:6834/3088490808 2023-07-14T19:16:56.798 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.784+0000 7fbbecff9700 1 -- 172.21.15.40:0/17126259 <== mds.4 v2:172.21.15.130:6834/3088490808 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18017 (crc 0 0 0) 0x5615c2179e60 con 0x5615c22447f0 2023-07-14T19:16:56.798 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.784+0000 7fbbecff9700 10 client.15006 handle_command_reply: tid=0 2023-07-14T19:16:56.798 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.784+0000 7fbc17b1bb80 1 client.15006 shutdown 2023-07-14T19:16:56.798 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.784+0000 7fbc17b1bb80 1 -- 172.21.15.40:0/17126259 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5615c2231460 msgr2=0x5615c2240e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:16:56.798 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.784+0000 7fbc17b1bb80 1 --2- 172.21.15.40:0/17126259 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5615c2231460 0x5615c2240e10 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7fbbfc0062a0 tx=0x7fbbfc003ce0 comp rx=0 tx=0).stop 2023-07-14T19:16:56.798 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.784+0000 7fbc17b1bb80 1 -- 172.21.15.40:0/17126259 shutdown_connections 2023-07-14T19:16:56.799 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.784+0000 7fbc17b1bb80 1 --2- 172.21.15.40:0/17126259 >> [v2:172.21.15.130:6834/3088490808,v1:172.21.15.130:6835/3088490808] conn(0x5615c22447f0 0x5615c2243d40 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:56.799 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.784+0000 7fbc17b1bb80 1 --2- 172.21.15.40:0/17126259 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5615c222ff10 0x5615c2239890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:56.799 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.784+0000 7fbc17b1bb80 1 --2- 172.21.15.40:0/17126259 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5615c2231460 0x5615c2240e10 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:56.799 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.784+0000 7fbc17b1bb80 1 --2- 172.21.15.40:0/17126259 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5615c2230860 0x5615c2239dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:56.799 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.784+0000 7fbc17b1bb80 1 -- 172.21.15.40:0/17126259 >> 172.21.15.40:0/17126259 conn(0x5615c2225630 msgr2=0x5615c2229600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:16:56.799 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.784+0000 7fbbecff9700 0 client.15006 ms_handle_reset on v2:172.21.15.130:6834/3088490808 2023-07-14T19:16:56.799 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.784+0000 7fbc17b1bb80 1 -- 172.21.15.40:0/17126259 shutdown_connections 2023-07-14T19:16:56.799 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.784+0000 7fbc17b1bb80 1 -- 172.21.15.40:0/17126259 wait complete. 2023-07-14T19:16:56.800 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.785+0000 7fbc17b1bb80 20 client.15006 trim_cache size 0 max 16384 2023-07-14T19:16:56.805 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.802+0000 7fbc17b1bb80 1 Processor -- start 2023-07-14T19:16:56.805 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.802+0000 7fbc17b1bb80 1 -- start start 2023-07-14T19:16:56.805 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.803+0000 7fbc17b1bb80 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5615c222ff10 0x5615c222a0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:16:56.805 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.803+0000 7fbc17b1bb80 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5615c2230860 0x5615c2240dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:16:56.805 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.803+0000 7fbc17b1bb80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5615c2231460 0x5615c22415e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:16:56.805 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.803+0000 7fbc17b1bb80 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x5615c212a470 con 0x5615c222ff10 2023-07-14T19:16:56.806 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.803+0000 7fbc17b1bb80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5615c214f790 con 0x5615c2231460 2023-07-14T19:16:56.806 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.803+0000 7fbc17b1bb80 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x5615c213f480 con 0x5615c2230860 2023-07-14T19:16:56.806 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.803+0000 7fbc0971f700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5615c222ff10 0x5615c222a0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:16:56.806 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.803+0000 7fbc0971f700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5615c222ff10 0x5615c222a0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:33718/0 (socket says 172.21.15.40:33718) 2023-07-14T19:16:56.806 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.803+0000 7fbc0971f700 1 -- 172.21.15.40:0/1439382876 learned_addr learned my addr 172.21.15.40:0/1439382876 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-07-14T19:16:56.806 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.803+0000 7fbc08f1e700 1 --2- 172.21.15.40:0/1439382876 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5615c2230860 0x5615c2240dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:16:56.806 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.803+0000 7fbc09f20700 1 --2- 172.21.15.40:0/1439382876 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5615c2231460 0x5615c22415e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:16:56.807 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.803+0000 7fbc0971f700 1 -- 172.21.15.40:0/1439382876 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5615c2230860 msgr2=0x5615c2240dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T19:16:56.807 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.803+0000 7fbc0971f700 1 --2- 172.21.15.40:0/1439382876 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5615c2230860 0x5615c2240dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:56.807 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.803+0000 7fbc0971f700 1 -- 172.21.15.40:0/1439382876 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5615c2231460 msgr2=0x5615c22415e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T19:16:56.807 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.803+0000 7fbc0971f700 1 --2- 172.21.15.40:0/1439382876 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5615c2231460 0x5615c22415e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:56.807 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.803+0000 7fbc0971f700 1 -- 172.21.15.40:0/1439382876 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5615c1fd5200 con 0x5615c222ff10 2023-07-14T19:16:56.807 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.803+0000 7fbc08f1e700 1 --2- 172.21.15.40:0/1439382876 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5615c2230860 0x5615c2240dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T19:16:56.807 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.803+0000 7fbc09f20700 1 --2- 172.21.15.40:0/1439382876 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5615c2231460 0x5615c22415e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-14T19:16:56.807 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.803+0000 7fbc0971f700 1 --2- 172.21.15.40:0/1439382876 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5615c222ff10 0x5615c222a0a0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7fbc0000c940 tx=0x7fbc00004680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6ef95e8c2f78b28b server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:16:56.808 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.803+0000 7fbbeffff700 1 -- 172.21.15.40:0/1439382876 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc000a1c60 con 0x5615c222ff10 2023-07-14T19:16:56.808 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.803+0000 7fbbeffff700 1 -- 172.21.15.40:0/1439382876 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fbc00097950 con 0x5615c222ff10 2023-07-14T19:16:56.808 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.803+0000 7fbbeffff700 1 -- 172.21.15.40:0/1439382876 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc00099a70 con 0x5615c222ff10 2023-07-14T19:16:56.808 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.804+0000 7fbc17b1bb80 1 -- 172.21.15.40:0/1439382876 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5615c222ff10 msgr2=0x5615c222a0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:16:56.808 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.804+0000 7fbc17b1bb80 1 --2- 172.21.15.40:0/1439382876 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5615c222ff10 0x5615c222a0a0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7fbc0000c940 tx=0x7fbc00004680 comp rx=0 tx=0).stop 2023-07-14T19:16:56.808 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.804+0000 7fbc17b1bb80 1 -- 172.21.15.40:0/1439382876 shutdown_connections 2023-07-14T19:16:56.808 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.804+0000 7fbc17b1bb80 1 --2- 172.21.15.40:0/1439382876 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5615c2231460 0x5615c22415e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:56.809 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.804+0000 7fbc17b1bb80 1 --2- 172.21.15.40:0/1439382876 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5615c222ff10 0x5615c222a0a0 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:56.809 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.804+0000 7fbc17b1bb80 1 --2- 172.21.15.40:0/1439382876 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5615c2230860 0x5615c2240dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:56.809 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.804+0000 7fbc17b1bb80 1 -- 172.21.15.40:0/1439382876 >> 172.21.15.40:0/1439382876 conn(0x5615c2225630 msgr2=0x5615c2226b00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:16:56.809 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.804+0000 7fbc17b1bb80 1 -- 172.21.15.40:0/1439382876 shutdown_connections 2023-07-14T19:16:56.809 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.804+0000 7fbc17b1bb80 1 -- 172.21.15.40:0/1439382876 wait complete. 2023-07-14T19:16:56.809 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.804+0000 7fbc17b1bb80 1 -- start start 2023-07-14T19:16:56.812 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.805+0000 7fbc17b1bb80 1 Processor -- start 2023-07-14T19:16:56.812 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.805+0000 7fbc17b1bb80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5615c222ff10 0x5615c2236530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:16:56.813 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.805+0000 7fbc17b1bb80 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5615c2230860 0x5615c2240dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:16:56.813 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.805+0000 7fbc17b1bb80 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5615c2231460 0x5615c2236af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:16:56.813 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.805+0000 7fbc17b1bb80 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x5615c2298470 con 0x5615c2230860 2023-07-14T19:16:56.813 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.805+0000 7fbc17b1bb80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5615c22985b0 con 0x5615c222ff10 2023-07-14T19:16:56.813 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.805+0000 7fbc17b1bb80 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x5615c2298740 con 0x5615c2231460 2023-07-14T19:16:56.813 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.805+0000 7fbc17b1bb80 10 client.0 _ll_register_callbacks cb 0x7fbbfc007520 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-14T19:16:56.813 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.805+0000 7fbc08f1e700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5615c2230860 0x5615c2240dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:16:56.814 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.805+0000 7fbc08f1e700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5615c2230860 0x5615c2240dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:33726/0 (socket says 172.21.15.40:33726) 2023-07-14T19:16:56.814 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.805+0000 7fbc08f1e700 1 -- 172.21.15.40:0/1010036764 learned_addr learned my addr 172.21.15.40:0/1010036764 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-07-14T19:16:56.814 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.805+0000 7fbc09f20700 1 --2- 172.21.15.40:0/1010036764 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5615c2231460 0x5615c2236af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:16:56.814 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.805+0000 7fbc0971f700 1 --2- 172.21.15.40:0/1010036764 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5615c222ff10 0x5615c2236530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:16:56.814 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.805+0000 7fbc08f1e700 1 -- 172.21.15.40:0/1010036764 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5615c2231460 msgr2=0x5615c2236af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T19:16:56.814 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.805+0000 7fbc08f1e700 1 --2- 172.21.15.40:0/1010036764 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5615c2231460 0x5615c2236af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:56.814 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.805+0000 7fbc08f1e700 1 -- 172.21.15.40:0/1010036764 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5615c222ff10 msgr2=0x5615c2236530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T19:16:56.815 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.805+0000 7fbc08f1e700 1 --2- 172.21.15.40:0/1010036764 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5615c222ff10 0x5615c2236530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:56.815 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.805+0000 7fbc08f1e700 1 -- 172.21.15.40:0/1010036764 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbc0419e0b0 con 0x5615c2230860 2023-07-14T19:16:56.815 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.805+0000 7fbc0971f700 1 --2- 172.21.15.40:0/1010036764 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5615c222ff10 0x5615c2236530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T19:16:56.815 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.805+0000 7fbc08f1e700 1 --2- 172.21.15.40:0/1010036764 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5615c2230860 0x5615c2240dd0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7fbbf400aab0 tx=0x7fbbf4004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c538fc3fab90c3a5 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:16:56.815 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.805+0000 7fbbecff9700 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2023-07-14T19:16:56.815 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.806+0000 7fbbecff9700 1 -- 172.21.15.40:0/1010036764 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbf4020860 con 0x5615c2230860 2023-07-14T19:16:56.815 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.806+0000 7fbc17b1bb80 1 -- 172.21.15.40:0/1010036764 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5615c20da4a0 con 0x5615c2230860 2023-07-14T19:16:56.815 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.806+0000 7fbbecff9700 1 -- 172.21.15.40:0/1010036764 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(83..83 src has 1..83) v4 ==== 8269+0+0 (secure 0 0 0) 0x7fbbf40209c0 con 0x5615c2230860 2023-07-14T19:16:56.816 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.806+0000 7fbbecff9700 1 -- 172.21.15.40:0/1010036764 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7fbbf4020bc0 con 0x5615c2230860 2023-07-14T19:16:56.816 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.806+0000 7fbc17b1bb80 10 client.15012 fetch_fsmap learned FSMap version 26 2023-07-14T19:16:56.816 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.806+0000 7fbc17b1bb80 1 -- 172.21.15.40:0/1010036764 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) v3 -- 0x5615c1fad260 con 0x5615c2230860 2023-07-14T19:16:56.816 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.806+0000 7fbbecff9700 1 -- 172.21.15.40:0/1010036764 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x5615c20da4a0 con 0x5615c2230860 2023-07-14T19:16:56.816 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.806+0000 7fbbecff9700 1 -- 172.21.15.40:0/1010036764 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 26) v1 ==== 7462+0+0 (secure 0 0 0) 0x7fbbf4032050 con 0x5615c2230860 2023-07-14T19:16:56.816 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.806+0000 7fbc17b1bb80 10 client.15012 fetch_fsmap finished waiting for FSMap version 26 2023-07-14T19:16:56.816 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.806+0000 7fbc17b1bb80 10 client.15012 resolve_mds: resolved name 'e' to daemon mds.e 2023-07-14T19:16:56.817 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.806+0000 7fbc17b1bb80 20 client.15012 populate_metadata read hostname 'smithi040' 2023-07-14T19:16:56.817 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.806+0000 7fbc17b1bb80 1 --2- 172.21.15.40:0/1010036764 >> [v2:172.21.15.130:6834/3088490808,v1:172.21.15.130:6835/3088490808] conn(0x5615c229d480 0x5615c2299df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:16:56.817 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.806+0000 7fbc17b1bb80 4 client.15012 mds_command: new command op to 24482 tid=0[{"prefix": "status"}] 2023-07-14T19:16:56.817 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.806+0000 7fbc17b1bb80 1 -- 172.21.15.40:0/1010036764 --> [v2:172.21.15.130:6834/3088490808,v1:172.21.15.130:6835/3088490808] -- command(tid 0: {"prefix": "status"}) v1 -- 0x5615c213c000 con 0x5615c229d480 2023-07-14T19:16:56.817 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.807+0000 7fbc0971f700 1 --2- 172.21.15.40:0/1010036764 >> [v2:172.21.15.130:6834/3088490808,v1:172.21.15.130:6835/3088490808] conn(0x5615c229d480 0x5615c2299df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:16:56.817 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.807+0000 7fbbecff9700 10 client.15012 ms_handle_connect on v2:172.21.15.130:6834/3088490808 2023-07-14T19:16:56.817 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.807+0000 7fbc0971f700 1 --2- 172.21.15.40:0/1010036764 >> [v2:172.21.15.130:6834/3088490808,v1:172.21.15.130:6835/3088490808] conn(0x5615c229d480 0x5615c2299df0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=e9dbd53917ab8f7e server_cookie=52091cf06072c9c4 in_seq=0 out_seq=0 2023-07-14T19:16:56.817 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.807+0000 7fbbecff9700 1 -- 172.21.15.40:0/1010036764 <== mds.4 v2:172.21.15.130:6834/3088490808 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+326 (crc 0 0 0) 0x5615c213c000 con 0x5615c229d480 2023-07-14T19:16:56.818 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.807+0000 7fbbecff9700 10 client.15012 handle_command_reply: tid=0 2023-07-14T19:16:56.818 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.807+0000 7fbc17b1bb80 1 client.15012 shutdown 2023-07-14T19:16:56.818 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.807+0000 7fbc17b1bb80 1 -- 172.21.15.40:0/1010036764 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5615c2230860 msgr2=0x5615c2240dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:16:56.818 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.807+0000 7fbc17b1bb80 1 --2- 172.21.15.40:0/1010036764 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5615c2230860 0x5615c2240dd0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7fbbf400aab0 tx=0x7fbbf4004750 comp rx=0 tx=0).stop 2023-07-14T19:16:56.818 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.807+0000 7fbc17b1bb80 1 -- 172.21.15.40:0/1010036764 shutdown_connections 2023-07-14T19:16:56.818 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.807+0000 7fbc17b1bb80 1 --2- 172.21.15.40:0/1010036764 >> [v2:172.21.15.130:6834/3088490808,v1:172.21.15.130:6835/3088490808] conn(0x5615c229d480 0x5615c2299df0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:56.818 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.807+0000 7fbc17b1bb80 1 --2- 172.21.15.40:0/1010036764 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5615c2231460 0x5615c2236af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:56.819 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.807+0000 7fbbecff9700 0 client.15012 ms_handle_reset on v2:172.21.15.130:6834/3088490808 2023-07-14T19:16:56.819 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.807+0000 7fbc17b1bb80 1 --2- 172.21.15.40:0/1010036764 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5615c222ff10 0x5615c2236530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:56.819 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.807+0000 7fbc17b1bb80 1 --2- 172.21.15.40:0/1010036764 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5615c2230860 0x5615c2240dd0 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:56.819 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.807+0000 7fbc17b1bb80 1 -- 172.21.15.40:0/1010036764 >> 172.21.15.40:0/1010036764 conn(0x5615c2225630 msgr2=0x5615c222a0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:16:56.819 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.808+0000 7fbc17b1bb80 1 -- 172.21.15.40:0/1010036764 shutdown_connections 2023-07-14T19:16:56.819 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.808+0000 7fbc17b1bb80 1 -- 172.21.15.40:0/1010036764 wait complete. 2023-07-14T19:16:56.819 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.808+0000 7fbc17b1bb80 20 client.15012 trim_cache size 0 max 16384 2023-07-14T19:16:56.820 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2023-07-14T19:16:56.820 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_fsid": "adc52f04-2279-11ee-9b33-001a4aab830c", 2023-07-14T19:16:56.820 INFO:teuthology.orchestra.run.smithi040.stdout: "whoami": 4, 2023-07-14T19:16:56.820 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 24482, 2023-07-14T19:16:56.820 INFO:teuthology.orchestra.run.smithi040.stdout: "want_state": "up:active", 2023-07-14T19:16:56.820 INFO:teuthology.orchestra.run.smithi040.stdout: "state": "up:active", 2023-07-14T19:16:56.821 INFO:teuthology.orchestra.run.smithi040.stdout: "fs_name": "cephfs", 2023-07-14T19:16:56.821 INFO:teuthology.orchestra.run.smithi040.stdout: "rank_uptime": 52.198443976, 2023-07-14T19:16:56.821 INFO:teuthology.orchestra.run.smithi040.stdout: "mdsmap_epoch": 26, 2023-07-14T19:16:56.821 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap_epoch": 83, 2023-07-14T19:16:56.821 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap_epoch_barrier": 81, 2023-07-14T19:16:56.821 INFO:teuthology.orchestra.run.smithi040.stdout: "uptime": 79.799545295000001 2023-07-14T19:16:56.821 INFO:teuthology.orchestra.run.smithi040.stdout:} 2023-07-14T19:16:56.822 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.808+0000 7fbc0b983700 1 -- 172.21.15.40:0/3446665732 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7fbbf00a1260 msgr2=0x7fbbf00a3710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:16:56.823 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.808+0000 7fbc0b983700 1 --2- 172.21.15.40:0/3446665732 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7fbbf00a1260 0x7fbbf00a3710 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fbbf4005fd0 tx=0x7fbbf4005e30 comp rx=0 tx=0).stop 2023-07-14T19:16:56.823 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.808+0000 7fbc0b983700 1 -- 172.21.15.40:0/3446665732 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbc040feb10 msgr2=0x7fbc04195a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:16:56.823 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.808+0000 7fbc0b983700 1 --2- 172.21.15.40:0/3446665732 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbc040feb10 0x7fbc04195a20 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7fbc0000c970 tx=0x7fbc0000cc80 comp rx=0 tx=0).stop 2023-07-14T19:16:56.823 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.808+0000 7fbc0b983700 1 -- 172.21.15.40:0/3446665732 shutdown_connections 2023-07-14T19:16:56.823 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.808+0000 7fbc0b983700 1 --2- 172.21.15.40:0/3446665732 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7fbbf00a1260 0x7fbbf00a3710 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:56.823 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.808+0000 7fbc0b983700 1 --2- 172.21.15.40:0/3446665732 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fbc04100690 0x7fbc0419cfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:56.823 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.808+0000 7fbc0b983700 1 --2- 172.21.15.40:0/3446665732 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbc040feb10 0x7fbc04195a20 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:56.824 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.808+0000 7fbc0b983700 1 --2- 172.21.15.40:0/3446665732 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbc040ffd00 0x7fbc04195f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:56.824 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.808+0000 7fbc0b983700 1 -- 172.21.15.40:0/3446665732 >> 172.21.15.40:0/3446665732 conn(0x7fbc040fa130 msgr2=0x7fbc04067780 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:16:56.824 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.808+0000 7fbc0b983700 1 -- 172.21.15.40:0/3446665732 shutdown_connections 2023-07-14T19:16:56.824 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.808+0000 7fbc0b983700 1 -- 172.21.15.40:0/3446665732 wait complete. 2023-07-14T19:16:56.824 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2023-07-14T19:16:56.952 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.951+0000 7fbee102a700 1 -- 172.21.15.40:0/2012303361 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbedc0feb10 msgr2=0x7fbedc0fef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:16:56.952 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.951+0000 7fbee102a700 1 --2- 172.21.15.40:0/2012303361 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbedc0feb10 0x7fbedc0fef20 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7fbec4009990 tx=0x7fbec4009ca0 comp rx=0 tx=0).stop 2023-07-14T19:16:56.952 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.951+0000 7fbee102a700 1 -- 172.21.15.40:0/2012303361 shutdown_connections 2023-07-14T19:16:56.952 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.951+0000 7fbee102a700 1 --2- 172.21.15.40:0/2012303361 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fbedc100690 0x7fbedc106f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:56.952 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.951+0000 7fbee102a700 1 --2- 172.21.15.40:0/2012303361 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbedc0feb10 0x7fbedc0fef20 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:56.953 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.951+0000 7fbee102a700 1 --2- 172.21.15.40:0/2012303361 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbedc0ffd00 0x7fbedc100150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:56.953 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.951+0000 7fbee102a700 1 -- 172.21.15.40:0/2012303361 >> 172.21.15.40:0/2012303361 conn(0x7fbedc0fa130 msgr2=0x7fbedc0fc560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:16:56.953 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.951+0000 7fbee102a700 1 -- 172.21.15.40:0/2012303361 shutdown_connections 2023-07-14T19:16:56.953 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.951+0000 7fbee102a700 1 -- 172.21.15.40:0/2012303361 wait complete. 2023-07-14T19:16:56.955 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.951+0000 7fbee102a700 1 Processor -- start 2023-07-14T19:16:56.955 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.952+0000 7fbee102a700 1 -- start start 2023-07-14T19:16:56.955 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.952+0000 7fbee102a700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbedc0feb10 0x7fbedc068310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:16:56.955 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.952+0000 7fbee102a700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fbedc0ffd00 0x7fbedc19a6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:16:56.955 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.952+0000 7fbee102a700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbedc100690 0x7fbedc19cd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:16:56.955 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.952+0000 7fbee102a700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbedc19d4a0 con 0x7fbedc100690 2023-07-14T19:16:56.955 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.952+0000 7fbee102a700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbedc19d610 con 0x7fbedc0ffd00 2023-07-14T19:16:56.956 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.952+0000 7fbee102a700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbedc19d7d0 con 0x7fbedc0feb10 2023-07-14T19:16:56.956 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.952+0000 7fbedb59e700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbedc100690 0x7fbedc19cd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:16:56.956 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.952+0000 7fbedb59e700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbedc100690 0x7fbedc19cd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:33742/0 (socket says 172.21.15.40:33742) 2023-07-14T19:16:56.956 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.952+0000 7fbedb59e700 1 -- 172.21.15.40:0/4003078640 learned_addr learned my addr 172.21.15.40:0/4003078640 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-07-14T19:16:56.956 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.952+0000 7fbeda59c700 1 --2- 172.21.15.40:0/4003078640 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fbedc0ffd00 0x7fbedc19a6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:16:56.956 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.952+0000 7fbedad9d700 1 --2- 172.21.15.40:0/4003078640 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbedc0feb10 0x7fbedc068310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:16:56.956 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.952+0000 7fbedb59e700 1 -- 172.21.15.40:0/4003078640 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbedc0feb10 msgr2=0x7fbedc068310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:16:56.957 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.952+0000 7fbedb59e700 1 --2- 172.21.15.40:0/4003078640 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbedc0feb10 0x7fbedc068310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:56.957 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.952+0000 7fbedb59e700 1 -- 172.21.15.40:0/4003078640 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fbedc0ffd00 msgr2=0x7fbedc19a6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:16:56.957 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.952+0000 7fbedb59e700 1 --2- 172.21.15.40:0/4003078640 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fbedc0ffd00 0x7fbedc19a6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:56.957 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.952+0000 7fbedb59e700 1 -- 172.21.15.40:0/4003078640 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbec40095f0 con 0x7fbedc100690 2023-07-14T19:16:56.957 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.952+0000 7fbedad9d700 1 --2- 172.21.15.40:0/4003078640 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbedc0feb10 0x7fbedc068310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-14T19:16:56.957 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.952+0000 7fbeda59c700 1 --2- 172.21.15.40:0/4003078640 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fbedc0ffd00 0x7fbedc19a6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T19:16:56.957 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.953+0000 7fbedb59e700 1 --2- 172.21.15.40:0/4003078640 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbedc100690 0x7fbedc19cd00 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7fbed4007ee0 tx=0x7fbed400d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:16:56.957 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.953+0000 7fbed3fff700 1 -- 172.21.15.40:0/4003078640 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbed400ddd0 con 0x7fbedc100690 2023-07-14T19:16:56.958 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.953+0000 7fbee102a700 1 -- 172.21.15.40:0/4003078640 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbedc19dab0 con 0x7fbedc100690 2023-07-14T19:16:56.958 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.953+0000 7fbed3fff700 1 -- 172.21.15.40:0/4003078640 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fbed400f040 con 0x7fbedc100690 2023-07-14T19:16:56.958 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.953+0000 7fbed3fff700 1 -- 172.21.15.40:0/4003078640 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbed4015400 con 0x7fbedc100690 2023-07-14T19:16:56.958 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.953+0000 7fbee102a700 1 -- 172.21.15.40:0/4003078640 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbedc19df70 con 0x7fbedc100690 2023-07-14T19:16:56.962 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.955+0000 7fbed3fff700 1 -- 172.21.15.40:0/4003078640 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7fbed4004020 con 0x7fbedc100690 2023-07-14T19:16:56.962 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.956+0000 7fbed3fff700 1 --2- 172.21.15.40:0/4003078640 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7fbec80a11e0 0x7fbec80a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:16:56.963 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.956+0000 7fbedad9d700 1 --2- 172.21.15.40:0/4003078640 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7fbec80a11e0 0x7fbec80a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:16:56.963 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.956+0000 7fbed3fff700 1 -- 172.21.15.40:0/4003078640 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 8269+0+0 (secure 0 0 0) 0x7fbed40cc470 con 0x7fbedc100690 2023-07-14T19:16:56.963 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.956+0000 7fbedad9d700 1 --2- 172.21.15.40:0/4003078640 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7fbec80a11e0 0x7fbec80a3690 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fbec4000c00 tx=0x7fbec4005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:16:56.963 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.957+0000 7fbeed1c2b80 1 Processor -- start 2023-07-14T19:16:56.963 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.957+0000 7fbeed1c2b80 1 -- start start 2023-07-14T19:16:56.963 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.957+0000 7fbeed1c2b80 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x556ac92ef320 0x556ac92ef730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:16:56.963 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.957+0000 7fbeed1c2b80 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556ac92efc70 0x556ac92f0290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:16:56.964 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.957+0000 7fbeed1c2b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556ac92f0870 0x556ac92f0ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:16:56.964 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.957+0000 7fbeed1c2b80 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x556ac91efc60 con 0x556ac92efc70 2023-07-14T19:16:56.964 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.957+0000 7fbeed1c2b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x556ac92429f0 con 0x556ac92f0870 2023-07-14T19:16:56.964 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.957+0000 7fbeed1c2b80 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x556ac923be80 con 0x556ac92ef320 2023-07-14T19:16:56.964 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.957+0000 7fbeda59c700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556ac92efc70 0x556ac92f0290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:16:56.964 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.957+0000 7fbeda59c700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556ac92efc70 0x556ac92f0290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:33748/0 (socket says 172.21.15.40:33748) 2023-07-14T19:16:56.964 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.957+0000 7fbeda59c700 1 -- 172.21.15.40:0/2138225993 learned_addr learned my addr 172.21.15.40:0/2138225993 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-07-14T19:16:56.964 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.957+0000 7fbedad9d700 1 --2- 172.21.15.40:0/2138225993 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x556ac92ef320 0x556ac92ef730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:16:56.965 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.957+0000 7fbedb59e700 1 --2- 172.21.15.40:0/2138225993 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556ac92f0870 0x556ac92f0ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:16:56.965 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.957+0000 7fbeda59c700 1 -- 172.21.15.40:0/2138225993 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x556ac92ef320 msgr2=0x556ac92ef730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T19:16:56.965 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.957+0000 7fbeda59c700 1 --2- 172.21.15.40:0/2138225993 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x556ac92ef320 0x556ac92ef730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:56.965 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.957+0000 7fbeda59c700 1 -- 172.21.15.40:0/2138225993 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556ac92f0870 msgr2=0x556ac92f0ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T19:16:56.965 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.957+0000 7fbeda59c700 1 --2- 172.21.15.40:0/2138225993 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556ac92f0870 0x556ac92f0ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:56.965 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.957+0000 7fbeda59c700 1 -- 172.21.15.40:0/2138225993 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x556ac9069d90 con 0x556ac92efc70 2023-07-14T19:16:56.965 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.958+0000 7fbeda59c700 1 --2- 172.21.15.40:0/2138225993 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556ac92efc70 0x556ac92f0290 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7fbecc008c70 tx=0x7fbecc008f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2c24b34b0a5be542 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:16:56.966 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.958+0000 7fbed17fa700 1 -- 172.21.15.40:0/2138225993 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbecc00ed70 con 0x556ac92efc70 2023-07-14T19:16:56.966 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.958+0000 7fbed17fa700 1 -- 172.21.15.40:0/2138225993 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fbecc00eed0 con 0x556ac92efc70 2023-07-14T19:16:56.966 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.958+0000 7fbed17fa700 1 -- 172.21.15.40:0/2138225993 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbecc010430 con 0x556ac92efc70 2023-07-14T19:16:56.966 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.958+0000 7fbeed1c2b80 1 -- 172.21.15.40:0/2138225993 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556ac92efc70 msgr2=0x556ac92f0290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:16:56.966 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.958+0000 7fbeed1c2b80 1 --2- 172.21.15.40:0/2138225993 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556ac92efc70 0x556ac92f0290 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7fbecc008c70 tx=0x7fbecc008f80 comp rx=0 tx=0).stop 2023-07-14T19:16:56.966 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.959+0000 7fbeed1c2b80 1 -- 172.21.15.40:0/2138225993 shutdown_connections 2023-07-14T19:16:56.966 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.959+0000 7fbeed1c2b80 1 --2- 172.21.15.40:0/2138225993 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x556ac92ef320 0x556ac92ef730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:56.966 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.959+0000 7fbeed1c2b80 1 --2- 172.21.15.40:0/2138225993 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556ac92f0870 0x556ac92f0ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:56.967 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.959+0000 7fbeed1c2b80 1 --2- 172.21.15.40:0/2138225993 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556ac92efc70 0x556ac92f0290 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:56.967 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.959+0000 7fbeed1c2b80 1 -- 172.21.15.40:0/2138225993 >> 172.21.15.40:0/2138225993 conn(0x556ac92e0630 msgr2=0x556ac92e5980 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:16:56.967 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.959+0000 7fbeed1c2b80 1 -- 172.21.15.40:0/2138225993 shutdown_connections 2023-07-14T19:16:56.967 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.959+0000 7fbeed1c2b80 1 -- 172.21.15.40:0/2138225993 wait complete. 2023-07-14T19:16:56.967 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.959+0000 7fbeed1c2b80 1 -- start start 2023-07-14T19:16:56.967 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.959+0000 7fbeed1c2b80 1 Processor -- start 2023-07-14T19:16:56.967 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.959+0000 7fbeed1c2b80 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x556ac92ef320 0x556ac92f8d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:16:56.968 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.959+0000 7fbeed1c2b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556ac92efc70 0x556ac92f9240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:16:56.968 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.959+0000 7fbeed1c2b80 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556ac92f0870 0x556ac9300280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:16:56.968 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.959+0000 7fbeed1c2b80 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x556ac91dba70 con 0x556ac92f0870 2023-07-14T19:16:56.968 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.959+0000 7fbeed1c2b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x556ac91db770 con 0x556ac92efc70 2023-07-14T19:16:56.968 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.959+0000 7fbeed1c2b80 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x556ac90bf8d0 con 0x556ac92ef320 2023-07-14T19:16:56.968 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.959+0000 7fbeed1c2b80 10 client.0 _ll_register_callbacks cb 0x556ac91d9710 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-14T19:16:56.968 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.959+0000 7fbedb59e700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556ac92f0870 0x556ac9300280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:16:56.968 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.959+0000 7fbedb59e700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556ac92f0870 0x556ac9300280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:33756/0 (socket says 172.21.15.40:33756) 2023-07-14T19:16:56.969 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.959+0000 7fbedb59e700 1 -- 172.21.15.40:0/3936182768 learned_addr learned my addr 172.21.15.40:0/3936182768 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-07-14T19:16:56.969 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.959+0000 7fbeda59c700 1 --2- 172.21.15.40:0/3936182768 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556ac92efc70 0x556ac92f9240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:16:56.969 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.960+0000 7fbedad9d700 1 --2- 172.21.15.40:0/3936182768 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x556ac92ef320 0x556ac92f8d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:16:56.969 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.960+0000 7fbedb59e700 1 -- 172.21.15.40:0/3936182768 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x556ac92ef320 msgr2=0x556ac92f8d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T19:16:56.969 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.960+0000 7fbedb59e700 1 --2- 172.21.15.40:0/3936182768 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x556ac92ef320 0x556ac92f8d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:56.969 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.960+0000 7fbedb59e700 1 -- 172.21.15.40:0/3936182768 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556ac92efc70 msgr2=0x556ac92f9240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T19:16:56.969 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.960+0000 7fbedb59e700 1 --2- 172.21.15.40:0/3936182768 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556ac92efc70 0x556ac92f9240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:56.969 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.960+0000 7fbedb59e700 1 -- 172.21.15.40:0/3936182768 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbecc0088d0 con 0x556ac92f0870 2023-07-14T19:16:56.970 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.960+0000 7fbedb59e700 1 --2- 172.21.15.40:0/3936182768 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556ac92f0870 0x556ac9300280 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7fbed40c9e30 tx=0x7fbed40990e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=75c258f47db5189 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:16:56.970 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.960+0000 7fbec27fc700 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2023-07-14T19:16:56.970 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.960+0000 7fbec27fc700 1 -- 172.21.15.40:0/3936182768 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbed40cc2b0 con 0x556ac92f0870 2023-07-14T19:16:56.970 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.960+0000 7fbec27fc700 1 -- 172.21.15.40:0/3936182768 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(83..83 src has 1..83) v4 ==== 8269+0+0 (secure 0 0 0) 0x7fbed40a1b50 con 0x556ac92f0870 2023-07-14T19:16:56.970 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.960+0000 7fbeed1c2b80 1 -- 172.21.15.40:0/3936182768 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x556ac91954a0 con 0x556ac92f0870 2023-07-14T19:16:56.970 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.960+0000 7fbec27fc700 1 -- 172.21.15.40:0/3936182768 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7fbed4099960 con 0x556ac92f0870 2023-07-14T19:16:56.970 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.960+0000 7fbeed1c2b80 10 client.15024 fetch_fsmap learned FSMap version 26 2023-07-14T19:16:56.971 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.960+0000 7fbeed1c2b80 1 -- 172.21.15.40:0/3936182768 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) v3 -- 0x556ac9052c10 con 0x556ac92f0870 2023-07-14T19:16:56.971 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.960+0000 7fbec27fc700 1 -- 172.21.15.40:0/3936182768 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fbed40c7070 con 0x556ac92f0870 2023-07-14T19:16:56.971 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.961+0000 7fbec27fc700 1 -- 172.21.15.40:0/3936182768 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 26) v1 ==== 7462+0+0 (secure 0 0 0) 0x7fbed4004bd0 con 0x556ac92f0870 2023-07-14T19:16:56.971 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.961+0000 7fbeed1c2b80 10 client.15024 fetch_fsmap finished waiting for FSMap version 26 2023-07-14T19:16:56.971 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.961+0000 7fbeed1c2b80 10 client.15024 resolve_mds: resolved name 'c' to daemon mds.c 2023-07-14T19:16:56.971 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.961+0000 7fbeed1c2b80 20 client.15024 populate_metadata read hostname 'smithi040' 2023-07-14T19:16:56.971 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.961+0000 7fbeed1c2b80 1 --2- 172.21.15.40:0/3936182768 >> [v2:172.21.15.191:6836/2406892975,v1:172.21.15.191:6837/2406892975] conn(0x556ac9303f70 0x556ac93033d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:16:56.971 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.961+0000 7fbeed1c2b80 4 client.15024 mds_command: new command op to 24490 tid=0[{"prefix": "get_command_descriptions"}] 2023-07-14T19:16:56.972 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.961+0000 7fbeed1c2b80 1 -- 172.21.15.40:0/3936182768 --> [v2:172.21.15.191:6836/2406892975,v1:172.21.15.191:6837/2406892975] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x556ac92b9d50 con 0x556ac9303f70 2023-07-14T19:16:56.972 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.961+0000 7fbedad9d700 1 --2- 172.21.15.40:0/3936182768 >> [v2:172.21.15.191:6836/2406892975,v1:172.21.15.191:6837/2406892975] conn(0x556ac9303f70 0x556ac93033d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:16:56.972 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.961+0000 7fbec27fc700 10 client.15024 ms_handle_connect on v2:172.21.15.191:6836/2406892975 2023-07-14T19:16:56.972 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.961+0000 7fbedad9d700 1 --2- 172.21.15.40:0/3936182768 >> [v2:172.21.15.191:6836/2406892975,v1:172.21.15.191:6837/2406892975] conn(0x556ac9303f70 0x556ac93033d0 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=abb9497474289683 server_cookie=c4dc72f4b68fb298 in_seq=0 out_seq=0 2023-07-14T19:16:56.973 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.962+0000 7fbec27fc700 1 -- 172.21.15.40:0/3936182768 <== mds.3 v2:172.21.15.191:6836/2406892975 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18017 (crc 0 0 0) 0x556ac92b9d50 con 0x556ac9303f70 2023-07-14T19:16:56.973 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.962+0000 7fbec27fc700 10 client.15024 handle_command_reply: tid=0 2023-07-14T19:16:56.973 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.962+0000 7fbeed1c2b80 1 client.15024 shutdown 2023-07-14T19:16:56.973 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.962+0000 7fbeed1c2b80 1 -- 172.21.15.40:0/3936182768 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556ac92f0870 msgr2=0x556ac9300280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:16:56.974 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.962+0000 7fbeed1c2b80 1 --2- 172.21.15.40:0/3936182768 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556ac92f0870 0x556ac9300280 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7fbed40c9e30 tx=0x7fbed40990e0 comp rx=0 tx=0).stop 2023-07-14T19:16:56.974 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.962+0000 7fbeed1c2b80 1 -- 172.21.15.40:0/3936182768 shutdown_connections 2023-07-14T19:16:56.974 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.962+0000 7fbeed1c2b80 1 --2- 172.21.15.40:0/3936182768 >> [v2:172.21.15.191:6836/2406892975,v1:172.21.15.191:6837/2406892975] conn(0x556ac9303f70 0x556ac93033d0 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:56.974 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.962+0000 7fbeed1c2b80 1 --2- 172.21.15.40:0/3936182768 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x556ac92ef320 0x556ac92f8d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:56.974 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.962+0000 7fbeed1c2b80 1 --2- 172.21.15.40:0/3936182768 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556ac92f0870 0x556ac9300280 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:56.974 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.962+0000 7fbeed1c2b80 1 --2- 172.21.15.40:0/3936182768 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556ac92efc70 0x556ac92f9240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:56.974 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.962+0000 7fbec27fc700 0 client.15024 ms_handle_reset on v2:172.21.15.191:6836/2406892975 2023-07-14T19:16:56.975 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.962+0000 7fbeed1c2b80 1 -- 172.21.15.40:0/3936182768 >> 172.21.15.40:0/3936182768 conn(0x556ac92e0630 msgr2=0x556ac92e4600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:16:56.975 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.962+0000 7fbeed1c2b80 1 -- 172.21.15.40:0/3936182768 shutdown_connections 2023-07-14T19:16:56.975 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.962+0000 7fbeed1c2b80 1 -- 172.21.15.40:0/3936182768 wait complete. 2023-07-14T19:16:56.975 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.963+0000 7fbeed1c2b80 20 client.15024 trim_cache size 0 max 16384 2023-07-14T19:16:56.975 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:16:56 smithi040 ceph-mon[96518]: pgmap v285: 97 pgs: 6 peering, 91 active+clean; 638 KiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 1 op/s; 83 B/s, 0 objects/s recovering 2023-07-14T19:16:56.975 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:16:56 smithi040 ceph-mon[96518]: from='client.? 172.21.15.40:0/3609328448' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T19:16:56.976 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:16:56 smithi040 ceph-mon[96518]: from='client.? 172.21.15.40:0/311948507' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T19:16:56.976 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:16:56 smithi040 ceph-mon[96518]: from='client.? 172.21.15.40:0/2968496269' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T19:16:56.981 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.980+0000 7fbeed1c2b80 1 Processor -- start 2023-07-14T19:16:56.981 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.981+0000 7fbeed1c2b80 1 -- start start 2023-07-14T19:16:56.982 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.981+0000 7fbeed1c2b80 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x556ac92ef320 0x556ac92e1b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:16:56.983 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.981+0000 7fbeed1c2b80 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556ac92efc70 0x556ac9300240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:16:56.983 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.981+0000 7fbeed1c2b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556ac92f0870 0x556ac92e50a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:16:56.983 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.981+0000 7fbeed1c2b80 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x556ac9301f70 con 0x556ac92efc70 2023-07-14T19:16:56.983 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.981+0000 7fbeed1c2b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x556ac92f9ef0 con 0x556ac92f0870 2023-07-14T19:16:56.983 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.981+0000 7fbeed1c2b80 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x556ac90cbde0 con 0x556ac92ef320 2023-07-14T19:16:56.983 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.981+0000 7fbeda59c700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556ac92efc70 0x556ac9300240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:16:56.984 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.981+0000 7fbeda59c700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556ac92efc70 0x556ac9300240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:33762/0 (socket says 172.21.15.40:33762) 2023-07-14T19:16:56.984 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.981+0000 7fbeda59c700 1 -- 172.21.15.40:0/642307836 learned_addr learned my addr 172.21.15.40:0/642307836 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-07-14T19:16:56.984 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.981+0000 7fbedad9d700 1 --2- 172.21.15.40:0/642307836 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x556ac92ef320 0x556ac92e1b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:16:56.984 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.981+0000 7fbedb59e700 1 --2- 172.21.15.40:0/642307836 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556ac92f0870 0x556ac92e50a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:16:56.984 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.981+0000 7fbeda59c700 1 -- 172.21.15.40:0/642307836 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x556ac92ef320 msgr2=0x556ac92e1b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T19:16:56.984 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.981+0000 7fbeda59c700 1 --2- 172.21.15.40:0/642307836 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x556ac92ef320 0x556ac92e1b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:56.984 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.981+0000 7fbeda59c700 1 -- 172.21.15.40:0/642307836 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556ac92f0870 msgr2=0x556ac92e50a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T19:16:56.985 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.981+0000 7fbeda59c700 1 --2- 172.21.15.40:0/642307836 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556ac92f0870 0x556ac92e50a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:56.985 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.981+0000 7fbeda59c700 1 -- 172.21.15.40:0/642307836 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x556ac9090200 con 0x556ac92efc70 2023-07-14T19:16:56.985 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.981+0000 7fbedb59e700 1 --2- 172.21.15.40:0/642307836 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556ac92f0870 0x556ac92e50a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-14T19:16:56.985 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.981+0000 7fbeda59c700 1 --2- 172.21.15.40:0/642307836 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556ac92efc70 0x556ac9300240 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7fbecc017000 tx=0x7fbecc008310 comp rx=0 tx=0).ready entity=mon.0 client_cookie=37f7e31986de6e94 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:16:56.985 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.982+0000 7fbed1ffb700 1 -- 172.21.15.40:0/642307836 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbecc01f040 con 0x556ac92efc70 2023-07-14T19:16:56.985 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.982+0000 7fbed1ffb700 1 -- 172.21.15.40:0/642307836 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fbecc024940 con 0x556ac92efc70 2023-07-14T19:16:56.985 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.982+0000 7fbed1ffb700 1 -- 172.21.15.40:0/642307836 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbecc0108c0 con 0x556ac92efc70 2023-07-14T19:16:56.986 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.982+0000 7fbeed1c2b80 1 -- 172.21.15.40:0/642307836 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556ac92efc70 msgr2=0x556ac9300240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:16:56.986 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.982+0000 7fbeed1c2b80 1 --2- 172.21.15.40:0/642307836 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556ac92efc70 0x556ac9300240 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7fbecc017000 tx=0x7fbecc008310 comp rx=0 tx=0).stop 2023-07-14T19:16:56.989 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.982+0000 7fbeed1c2b80 1 -- 172.21.15.40:0/642307836 shutdown_connections 2023-07-14T19:16:56.989 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.982+0000 7fbeed1c2b80 1 --2- 172.21.15.40:0/642307836 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x556ac92ef320 0x556ac92e1b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:56.989 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.982+0000 7fbeed1c2b80 1 --2- 172.21.15.40:0/642307836 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556ac92f0870 0x556ac92e50a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:56.989 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.982+0000 7fbeed1c2b80 1 --2- 172.21.15.40:0/642307836 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556ac92efc70 0x556ac9300240 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:56.989 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.982+0000 7fbeed1c2b80 1 -- 172.21.15.40:0/642307836 >> 172.21.15.40:0/642307836 conn(0x556ac92e0630 msgr2=0x556ac92e5950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:16:56.989 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.982+0000 7fbeed1c2b80 1 -- 172.21.15.40:0/642307836 shutdown_connections 2023-07-14T19:16:56.989 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.982+0000 7fbeed1c2b80 1 -- 172.21.15.40:0/642307836 wait complete. 2023-07-14T19:16:56.990 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.983+0000 7fbeed1c2b80 1 -- start start 2023-07-14T19:16:56.990 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.983+0000 7fbeed1c2b80 1 Processor -- start 2023-07-14T19:16:56.990 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.983+0000 7fbeed1c2b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556ac92ef320 0x556ac92e5850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:16:56.990 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.983+0000 7fbeed1c2b80 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556ac92efc70 0x556ac9300240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:16:56.990 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.983+0000 7fbeed1c2b80 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x556ac92f0870 0x556ac9357bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:16:56.990 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.983+0000 7fbeed1c2b80 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x556ac9358130 con 0x556ac92efc70 2023-07-14T19:16:56.990 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.983+0000 7fbeed1c2b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x556ac93582a0 con 0x556ac92ef320 2023-07-14T19:16:56.991 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.983+0000 7fbeed1c2b80 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x556ac9358460 con 0x556ac92f0870 2023-07-14T19:16:56.991 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.983+0000 7fbeed1c2b80 10 client.0 _ll_register_callbacks cb 0x7fbed40994d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-14T19:16:56.991 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.983+0000 7fbeda59c700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556ac92efc70 0x556ac9300240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:16:56.991 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.983+0000 7fbeda59c700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556ac92efc70 0x556ac9300240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:33770/0 (socket says 172.21.15.40:33770) 2023-07-14T19:16:56.991 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.983+0000 7fbeda59c700 1 -- 172.21.15.40:0/3941797795 learned_addr learned my addr 172.21.15.40:0/3941797795 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-07-14T19:16:56.991 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.983+0000 7fbedad9d700 1 --2- 172.21.15.40:0/3941797795 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556ac92ef320 0x556ac92e5850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:16:56.991 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.983+0000 7fbedb59e700 1 --2- 172.21.15.40:0/3941797795 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x556ac92f0870 0x556ac9357bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:16:56.991 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.983+0000 7fbeda59c700 1 -- 172.21.15.40:0/3941797795 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x556ac92f0870 msgr2=0x556ac9357bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T19:16:56.992 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.983+0000 7fbeda59c700 1 --2- 172.21.15.40:0/3941797795 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x556ac92f0870 0x556ac9357bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:56.992 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.983+0000 7fbeda59c700 1 -- 172.21.15.40:0/3941797795 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556ac92ef320 msgr2=0x556ac92e5850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T19:16:56.992 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.983+0000 7fbeda59c700 1 --2- 172.21.15.40:0/3941797795 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556ac92ef320 0x556ac92e5850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:56.992 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.983+0000 7fbeda59c700 1 -- 172.21.15.40:0/3941797795 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x556ac9069d90 con 0x556ac92efc70 2023-07-14T19:16:56.992 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.983+0000 7fbedad9d700 1 --2- 172.21.15.40:0/3941797795 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556ac92ef320 0x556ac92e5850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-14T19:16:56.992 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.984+0000 7fbeda59c700 1 --2- 172.21.15.40:0/3941797795 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556ac92efc70 0x556ac9300240 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7fbecc010510 tx=0x7fbecc0245b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=32a2753a44f389a6 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:16:56.992 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.984+0000 7fbec27fc700 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2023-07-14T19:16:56.993 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.984+0000 7fbec27fc700 1 -- 172.21.15.40:0/3941797795 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbecc01f040 con 0x556ac92efc70 2023-07-14T19:16:56.993 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.984+0000 7fbec27fc700 1 -- 172.21.15.40:0/3941797795 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(83..83 src has 1..83) v4 ==== 8269+0+0 (secure 0 0 0) 0x7fbecc010840 con 0x556ac92efc70 2023-07-14T19:16:56.993 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.984+0000 7fbeed1c2b80 1 -- 172.21.15.40:0/3941797795 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fbecc00eed0 con 0x556ac92efc70 2023-07-14T19:16:56.993 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.984+0000 7fbec27fc700 1 -- 172.21.15.40:0/3941797795 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7fbecc010a20 con 0x556ac92efc70 2023-07-14T19:16:56.993 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.984+0000 7fbeed1c2b80 10 client.15030 fetch_fsmap learned FSMap version 26 2023-07-14T19:16:56.993 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.984+0000 7fbeed1c2b80 1 -- 172.21.15.40:0/3941797795 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) v3 -- 0x556ac9068260 con 0x556ac92efc70 2023-07-14T19:16:56.993 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.984+0000 7fbec27fc700 1 -- 172.21.15.40:0/3941797795 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fbecc03e080 con 0x556ac92efc70 2023-07-14T19:16:56.993 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.984+0000 7fbec27fc700 1 -- 172.21.15.40:0/3941797795 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 26) v1 ==== 7462+0+0 (secure 0 0 0) 0x7fbecc038030 con 0x556ac92efc70 2023-07-14T19:16:56.994 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.984+0000 7fbeed1c2b80 10 client.15030 fetch_fsmap finished waiting for FSMap version 26 2023-07-14T19:16:56.994 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.984+0000 7fbeed1c2b80 10 client.15030 resolve_mds: resolved name 'c' to daemon mds.c 2023-07-14T19:16:56.994 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.984+0000 7fbeed1c2b80 20 client.15030 populate_metadata read hostname 'smithi040' 2023-07-14T19:16:56.994 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.984+0000 7fbeed1c2b80 1 --2- 172.21.15.40:0/3941797795 >> [v2:172.21.15.191:6836/2406892975,v1:172.21.15.191:6837/2406892975] conn(0x556ac935cd20 0x556ac9359650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:16:56.994 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.984+0000 7fbeed1c2b80 4 client.15030 mds_command: new command op to 24490 tid=0[{"prefix": "status"}] 2023-07-14T19:16:56.994 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.984+0000 7fbeed1c2b80 1 -- 172.21.15.40:0/3941797795 --> [v2:172.21.15.191:6836/2406892975,v1:172.21.15.191:6837/2406892975] -- command(tid 0: {"prefix": "status"}) v1 -- 0x556ac9234e60 con 0x556ac935cd20 2023-07-14T19:16:56.994 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.985+0000 7fbedad9d700 1 --2- 172.21.15.40:0/3941797795 >> [v2:172.21.15.191:6836/2406892975,v1:172.21.15.191:6837/2406892975] conn(0x556ac935cd20 0x556ac9359650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:16:56.995 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.985+0000 7fbec27fc700 10 client.15030 ms_handle_connect on v2:172.21.15.191:6836/2406892975 2023-07-14T19:16:56.995 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.985+0000 7fbedad9d700 1 --2- 172.21.15.40:0/3941797795 >> [v2:172.21.15.191:6836/2406892975,v1:172.21.15.191:6837/2406892975] conn(0x556ac935cd20 0x556ac9359650 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=c4511f04c2559ea9 server_cookie=51376ab955a32007 in_seq=0 out_seq=0 2023-07-14T19:16:56.995 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.985+0000 7fbec27fc700 1 -- 172.21.15.40:0/3941797795 <== mds.3 v2:172.21.15.191:6836/2406892975 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+332 (crc 0 0 0) 0x556ac9234e60 con 0x556ac935cd20 2023-07-14T19:16:56.995 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.985+0000 7fbec27fc700 10 client.15030 handle_command_reply: tid=0 2023-07-14T19:16:56.995 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.985+0000 7fbeed1c2b80 1 client.15030 shutdown 2023-07-14T19:16:56.995 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.985+0000 7fbeed1c2b80 1 -- 172.21.15.40:0/3941797795 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556ac92efc70 msgr2=0x556ac9300240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:16:56.995 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.985+0000 7fbeed1c2b80 1 --2- 172.21.15.40:0/3941797795 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556ac92efc70 0x556ac9300240 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7fbecc010510 tx=0x7fbecc0245b0 comp rx=0 tx=0).stop 2023-07-14T19:16:56.995 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.985+0000 7fbeed1c2b80 1 -- 172.21.15.40:0/3941797795 shutdown_connections 2023-07-14T19:16:56.996 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.985+0000 7fbeed1c2b80 1 --2- 172.21.15.40:0/3941797795 >> [v2:172.21.15.191:6836/2406892975,v1:172.21.15.191:6837/2406892975] conn(0x556ac935cd20 0x556ac9359650 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:56.996 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.985+0000 7fbeed1c2b80 1 --2- 172.21.15.40:0/3941797795 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x556ac92f0870 0x556ac9357bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:56.996 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.985+0000 7fbeed1c2b80 1 --2- 172.21.15.40:0/3941797795 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556ac92ef320 0x556ac92e5850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:56.996 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.985+0000 7fbec27fc700 0 client.15030 ms_handle_reset on v2:172.21.15.191:6836/2406892975 2023-07-14T19:16:56.996 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.985+0000 7fbeed1c2b80 1 --2- 172.21.15.40:0/3941797795 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556ac92efc70 0x556ac9300240 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:56.996 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.985+0000 7fbeed1c2b80 1 -- 172.21.15.40:0/3941797795 >> 172.21.15.40:0/3941797795 conn(0x556ac92e0630 msgr2=0x556ac92f6c00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:16:56.996 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.985+0000 7fbeed1c2b80 1 -- 172.21.15.40:0/3941797795 shutdown_connections 2023-07-14T19:16:56.997 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.986+0000 7fbeed1c2b80 1 -- 172.21.15.40:0/3941797795 wait complete. 2023-07-14T19:16:56.997 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.986+0000 7fbeed1c2b80 20 client.15030 trim_cache size 0 max 16384 2023-07-14T19:16:56.997 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2023-07-14T19:16:56.997 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_fsid": "adc52f04-2279-11ee-9b33-001a4aab830c", 2023-07-14T19:16:56.997 INFO:teuthology.orchestra.run.smithi040.stdout: "whoami": 3, 2023-07-14T19:16:56.997 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 24490, 2023-07-14T19:16:56.998 INFO:teuthology.orchestra.run.smithi040.stdout: "want_state": "up:active", 2023-07-14T19:16:56.998 INFO:teuthology.orchestra.run.smithi040.stdout: "state": "up:active", 2023-07-14T19:16:56.998 INFO:teuthology.orchestra.run.smithi040.stdout: "fs_name": "cephfs", 2023-07-14T19:16:56.998 INFO:teuthology.orchestra.run.smithi040.stdout: "rank_uptime": 53.382341412000002, 2023-07-14T19:16:56.998 INFO:teuthology.orchestra.run.smithi040.stdout: "mdsmap_epoch": 26, 2023-07-14T19:16:56.998 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap_epoch": 83, 2023-07-14T19:16:56.998 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap_epoch_barrier": 81, 2023-07-14T19:16:56.999 INFO:teuthology.orchestra.run.smithi040.stdout: "uptime": 75.712242614000004 2023-07-14T19:16:56.999 INFO:teuthology.orchestra.run.smithi040.stdout:} 2023-07-14T19:16:57.000 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.986+0000 7fbee102a700 1 -- 172.21.15.40:0/4003078640 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7fbec80a11e0 msgr2=0x7fbec80a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:16:57.000 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.986+0000 7fbee102a700 1 --2- 172.21.15.40:0/4003078640 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7fbec80a11e0 0x7fbec80a3690 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fbec4000c00 tx=0x7fbec4005fb0 comp rx=0 tx=0).stop 2023-07-14T19:16:57.000 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.986+0000 7fbee102a700 1 -- 172.21.15.40:0/4003078640 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbedc100690 msgr2=0x7fbedc19cd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:16:57.000 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.986+0000 7fbee102a700 1 --2- 172.21.15.40:0/4003078640 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbedc100690 0x7fbedc19cd00 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7fbed4007ee0 tx=0x7fbed400d3b0 comp rx=0 tx=0).stop 2023-07-14T19:16:57.000 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.986+0000 7fbee102a700 1 -- 172.21.15.40:0/4003078640 shutdown_connections 2023-07-14T19:16:57.001 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.986+0000 7fbee102a700 1 --2- 172.21.15.40:0/4003078640 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7fbec80a11e0 0x7fbec80a3690 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:57.001 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.986+0000 7fbee102a700 1 --2- 172.21.15.40:0/4003078640 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbedc0feb10 0x7fbedc068310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:57.001 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.986+0000 7fbee102a700 1 --2- 172.21.15.40:0/4003078640 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbedc100690 0x7fbedc19cd00 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:57.001 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.986+0000 7fbee102a700 1 --2- 172.21.15.40:0/4003078640 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fbedc0ffd00 0x7fbedc19a6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:57.001 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.986+0000 7fbee102a700 1 -- 172.21.15.40:0/4003078640 >> 172.21.15.40:0/4003078640 conn(0x7fbedc0fa130 msgr2=0x7fbedc0fc410 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:16:57.001 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.986+0000 7fbee102a700 1 -- 172.21.15.40:0/4003078640 shutdown_connections 2023-07-14T19:16:57.001 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:56.986+0000 7fbee102a700 1 -- 172.21.15.40:0/4003078640 wait complete. 2023-07-14T19:16:57.002 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.h status 2023-07-14T19:16:57.128 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.128+0000 7fdaf6240700 1 -- 172.21.15.40:0/175808048 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdaf00fdec0 msgr2=0x7fdaf00fe2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:16:57.129 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.128+0000 7fdaf6240700 1 --2- 172.21.15.40:0/175808048 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdaf00fdec0 0x7fdaf00fe2d0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7fdae0009990 tx=0x7fdae0009ca0 comp rx=0 tx=0).stop 2023-07-14T19:16:57.129 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.128+0000 7fdaf6240700 1 -- 172.21.15.40:0/175808048 shutdown_connections 2023-07-14T19:16:57.129 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.128+0000 7fdaf6240700 1 --2- 172.21.15.40:0/175808048 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fdaf0101820 0x7fdaf01080e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:57.129 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.128+0000 7fdaf6240700 1 --2- 172.21.15.40:0/175808048 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdaf00fdec0 0x7fdaf00fe2d0 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:57.129 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.128+0000 7fdaf6240700 1 --2- 172.21.15.40:0/175808048 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fdaf0100e90 0x7fdaf01012e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:57.130 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.128+0000 7fdaf6240700 1 -- 172.21.15.40:0/175808048 >> 172.21.15.40:0/175808048 conn(0x7fdaf00f98a0 msgr2=0x7fdaf00fbcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:16:57.130 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.128+0000 7fdaf6240700 1 -- 172.21.15.40:0/175808048 shutdown_connections 2023-07-14T19:16:57.130 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.128+0000 7fdaf6240700 1 -- 172.21.15.40:0/175808048 wait complete. 2023-07-14T19:16:57.131 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.128+0000 7fdaf6240700 1 Processor -- start 2023-07-14T19:16:57.132 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.128+0000 7fdaf6240700 1 -- start start 2023-07-14T19:16:57.132 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.129+0000 7fdaf6240700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fdaf00fdec0 0x7fdaf0100800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:16:57.132 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.129+0000 7fdaf6240700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fdaf0100e90 0x7fdaf00fee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:16:57.132 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.129+0000 7fdaf6240700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdaf0101820 0x7fdaf00ff380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:16:57.132 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.129+0000 7fdaf6240700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdaf00ffb20 con 0x7fdaf0101820 2023-07-14T19:16:57.132 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.129+0000 7fdaf6240700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdaf00ffc90 con 0x7fdaf00fdec0 2023-07-14T19:16:57.132 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.129+0000 7fdaf6240700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdaf019fa20 con 0x7fdaf0100e90 2023-07-14T19:16:57.133 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.129+0000 7fdaeffff700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdaf0101820 0x7fdaf00ff380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:16:57.133 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.129+0000 7fdaeffff700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdaf0101820 0x7fdaf00ff380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:33784/0 (socket says 172.21.15.40:33784) 2023-07-14T19:16:57.133 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.129+0000 7fdaeffff700 1 -- 172.21.15.40:0/2667838743 learned_addr learned my addr 172.21.15.40:0/2667838743 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-07-14T19:16:57.133 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.129+0000 7fdaef7fe700 1 --2- 172.21.15.40:0/2667838743 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fdaf00fdec0 0x7fdaf0100800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:16:57.133 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.129+0000 7fdaeeffd700 1 --2- 172.21.15.40:0/2667838743 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fdaf0100e90 0x7fdaf00fee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:16:57.133 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.129+0000 7fdaeffff700 1 -- 172.21.15.40:0/2667838743 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fdaf0100e90 msgr2=0x7fdaf00fee40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:16:57.133 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.129+0000 7fdaeffff700 1 --2- 172.21.15.40:0/2667838743 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fdaf0100e90 0x7fdaf00fee40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:57.133 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.129+0000 7fdaeffff700 1 -- 172.21.15.40:0/2667838743 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fdaf00fdec0 msgr2=0x7fdaf0100800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:16:57.134 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.129+0000 7fdaeffff700 1 --2- 172.21.15.40:0/2667838743 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fdaf00fdec0 0x7fdaf0100800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:57.134 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.129+0000 7fdaeffff700 1 -- 172.21.15.40:0/2667838743 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdae00095f0 con 0x7fdaf0101820 2023-07-14T19:16:57.134 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.129+0000 7fdaeffff700 1 --2- 172.21.15.40:0/2667838743 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdaf0101820 0x7fdaf00ff380 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7fdae8007ee0 tx=0x7fdae800d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:16:57.134 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.129+0000 7fdaecff9700 1 -- 172.21.15.40:0/2667838743 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdae800dd20 con 0x7fdaf0101820 2023-07-14T19:16:57.134 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.129+0000 7fdaecff9700 1 -- 172.21.15.40:0/2667838743 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fdae800f040 con 0x7fdaf0101820 2023-07-14T19:16:57.134 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.129+0000 7fdaf6240700 1 -- 172.21.15.40:0/2667838743 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdaf019fd00 con 0x7fdaf0101820 2023-07-14T19:16:57.134 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.129+0000 7fdaecff9700 1 -- 172.21.15.40:0/2667838743 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdae80048a0 con 0x7fdaf0101820 2023-07-14T19:16:57.135 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.130+0000 7fdaf6240700 1 -- 172.21.15.40:0/2667838743 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdaf01a01c0 con 0x7fdaf0101820 2023-07-14T19:16:57.139 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.132+0000 7fdaecff9700 1 -- 172.21.15.40:0/2667838743 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7fdae8003a00 con 0x7fdaf0101820 2023-07-14T19:16:57.139 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.132+0000 7fdaecff9700 1 --2- 172.21.15.40:0/2667838743 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7fdad80a11e0 0x7fdad80a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:16:57.139 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.132+0000 7fdaecff9700 1 -- 172.21.15.40:0/2667838743 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 8269+0+0 (secure 0 0 0) 0x7fdae80cc2d0 con 0x7fdaf0101820 2023-07-14T19:16:57.140 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.132+0000 7fdaef7fe700 1 --2- 172.21.15.40:0/2667838743 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7fdad80a11e0 0x7fdad80a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:16:57.140 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.133+0000 7fdaef7fe700 1 --2- 172.21.15.40:0/2667838743 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7fdad80a11e0 0x7fdad80a3690 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fdae000b5c0 tx=0x7fdae0005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:16:57.140 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.133+0000 7fdb023d8b80 1 Processor -- start 2023-07-14T19:16:57.140 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.133+0000 7fdb023d8b80 1 -- start start 2023-07-14T19:16:57.140 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.133+0000 7fdb023d8b80 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56242eae4f10 0x56242eae5320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:16:57.140 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.133+0000 7fdb023d8b80 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x56242eae5860 0x56242eae5e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:16:57.140 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.133+0000 7fdb023d8b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56242eae6460 0x56242eae68d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:16:57.141 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.133+0000 7fdb023d8b80 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x56242e9e9c60 con 0x56242eae4f10 2023-07-14T19:16:57.141 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.133+0000 7fdb023d8b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x56242ea3c9f0 con 0x56242eae6460 2023-07-14T19:16:57.141 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.133+0000 7fdb023d8b80 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x56242ea35e80 con 0x56242eae5860 2023-07-14T19:16:57.141 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.133+0000 7fdaef7fe700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56242eae4f10 0x56242eae5320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:16:57.141 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.133+0000 7fdaef7fe700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56242eae4f10 0x56242eae5320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:33788/0 (socket says 172.21.15.40:33788) 2023-07-14T19:16:57.141 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.133+0000 7fdaef7fe700 1 -- 172.21.15.40:0/3875299102 learned_addr learned my addr 172.21.15.40:0/3875299102 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-07-14T19:16:57.141 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.134+0000 7fdaeeffd700 1 --2- 172.21.15.40:0/3875299102 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x56242eae5860 0x56242eae5e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:16:57.142 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.134+0000 7fdaeffff700 1 --2- 172.21.15.40:0/3875299102 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56242eae6460 0x56242eae68d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:16:57.142 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.134+0000 7fdaef7fe700 1 -- 172.21.15.40:0/3875299102 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x56242eae5860 msgr2=0x56242eae5e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T19:16:57.142 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.134+0000 7fdaef7fe700 1 --2- 172.21.15.40:0/3875299102 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x56242eae5860 0x56242eae5e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:57.142 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.134+0000 7fdaef7fe700 1 -- 172.21.15.40:0/3875299102 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56242eae6460 msgr2=0x56242eae68d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T19:16:57.142 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.134+0000 7fdaef7fe700 1 --2- 172.21.15.40:0/3875299102 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56242eae6460 0x56242eae68d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:57.142 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.134+0000 7fdaef7fe700 1 -- 172.21.15.40:0/3875299102 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56242e863d90 con 0x56242eae4f10 2023-07-14T19:16:57.142 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.134+0000 7fdaeeffd700 1 --2- 172.21.15.40:0/3875299102 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x56242eae5860 0x56242eae5e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-14T19:16:57.142 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.134+0000 7fdaeffff700 1 --2- 172.21.15.40:0/3875299102 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56242eae6460 0x56242eae68d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-14T19:16:57.143 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.134+0000 7fdaef7fe700 1 --2- 172.21.15.40:0/3875299102 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56242eae4f10 0x56242eae5320 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7fdae000b9d0 tx=0x7fdae0020000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b2e7d4e258f635ad server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:16:57.143 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.134+0000 7fdad5ffb700 1 -- 172.21.15.40:0/3875299102 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdae002a070 con 0x56242eae4f10 2023-07-14T19:16:57.143 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.134+0000 7fdad5ffb700 1 -- 172.21.15.40:0/3875299102 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fdae001e040 con 0x56242eae4f10 2023-07-14T19:16:57.143 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.134+0000 7fdad5ffb700 1 -- 172.21.15.40:0/3875299102 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdae0023a40 con 0x56242eae4f10 2023-07-14T19:16:57.143 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.135+0000 7fdb023d8b80 1 -- 172.21.15.40:0/3875299102 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56242eae4f10 msgr2=0x56242eae5320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:16:57.143 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.135+0000 7fdb023d8b80 1 --2- 172.21.15.40:0/3875299102 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56242eae4f10 0x56242eae5320 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7fdae000b9d0 tx=0x7fdae0020000 comp rx=0 tx=0).stop 2023-07-14T19:16:57.143 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.135+0000 7fdb023d8b80 1 -- 172.21.15.40:0/3875299102 shutdown_connections 2023-07-14T19:16:57.143 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.135+0000 7fdb023d8b80 1 --2- 172.21.15.40:0/3875299102 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56242eae6460 0x56242eae68d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:57.144 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.135+0000 7fdb023d8b80 1 --2- 172.21.15.40:0/3875299102 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56242eae4f10 0x56242eae5320 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:57.144 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.135+0000 7fdb023d8b80 1 --2- 172.21.15.40:0/3875299102 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x56242eae5860 0x56242eae5e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:57.144 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.135+0000 7fdb023d8b80 1 -- 172.21.15.40:0/3875299102 >> 172.21.15.40:0/3875299102 conn(0x56242eada630 msgr2=0x56242eadf980 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:16:57.144 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.135+0000 7fdb023d8b80 1 -- 172.21.15.40:0/3875299102 shutdown_connections 2023-07-14T19:16:57.144 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.135+0000 7fdb023d8b80 1 -- 172.21.15.40:0/3875299102 wait complete. 2023-07-14T19:16:57.144 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.135+0000 7fdb023d8b80 1 -- start start 2023-07-14T19:16:57.144 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.135+0000 7fdb023d8b80 1 Processor -- start 2023-07-14T19:16:57.145 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.135+0000 7fdb023d8b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56242eae4f10 0x56242eaee7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:16:57.145 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.135+0000 7fdb023d8b80 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x56242eae5860 0x56242eaeed20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:16:57.145 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.135+0000 7fdb023d8b80 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56242eae6460 0x56242eaf5d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:16:57.145 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.136+0000 7fdb023d8b80 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x56242e9d5a70 con 0x56242eae6460 2023-07-14T19:16:57.145 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.136+0000 7fdb023d8b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x56242e9d5770 con 0x56242eae4f10 2023-07-14T19:16:57.145 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.136+0000 7fdb023d8b80 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x56242e8b98d0 con 0x56242eae5860 2023-07-14T19:16:57.145 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.136+0000 7fdb023d8b80 10 client.0 _ll_register_callbacks cb 0x56242e9d3710 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-14T19:16:57.145 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.136+0000 7fdaeffff700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56242eae6460 0x56242eaf5d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:16:57.146 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.136+0000 7fdaeffff700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56242eae6460 0x56242eaf5d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:33798/0 (socket says 172.21.15.40:33798) 2023-07-14T19:16:57.146 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.136+0000 7fdaeffff700 1 -- 172.21.15.40:0/3320893732 learned_addr learned my addr 172.21.15.40:0/3320893732 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-07-14T19:16:57.146 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.136+0000 7fdaef7fe700 1 --2- 172.21.15.40:0/3320893732 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56242eae4f10 0x56242eaee7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:16:57.146 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.136+0000 7fdaeeffd700 1 --2- 172.21.15.40:0/3320893732 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x56242eae5860 0x56242eaeed20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:16:57.146 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.136+0000 7fdaeffff700 1 -- 172.21.15.40:0/3320893732 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x56242eae5860 msgr2=0x56242eaeed20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T19:16:57.146 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.136+0000 7fdaeffff700 1 --2- 172.21.15.40:0/3320893732 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x56242eae5860 0x56242eaeed20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:57.146 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.136+0000 7fdaeffff700 1 -- 172.21.15.40:0/3320893732 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56242eae4f10 msgr2=0x56242eaee7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T19:16:57.147 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.136+0000 7fdaeffff700 1 --2- 172.21.15.40:0/3320893732 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56242eae4f10 0x56242eaee7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:57.147 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.136+0000 7fdaeffff700 1 -- 172.21.15.40:0/3320893732 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdaf0108950 con 0x56242eae6460 2023-07-14T19:16:57.147 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.136+0000 7fdaeeffd700 1 --2- 172.21.15.40:0/3320893732 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x56242eae5860 0x56242eaeed20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T19:16:57.147 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.136+0000 7fdaeffff700 1 --2- 172.21.15.40:0/3320893732 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56242eae6460 0x56242eaf5d60 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7fdae80994f0 tx=0x7fdae8098f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=92ab627398edf1e0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:16:57.147 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.136+0000 7fdaceffd700 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2023-07-14T19:16:57.147 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.136+0000 7fdaceffd700 1 -- 172.21.15.40:0/3320893732 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdae809d590 con 0x56242eae6460 2023-07-14T19:16:57.147 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.136+0000 7fdaceffd700 1 -- 172.21.15.40:0/3320893732 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(83..83 src has 1..83) v4 ==== 8269+0+0 (secure 0 0 0) 0x7fdae80a19b0 con 0x56242eae6460 2023-07-14T19:16:57.147 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.136+0000 7fdb023d8b80 1 -- 172.21.15.40:0/3320893732 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x56242e98f4a0 con 0x56242eae6460 2023-07-14T19:16:57.148 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.136+0000 7fdaceffd700 1 -- 172.21.15.40:0/3320893732 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7fdae8009410 con 0x56242eae6460 2023-07-14T19:16:57.148 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.136+0000 7fdb023d8b80 10 client.15042 fetch_fsmap learned FSMap version 26 2023-07-14T19:16:57.148 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.136+0000 7fdb023d8b80 1 -- 172.21.15.40:0/3320893732 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) v3 -- 0x56242e84cc10 con 0x56242eae6460 2023-07-14T19:16:57.148 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.136+0000 7fdaceffd700 1 -- 172.21.15.40:0/3320893732 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fdae80c7070 con 0x56242eae6460 2023-07-14T19:16:57.148 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.137+0000 7fdaceffd700 1 -- 172.21.15.40:0/3320893732 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 26) v1 ==== 7462+0+0 (secure 0 0 0) 0x7fdae8098a50 con 0x56242eae6460 2023-07-14T19:16:57.148 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.137+0000 7fdb023d8b80 10 client.15042 fetch_fsmap finished waiting for FSMap version 26 2023-07-14T19:16:57.148 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.137+0000 7fdb023d8b80 10 client.15042 resolve_mds: resolved name 'h' to daemon mds.h 2023-07-14T19:16:57.149 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.137+0000 7fdb023d8b80 20 client.15042 populate_metadata read hostname 'smithi040' 2023-07-14T19:16:57.149 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.137+0000 7fdb023d8b80 1 --2- 172.21.15.40:0/3320893732 >> [v2:172.21.15.130:6836/2604019369,v1:172.21.15.130:6837/2604019369] conn(0x56242eaf99f0 0x56242eaf8e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:16:57.149 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.137+0000 7fdb023d8b80 4 client.15042 mds_command: new command op to 24506 tid=0[{"prefix": "get_command_descriptions"}] 2023-07-14T19:16:57.149 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.137+0000 7fdb023d8b80 1 -- 172.21.15.40:0/3320893732 --> [v2:172.21.15.130:6836/2604019369,v1:172.21.15.130:6837/2604019369] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x56242eab3d50 con 0x56242eaf99f0 2023-07-14T19:16:57.149 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.137+0000 7fdaef7fe700 1 --2- 172.21.15.40:0/3320893732 >> [v2:172.21.15.130:6836/2604019369,v1:172.21.15.130:6837/2604019369] conn(0x56242eaf99f0 0x56242eaf8e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:16:57.149 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.137+0000 7fdaceffd700 10 client.15042 ms_handle_connect on v2:172.21.15.130:6836/2604019369 2023-07-14T19:16:57.149 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.137+0000 7fdaef7fe700 1 --2- 172.21.15.40:0/3320893732 >> [v2:172.21.15.130:6836/2604019369,v1:172.21.15.130:6837/2604019369] conn(0x56242eaf99f0 0x56242eaf8e80 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=cafde8489b459f2a server_cookie=666dbe70c42e4f54 in_seq=0 out_seq=0 2023-07-14T19:16:57.149 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.138+0000 7fdaceffd700 1 -- 172.21.15.40:0/3320893732 <== mds.2 v2:172.21.15.130:6836/2604019369 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18017 (crc 0 0 0) 0x56242eab3d50 con 0x56242eaf99f0 2023-07-14T19:16:57.150 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.138+0000 7fdaceffd700 10 client.15042 handle_command_reply: tid=0 2023-07-14T19:16:57.150 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.138+0000 7fdb023d8b80 1 client.15042 shutdown 2023-07-14T19:16:57.150 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.138+0000 7fdb023d8b80 1 -- 172.21.15.40:0/3320893732 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56242eae6460 msgr2=0x56242eaf5d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:16:57.150 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.138+0000 7fdb023d8b80 1 --2- 172.21.15.40:0/3320893732 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56242eae6460 0x56242eaf5d60 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7fdae80994f0 tx=0x7fdae8098f40 comp rx=0 tx=0).stop 2023-07-14T19:16:57.150 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.138+0000 7fdb023d8b80 1 -- 172.21.15.40:0/3320893732 shutdown_connections 2023-07-14T19:16:57.150 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.138+0000 7fdb023d8b80 1 --2- 172.21.15.40:0/3320893732 >> [v2:172.21.15.130:6836/2604019369,v1:172.21.15.130:6837/2604019369] conn(0x56242eaf99f0 0x56242eaf8e80 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:57.150 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.138+0000 7fdb023d8b80 1 --2- 172.21.15.40:0/3320893732 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56242eae6460 0x56242eaf5d60 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:57.151 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.138+0000 7fdaceffd700 0 client.15042 ms_handle_reset on v2:172.21.15.130:6836/2604019369 2023-07-14T19:16:57.151 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.138+0000 7fdb023d8b80 1 --2- 172.21.15.40:0/3320893732 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56242eae4f10 0x56242eaee7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:57.151 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.138+0000 7fdb023d8b80 1 --2- 172.21.15.40:0/3320893732 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x56242eae5860 0x56242eaeed20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:57.151 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.138+0000 7fdb023d8b80 1 -- 172.21.15.40:0/3320893732 >> 172.21.15.40:0/3320893732 conn(0x56242eada630 msgr2=0x56242eade600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:16:57.151 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.138+0000 7fdb023d8b80 1 -- 172.21.15.40:0/3320893732 shutdown_connections 2023-07-14T19:16:57.151 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.138+0000 7fdb023d8b80 1 -- 172.21.15.40:0/3320893732 wait complete. 2023-07-14T19:16:57.151 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.139+0000 7fdb023d8b80 20 client.15042 trim_cache size 0 max 16384 2023-07-14T19:16:57.157 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.156+0000 7fdb023d8b80 1 Processor -- start 2023-07-14T19:16:57.158 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.156+0000 7fdb023d8b80 1 -- start start 2023-07-14T19:16:57.158 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.156+0000 7fdb023d8b80 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x56242eae4f10 0x56242eaf8e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:16:57.158 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.156+0000 7fdb023d8b80 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56242eae5860 0x56242eaf5d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:16:57.158 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.156+0000 7fdb023d8b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56242eae6460 0x56242eaec7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:16:57.158 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.156+0000 7fdb023d8b80 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x56242e9d1d50 con 0x56242eae5860 2023-07-14T19:16:57.158 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.156+0000 7fdb023d8b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x56242e9df470 con 0x56242eae6460 2023-07-14T19:16:57.158 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.156+0000 7fdb023d8b80 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x56242eaee350 con 0x56242eae4f10 2023-07-14T19:16:57.159 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.156+0000 7fdaeeffd700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56242eae5860 0x56242eaf5d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:16:57.159 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.156+0000 7fdaeeffd700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56242eae5860 0x56242eaf5d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:33802/0 (socket says 172.21.15.40:33802) 2023-07-14T19:16:57.159 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.156+0000 7fdaeeffd700 1 -- 172.21.15.40:0/2026841930 learned_addr learned my addr 172.21.15.40:0/2026841930 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-07-14T19:16:57.159 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.156+0000 7fdaeeffd700 1 -- 172.21.15.40:0/2026841930 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x56242eae4f10 msgr2=0x56242eaf8e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T19:16:57.159 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.156+0000 7fdaeeffd700 1 --2- 172.21.15.40:0/2026841930 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x56242eae4f10 0x56242eaf8e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:57.159 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.156+0000 7fdaeeffd700 1 -- 172.21.15.40:0/2026841930 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56242eae6460 msgr2=0x56242eaec7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T19:16:57.159 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.156+0000 7fdaeeffd700 1 --2- 172.21.15.40:0/2026841930 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56242eae6460 0x56242eaec7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:57.160 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.156+0000 7fdaeeffd700 1 -- 172.21.15.40:0/2026841930 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56242e88a200 con 0x56242eae5860 2023-07-14T19:16:57.160 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.157+0000 7fdaeeffd700 1 --2- 172.21.15.40:0/2026841930 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56242eae5860 0x56242eaf5d20 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7fdae400ace0 tx=0x7fdae4013460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7f81d810cd1af2f3 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:16:57.160 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.157+0000 7fdad67fc700 1 -- 172.21.15.40:0/2026841930 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdae4004020 con 0x56242eae5860 2023-07-14T19:16:57.160 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.157+0000 7fdad67fc700 1 -- 172.21.15.40:0/2026841930 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fdae4003680 con 0x56242eae5860 2023-07-14T19:16:57.160 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.157+0000 7fdad67fc700 1 -- 172.21.15.40:0/2026841930 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdae400f7c0 con 0x56242eae5860 2023-07-14T19:16:57.163 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.157+0000 7fdb023d8b80 1 -- 172.21.15.40:0/2026841930 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56242eae5860 msgr2=0x56242eaf5d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:16:57.164 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.157+0000 7fdb023d8b80 1 --2- 172.21.15.40:0/2026841930 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56242eae5860 0x56242eaf5d20 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7fdae400ace0 tx=0x7fdae4013460 comp rx=0 tx=0).stop 2023-07-14T19:16:57.164 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.157+0000 7fdb023d8b80 1 -- 172.21.15.40:0/2026841930 shutdown_connections 2023-07-14T19:16:57.164 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.157+0000 7fdb023d8b80 1 --2- 172.21.15.40:0/2026841930 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x56242eae4f10 0x56242eaf8e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:57.164 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.157+0000 7fdb023d8b80 1 --2- 172.21.15.40:0/2026841930 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56242eae6460 0x56242eaec7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:57.164 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.157+0000 7fdb023d8b80 1 --2- 172.21.15.40:0/2026841930 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56242eae5860 0x56242eaf5d20 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:57.164 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.157+0000 7fdb023d8b80 1 -- 172.21.15.40:0/2026841930 >> 172.21.15.40:0/2026841930 conn(0x56242eada630 msgr2=0x56242eaf6590 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:16:57.165 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.157+0000 7fdb023d8b80 1 -- 172.21.15.40:0/2026841930 shutdown_connections 2023-07-14T19:16:57.165 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.157+0000 7fdb023d8b80 1 -- 172.21.15.40:0/2026841930 wait complete. 2023-07-14T19:16:57.165 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.158+0000 7fdb023d8b80 1 -- start start 2023-07-14T19:16:57.165 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.158+0000 7fdb023d8b80 1 Processor -- start 2023-07-14T19:16:57.165 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.158+0000 7fdb023d8b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56242eae4f10 0x56242eaf1f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:16:57.165 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.158+0000 7fdb023d8b80 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56242eae5860 0x56242eaf24f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:16:57.165 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.158+0000 7fdb023d8b80 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x56242eae6460 0x56242eaf51f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:16:57.165 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.158+0000 7fdb023d8b80 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x56242eaf5730 con 0x56242eae5860 2023-07-14T19:16:57.166 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.158+0000 7fdb023d8b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x56242eaf58a0 con 0x56242eae4f10 2023-07-14T19:16:57.166 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.158+0000 7fdb023d8b80 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x56242eaf5a10 con 0x56242eae6460 2023-07-14T19:16:57.166 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.158+0000 7fdb023d8b80 10 client.0 _ll_register_callbacks cb 0x7fdae8099400 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-14T19:16:57.166 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.158+0000 7fdaeeffd700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56242eae5860 0x56242eaf24f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:16:57.166 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.158+0000 7fdaeeffd700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56242eae5860 0x56242eaf24f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:33810/0 (socket says 172.21.15.40:33810) 2023-07-14T19:16:57.166 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.158+0000 7fdaeeffd700 1 -- 172.21.15.40:0/4246817436 learned_addr learned my addr 172.21.15.40:0/4246817436 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-07-14T19:16:57.166 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.159+0000 7fdaef7fe700 1 --2- 172.21.15.40:0/4246817436 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56242eae4f10 0x56242eaf1f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:16:57.167 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.159+0000 7fdaeeffd700 1 -- 172.21.15.40:0/4246817436 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x56242eae6460 msgr2=0x56242eaf51f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T19:16:57.167 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.159+0000 7fdaeeffd700 1 --2- 172.21.15.40:0/4246817436 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x56242eae6460 0x56242eaf51f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:57.167 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.159+0000 7fdaeeffd700 1 -- 172.21.15.40:0/4246817436 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56242eae4f10 msgr2=0x56242eaf1f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T19:16:57.167 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.159+0000 7fdaeeffd700 1 --2- 172.21.15.40:0/4246817436 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56242eae4f10 0x56242eaf1f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:57.167 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.159+0000 7fdaeeffd700 1 -- 172.21.15.40:0/4246817436 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdae400a960 con 0x56242eae5860 2023-07-14T19:16:57.167 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.159+0000 7fdaef7fe700 1 --2- 172.21.15.40:0/4246817436 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56242eae4f10 0x56242eaf1f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T19:16:57.167 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.159+0000 7fdaeeffd700 1 --2- 172.21.15.40:0/4246817436 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56242eae5860 0x56242eaf24f0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7fdae4013860 tx=0x7fdae4005810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bdd1a581264debd5 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:16:57.167 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.159+0000 7fdaceffd700 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2023-07-14T19:16:57.168 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.159+0000 7fdaceffd700 1 -- 172.21.15.40:0/4246817436 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdae4004480 con 0x56242eae5860 2023-07-14T19:16:57.168 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.159+0000 7fdaceffd700 1 -- 172.21.15.40:0/4246817436 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(83..83 src has 1..83) v4 ==== 8269+0+0 (secure 0 0 0) 0x7fdae4013d40 con 0x56242eae5860 2023-07-14T19:16:57.168 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.159+0000 7fdb023d8b80 1 -- 172.21.15.40:0/4246817436 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fdae001e040 con 0x56242eae5860 2023-07-14T19:16:57.168 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.159+0000 7fdaceffd700 1 -- 172.21.15.40:0/4246817436 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7fdae400f6d0 con 0x56242eae5860 2023-07-14T19:16:57.168 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.159+0000 7fdb023d8b80 10 client.15048 fetch_fsmap learned FSMap version 26 2023-07-14T19:16:57.168 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.159+0000 7fdb023d8b80 1 -- 172.21.15.40:0/4246817436 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) v3 -- 0x56242e862260 con 0x56242eae5860 2023-07-14T19:16:57.168 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.159+0000 7fdaceffd700 1 -- 172.21.15.40:0/4246817436 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fdae001e040 con 0x56242eae5860 2023-07-14T19:16:57.169 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.160+0000 7fdaceffd700 1 -- 172.21.15.40:0/4246817436 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 26) v1 ==== 7462+0+0 (secure 0 0 0) 0x7fdae4024a00 con 0x56242eae5860 2023-07-14T19:16:57.169 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.160+0000 7fdb023d8b80 10 client.15048 fetch_fsmap finished waiting for FSMap version 26 2023-07-14T19:16:57.169 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.160+0000 7fdb023d8b80 10 client.15048 resolve_mds: resolved name 'h' to daemon mds.h 2023-07-14T19:16:57.169 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.160+0000 7fdb023d8b80 20 client.15048 populate_metadata read hostname 'smithi040' 2023-07-14T19:16:57.169 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.160+0000 7fdb023d8b80 1 --2- 172.21.15.40:0/4246817436 >> [v2:172.21.15.130:6836/2604019369,v1:172.21.15.130:6837/2604019369] conn(0x56242eb53a50 0x56242eb503c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:16:57.169 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.160+0000 7fdb023d8b80 4 client.15048 mds_command: new command op to 24506 tid=0[{"prefix": "status"}] 2023-07-14T19:16:57.169 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.160+0000 7fdb023d8b80 1 -- 172.21.15.40:0/4246817436 --> [v2:172.21.15.130:6836/2604019369,v1:172.21.15.130:6837/2604019369] -- command(tid 0: {"prefix": "status"}) v1 -- 0x56242ea2ee60 con 0x56242eb53a50 2023-07-14T19:16:57.170 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.160+0000 7fdaef7fe700 1 --2- 172.21.15.40:0/4246817436 >> [v2:172.21.15.130:6836/2604019369,v1:172.21.15.130:6837/2604019369] conn(0x56242eb53a50 0x56242eb503c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:16:57.170 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.160+0000 7fdaceffd700 10 client.15048 ms_handle_connect on v2:172.21.15.130:6836/2604019369 2023-07-14T19:16:57.170 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.160+0000 7fdaef7fe700 1 --2- 172.21.15.40:0/4246817436 >> [v2:172.21.15.130:6836/2604019369,v1:172.21.15.130:6837/2604019369] conn(0x56242eb53a50 0x56242eb503c0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=2216330b368a0871 server_cookie=6d74016c004465bc in_seq=0 out_seq=0 2023-07-14T19:16:57.170 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.160+0000 7fdaceffd700 1 -- 172.21.15.40:0/4246817436 <== mds.2 v2:172.21.15.130:6836/2604019369 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+332 (crc 0 0 0) 0x56242ea2ee60 con 0x56242eb53a50 2023-07-14T19:16:57.170 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.160+0000 7fdaceffd700 10 client.15048 handle_command_reply: tid=0 2023-07-14T19:16:57.170 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.160+0000 7fdb023d8b80 1 client.15048 shutdown 2023-07-14T19:16:57.170 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.161+0000 7fdb023d8b80 1 -- 172.21.15.40:0/4246817436 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56242eae5860 msgr2=0x56242eaf24f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:16:57.170 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.161+0000 7fdb023d8b80 1 --2- 172.21.15.40:0/4246817436 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56242eae5860 0x56242eaf24f0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7fdae4013860 tx=0x7fdae4005810 comp rx=0 tx=0).stop 2023-07-14T19:16:57.171 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.161+0000 7fdb023d8b80 1 -- 172.21.15.40:0/4246817436 shutdown_connections 2023-07-14T19:16:57.171 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.161+0000 7fdb023d8b80 1 --2- 172.21.15.40:0/4246817436 >> [v2:172.21.15.130:6836/2604019369,v1:172.21.15.130:6837/2604019369] conn(0x56242eb53a50 0x56242eb503c0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:57.171 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.161+0000 7fdb023d8b80 1 --2- 172.21.15.40:0/4246817436 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x56242eae6460 0x56242eaf51f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:57.171 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.161+0000 7fdb023d8b80 1 --2- 172.21.15.40:0/4246817436 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56242eae4f10 0x56242eaf1f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:57.171 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.161+0000 7fdaceffd700 0 client.15048 ms_handle_reset on v2:172.21.15.130:6836/2604019369 2023-07-14T19:16:57.171 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.161+0000 7fdb023d8b80 1 --2- 172.21.15.40:0/4246817436 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56242eae5860 0x56242eaf24f0 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:57.171 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.161+0000 7fdb023d8b80 1 -- 172.21.15.40:0/4246817436 >> 172.21.15.40:0/4246817436 conn(0x56242eada630 msgr2=0x56242eaec7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:16:57.171 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.161+0000 7fdb023d8b80 1 -- 172.21.15.40:0/4246817436 shutdown_connections 2023-07-14T19:16:57.172 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.161+0000 7fdb023d8b80 1 -- 172.21.15.40:0/4246817436 wait complete. 2023-07-14T19:16:57.172 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.161+0000 7fdb023d8b80 20 client.15048 trim_cache size 0 max 16384 2023-07-14T19:16:57.172 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2023-07-14T19:16:57.172 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_fsid": "adc52f04-2279-11ee-9b33-001a4aab830c", 2023-07-14T19:16:57.173 INFO:teuthology.orchestra.run.smithi040.stdout: "whoami": 2, 2023-07-14T19:16:57.173 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 24506, 2023-07-14T19:16:57.173 INFO:teuthology.orchestra.run.smithi040.stdout: "want_state": "up:active", 2023-07-14T19:16:57.173 INFO:teuthology.orchestra.run.smithi040.stdout: "state": "up:active", 2023-07-14T19:16:57.173 INFO:teuthology.orchestra.run.smithi040.stdout: "fs_name": "cephfs", 2023-07-14T19:16:57.173 INFO:teuthology.orchestra.run.smithi040.stdout: "rank_uptime": 54.577442566999999, 2023-07-14T19:16:57.174 INFO:teuthology.orchestra.run.smithi040.stdout: "mdsmap_epoch": 26, 2023-07-14T19:16:57.174 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap_epoch": 83, 2023-07-14T19:16:57.174 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap_epoch_barrier": 81, 2023-07-14T19:16:57.174 INFO:teuthology.orchestra.run.smithi040.stdout: "uptime": 73.638034794999996 2023-07-14T19:16:57.174 INFO:teuthology.orchestra.run.smithi040.stdout:} 2023-07-14T19:16:57.175 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.161+0000 7fdaf6240700 1 -- 172.21.15.40:0/2667838743 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7fdad80a11e0 msgr2=0x7fdad80a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:16:57.175 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.161+0000 7fdaf6240700 1 --2- 172.21.15.40:0/2667838743 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7fdad80a11e0 0x7fdad80a3690 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fdae000b5c0 tx=0x7fdae0005fd0 comp rx=0 tx=0).stop 2023-07-14T19:16:57.176 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.161+0000 7fdaf6240700 1 -- 172.21.15.40:0/2667838743 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdaf0101820 msgr2=0x7fdaf00ff380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:16:57.176 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.161+0000 7fdaf6240700 1 --2- 172.21.15.40:0/2667838743 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdaf0101820 0x7fdaf00ff380 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7fdae8007ee0 tx=0x7fdae800d3b0 comp rx=0 tx=0).stop 2023-07-14T19:16:57.176 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.161+0000 7fdaf6240700 1 -- 172.21.15.40:0/2667838743 shutdown_connections 2023-07-14T19:16:57.176 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.161+0000 7fdaf6240700 1 --2- 172.21.15.40:0/2667838743 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7fdad80a11e0 0x7fdad80a3690 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:57.176 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.161+0000 7fdaf6240700 1 --2- 172.21.15.40:0/2667838743 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdaf0101820 0x7fdaf00ff380 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:57.176 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.161+0000 7fdaf6240700 1 --2- 172.21.15.40:0/2667838743 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fdaf00fdec0 0x7fdaf0100800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:57.176 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.161+0000 7fdaf6240700 1 --2- 172.21.15.40:0/2667838743 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fdaf0100e90 0x7fdaf00fee40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:57.177 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.161+0000 7fdaf6240700 1 -- 172.21.15.40:0/2667838743 >> 172.21.15.40:0/2667838743 conn(0x7fdaf00f98a0 msgr2=0x7fdaf00fbbd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:16:57.177 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.162+0000 7fdaf6240700 1 -- 172.21.15.40:0/2667838743 shutdown_connections 2023-07-14T19:16:57.177 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.162+0000 7fdaf6240700 1 -- 172.21.15.40:0/2667838743 wait complete. 2023-07-14T19:16:57.177 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.i status 2023-07-14T19:16:57.222 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:16:56 smithi191 ceph-mon[98311]: pgmap v285: 97 pgs: 6 peering, 91 active+clean; 638 KiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 1 op/s; 83 B/s, 0 objects/s recovering 2023-07-14T19:16:57.222 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:16:56 smithi191 ceph-mon[98311]: from='client.? 172.21.15.40:0/3609328448' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T19:16:57.222 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:16:56 smithi191 ceph-mon[98311]: from='client.? 172.21.15.40:0/311948507' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T19:16:57.222 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:16:56 smithi191 ceph-mon[98311]: from='client.? 172.21.15.40:0/2968496269' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T19:16:57.242 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:16:56 smithi130 ceph-mon[97832]: pgmap v285: 97 pgs: 6 peering, 91 active+clean; 638 KiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 1 op/s; 83 B/s, 0 objects/s recovering 2023-07-14T19:16:57.242 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:16:56 smithi130 ceph-mon[97832]: from='client.? 172.21.15.40:0/3609328448' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T19:16:57.242 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:16:56 smithi130 ceph-mon[97832]: from='client.? 172.21.15.40:0/311948507' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T19:16:57.243 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:16:56 smithi130 ceph-mon[97832]: from='client.? 172.21.15.40:0/2968496269' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T19:16:57.304 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.303+0000 7f63d483c700 1 -- 172.21.15.40:0/550823056 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f63d00feb10 msgr2=0x7f63d00fef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:16:57.304 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.303+0000 7f63d483c700 1 --2- 172.21.15.40:0/550823056 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f63d00feb10 0x7f63d00fef20 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f63c0009990 tx=0x7f63c0009ca0 comp rx=0 tx=0).stop 2023-07-14T19:16:57.304 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.303+0000 7f63d483c700 1 -- 172.21.15.40:0/550823056 shutdown_connections 2023-07-14T19:16:57.304 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.303+0000 7f63d483c700 1 --2- 172.21.15.40:0/550823056 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f63d0100690 0x7f63d0106f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:57.305 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.303+0000 7f63d483c700 1 --2- 172.21.15.40:0/550823056 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f63d00feb10 0x7f63d00fef20 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:57.305 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.303+0000 7f63d483c700 1 --2- 172.21.15.40:0/550823056 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f63d00ffd00 0x7f63d0100150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:57.305 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.303+0000 7f63d483c700 1 -- 172.21.15.40:0/550823056 >> 172.21.15.40:0/550823056 conn(0x7f63d00fa130 msgr2=0x7f63d00fc560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:16:57.305 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.303+0000 7f63d483c700 1 -- 172.21.15.40:0/550823056 shutdown_connections 2023-07-14T19:16:57.305 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.303+0000 7f63d483c700 1 -- 172.21.15.40:0/550823056 wait complete. 2023-07-14T19:16:57.308 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.304+0000 7f63d483c700 1 Processor -- start 2023-07-14T19:16:57.308 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.304+0000 7f63d483c700 1 -- start start 2023-07-14T19:16:57.309 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.304+0000 7f63d483c700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f63d00feb10 0x7f63d0068310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:16:57.309 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.304+0000 7f63d483c700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f63d00ffd00 0x7f63d019a710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:16:57.309 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.304+0000 7f63d483c700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f63d0100690 0x7f63d019cd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:16:57.310 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.304+0000 7f63d483c700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63d019d4f0 con 0x7f63d00ffd00 2023-07-14T19:16:57.310 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.304+0000 7f63d483c700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63d019d660 con 0x7f63d0100690 2023-07-14T19:16:57.310 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.304+0000 7f63d483c700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63d019d820 con 0x7f63d00feb10 2023-07-14T19:16:57.311 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.304+0000 7f63c7fff700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f63d00ffd00 0x7f63d019a710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:16:57.311 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.304+0000 7f63c7fff700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f63d00ffd00 0x7f63d019a710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:33824/0 (socket says 172.21.15.40:33824) 2023-07-14T19:16:57.311 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.304+0000 7f63c7fff700 1 -- 172.21.15.40:0/1982785021 learned_addr learned my addr 172.21.15.40:0/1982785021 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-07-14T19:16:57.312 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.304+0000 7f63ce59c700 1 --2- 172.21.15.40:0/1982785021 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f63d00feb10 0x7f63d0068310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:16:57.312 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.304+0000 7f63ced9d700 1 --2- 172.21.15.40:0/1982785021 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f63d0100690 0x7f63d019cd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:16:57.312 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.304+0000 7f63c7fff700 1 -- 172.21.15.40:0/1982785021 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f63d00feb10 msgr2=0x7f63d0068310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:16:57.312 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.304+0000 7f63c7fff700 1 --2- 172.21.15.40:0/1982785021 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f63d00feb10 0x7f63d0068310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:57.313 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.304+0000 7f63c7fff700 1 -- 172.21.15.40:0/1982785021 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f63d0100690 msgr2=0x7f63d019cd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:16:57.313 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.304+0000 7f63c7fff700 1 --2- 172.21.15.40:0/1982785021 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f63d0100690 0x7f63d019cd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:57.313 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.304+0000 7f63c7fff700 1 -- 172.21.15.40:0/1982785021 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63c00095f0 con 0x7f63d00ffd00 2023-07-14T19:16:57.314 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.305+0000 7f63c7fff700 1 --2- 172.21.15.40:0/1982785021 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f63d00ffd00 0x7f63d019a710 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f63b800d980 tx=0x7f63b800dd40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:16:57.314 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.305+0000 7f63c77fe700 1 -- 172.21.15.40:0/1982785021 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63b800bb60 con 0x7f63d00ffd00 2023-07-14T19:16:57.314 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.305+0000 7f63c77fe700 1 -- 172.21.15.40:0/1982785021 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f63b8007640 con 0x7f63d00ffd00 2023-07-14T19:16:57.314 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.305+0000 7f63d483c700 1 -- 172.21.15.40:0/1982785021 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f63d019db00 con 0x7f63d00ffd00 2023-07-14T19:16:57.314 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.305+0000 7f63c77fe700 1 -- 172.21.15.40:0/1982785021 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63b8004e90 con 0x7f63d00ffd00 2023-07-14T19:16:57.315 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.305+0000 7f63d483c700 1 -- 172.21.15.40:0/1982785021 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f63d019e060 con 0x7f63d00ffd00 2023-07-14T19:16:57.315 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.307+0000 7f63c77fe700 1 -- 172.21.15.40:0/1982785021 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7f63b800bcc0 con 0x7f63d00ffd00 2023-07-14T19:16:57.321 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.308+0000 7f63c77fe700 1 --2- 172.21.15.40:0/1982785021 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f63bc0a1260 0x7f63bc0a3710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:16:57.321 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.308+0000 7f63c77fe700 1 -- 172.21.15.40:0/1982785021 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 8269+0+0 (secure 0 0 0) 0x7f63b80cc9c0 con 0x7f63d00ffd00 2023-07-14T19:16:57.321 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.308+0000 7f63ce59c700 1 --2- 172.21.15.40:0/1982785021 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f63bc0a1260 0x7f63bc0a3710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:16:57.321 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.309+0000 7f63ce59c700 1 --2- 172.21.15.40:0/1982785021 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f63bc0a1260 0x7f63bc0a3710 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f63c000b2e0 tx=0x7f63c0005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:16:57.322 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.309+0000 7f63e09d4b80 1 Processor -- start 2023-07-14T19:16:57.322 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.309+0000 7f63e09d4b80 1 -- start start 2023-07-14T19:16:57.322 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.309+0000 7f63e09d4b80 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5653aef0df10 0x5653aef0e320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:16:57.322 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.309+0000 7f63e09d4b80 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5653aef0e860 0x5653aef0ee80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:16:57.322 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.309+0000 7f63e09d4b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5653aef0f460 0x5653aef0f8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:16:57.322 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.309+0000 7f63e09d4b80 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x5653aee12c60 con 0x5653aef0e860 2023-07-14T19:16:57.322 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.309+0000 7f63e09d4b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5653aee659f0 con 0x5653aef0f460 2023-07-14T19:16:57.323 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.309+0000 7f63e09d4b80 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x5653aee5ee80 con 0x5653aef0df10 2023-07-14T19:16:57.323 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.309+0000 7f63c7fff700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5653aef0e860 0x5653aef0ee80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:16:57.323 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.309+0000 7f63ce59c700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5653aef0df10 0x5653aef0e320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:16:57.323 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.309+0000 7f63c7fff700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5653aef0e860 0x5653aef0ee80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:33832/0 (socket says 172.21.15.40:33832) 2023-07-14T19:16:57.323 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.309+0000 7f63c7fff700 1 -- 172.21.15.40:0/826201084 learned_addr learned my addr 172.21.15.40:0/826201084 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-07-14T19:16:57.323 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.310+0000 7f63ced9d700 1 --2- 172.21.15.40:0/826201084 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5653aef0f460 0x5653aef0f8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:16:57.323 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.310+0000 7f63c7fff700 1 -- 172.21.15.40:0/826201084 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5653aef0df10 msgr2=0x5653aef0e320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T19:16:57.323 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.310+0000 7f63c7fff700 1 --2- 172.21.15.40:0/826201084 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5653aef0df10 0x5653aef0e320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:57.324 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.310+0000 7f63c7fff700 1 -- 172.21.15.40:0/826201084 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5653aef0f460 msgr2=0x5653aef0f8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T19:16:57.324 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.310+0000 7f63c7fff700 1 --2- 172.21.15.40:0/826201084 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5653aef0f460 0x5653aef0f8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:57.324 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.310+0000 7f63c7fff700 1 -- 172.21.15.40:0/826201084 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5653aec8cd90 con 0x5653aef0e860 2023-07-14T19:16:57.324 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.310+0000 7f63ced9d700 1 --2- 172.21.15.40:0/826201084 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5653aef0f460 0x5653aef0f8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-14T19:16:57.324 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.310+0000 7f63c7fff700 1 --2- 172.21.15.40:0/826201084 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5653aef0e860 0x5653aef0ee80 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f63b8099be0 tx=0x7f63b80996e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a6dcfa304fe99a39 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:16:57.324 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.310+0000 7f63c4ff9700 1 -- 172.21.15.40:0/826201084 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63b809dc80 con 0x5653aef0e860 2023-07-14T19:16:57.325 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.310+0000 7f63c4ff9700 1 -- 172.21.15.40:0/826201084 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f63b80a20a0 con 0x5653aef0e860 2023-07-14T19:16:57.325 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.310+0000 7f63c4ff9700 1 -- 172.21.15.40:0/826201084 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63b80993a0 con 0x5653aef0e860 2023-07-14T19:16:57.325 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.311+0000 7f63e09d4b80 1 -- 172.21.15.40:0/826201084 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5653aef0e860 msgr2=0x5653aef0ee80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:16:57.325 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.311+0000 7f63e09d4b80 1 --2- 172.21.15.40:0/826201084 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5653aef0e860 0x5653aef0ee80 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f63b8099be0 tx=0x7f63b80996e0 comp rx=0 tx=0).stop 2023-07-14T19:16:57.325 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.311+0000 7f63e09d4b80 1 -- 172.21.15.40:0/826201084 shutdown_connections 2023-07-14T19:16:57.325 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.311+0000 7f63e09d4b80 1 --2- 172.21.15.40:0/826201084 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5653aef0df10 0x5653aef0e320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:57.325 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.311+0000 7f63e09d4b80 1 --2- 172.21.15.40:0/826201084 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5653aef0f460 0x5653aef0f8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:57.326 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.311+0000 7f63e09d4b80 1 --2- 172.21.15.40:0/826201084 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5653aef0e860 0x5653aef0ee80 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:57.326 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.311+0000 7f63e09d4b80 1 -- 172.21.15.40:0/826201084 >> 172.21.15.40:0/826201084 conn(0x5653aef03630 msgr2=0x5653aef08980 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:16:57.326 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.311+0000 7f63e09d4b80 1 -- 172.21.15.40:0/826201084 shutdown_connections 2023-07-14T19:16:57.326 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.311+0000 7f63e09d4b80 1 -- 172.21.15.40:0/826201084 wait complete. 2023-07-14T19:16:57.326 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.311+0000 7f63e09d4b80 1 -- start start 2023-07-14T19:16:57.327 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.311+0000 7f63e09d4b80 1 Processor -- start 2023-07-14T19:16:57.327 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.311+0000 7f63e09d4b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5653aef0df10 0x5653aef17970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:16:57.327 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.311+0000 7f63e09d4b80 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5653aef0e860 0x5653aef17eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:16:57.327 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.311+0000 7f63e09d4b80 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5653aef0f460 0x5653aef1eef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:16:57.327 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.311+0000 7f63e09d4b80 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x5653aedfea70 con 0x5653aef0e860 2023-07-14T19:16:57.327 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.311+0000 7f63e09d4b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5653aedfe770 con 0x5653aef0df10 2023-07-14T19:16:57.327 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.311+0000 7f63e09d4b80 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x5653aece28d0 con 0x5653aef0f460 2023-07-14T19:16:57.328 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.312+0000 7f63e09d4b80 10 client.0 _ll_register_callbacks cb 0x5653aedfc710 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-14T19:16:57.328 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.312+0000 7f63c7fff700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5653aef0e860 0x5653aef17eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:16:57.328 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.312+0000 7f63c7fff700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5653aef0e860 0x5653aef17eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:33838/0 (socket says 172.21.15.40:33838) 2023-07-14T19:16:57.328 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.312+0000 7f63c7fff700 1 -- 172.21.15.40:0/730678969 learned_addr learned my addr 172.21.15.40:0/730678969 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-07-14T19:16:57.328 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.312+0000 7f63ced9d700 1 --2- 172.21.15.40:0/730678969 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5653aef0f460 0x5653aef1eef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:16:57.328 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.312+0000 7f63ce59c700 1 --2- 172.21.15.40:0/730678969 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5653aef0df10 0x5653aef17970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:16:57.329 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.312+0000 7f63c7fff700 1 -- 172.21.15.40:0/730678969 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5653aef0f460 msgr2=0x5653aef1eef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T19:16:57.329 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.312+0000 7f63c7fff700 1 --2- 172.21.15.40:0/730678969 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5653aef0f460 0x5653aef1eef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:57.329 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.312+0000 7f63c7fff700 1 -- 172.21.15.40:0/730678969 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5653aef0df10 msgr2=0x5653aef17970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T19:16:57.329 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.312+0000 7f63c7fff700 1 --2- 172.21.15.40:0/730678969 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5653aef0df10 0x5653aef17970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:57.329 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.312+0000 7f63c7fff700 1 -- 172.21.15.40:0/730678969 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f63d019e060 con 0x5653aef0e860 2023-07-14T19:16:57.329 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.312+0000 7f63c7fff700 1 --2- 172.21.15.40:0/730678969 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5653aef0e860 0x5653aef17eb0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f63b800b540 tx=0x7f63b80998c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dd7201195988968 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:16:57.329 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.312+0000 7f63b5ffb700 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2023-07-14T19:16:57.330 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.312+0000 7f63b5ffb700 1 -- 172.21.15.40:0/730678969 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63b809dc80 con 0x5653aef0e860 2023-07-14T19:16:57.330 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.312+0000 7f63b5ffb700 1 -- 172.21.15.40:0/730678969 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(83..83 src has 1..83) v4 ==== 8269+0+0 (secure 0 0 0) 0x7f63b8099350 con 0x5653aef0e860 2023-07-14T19:16:57.330 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.312+0000 7f63e09d4b80 1 -- 172.21.15.40:0/730678969 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5653aedb84a0 con 0x5653aef0e860 2023-07-14T19:16:57.330 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.312+0000 7f63b5ffb700 1 -- 172.21.15.40:0/730678969 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7f63b80a1c10 con 0x5653aef0e860 2023-07-14T19:16:57.330 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.312+0000 7f63e09d4b80 10 client.15060 fetch_fsmap learned FSMap version 26 2023-07-14T19:16:57.330 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.312+0000 7f63e09d4b80 1 -- 172.21.15.40:0/730678969 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) v3 -- 0x5653aec75c10 con 0x5653aef0e860 2023-07-14T19:16:57.330 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.313+0000 7f63b5ffb700 1 -- 172.21.15.40:0/730678969 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f63b807c080 con 0x5653aef0e860 2023-07-14T19:16:57.330 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.313+0000 7f63b5ffb700 1 -- 172.21.15.40:0/730678969 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 26) v1 ==== 7462+0+0 (secure 0 0 0) 0x7f63b80b9300 con 0x5653aef0e860 2023-07-14T19:16:57.331 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.313+0000 7f63e09d4b80 10 client.15060 fetch_fsmap finished waiting for FSMap version 26 2023-07-14T19:16:57.331 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.313+0000 7f63e09d4b80 10 client.15060 resolve_mds: resolved name 'i' to daemon mds.i 2023-07-14T19:16:57.331 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.313+0000 7f63e09d4b80 20 client.15060 populate_metadata read hostname 'smithi040' 2023-07-14T19:16:57.331 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.313+0000 7f63e09d4b80 1 --2- 172.21.15.40:0/730678969 >> [v2:172.21.15.191:6838/1977507302,v1:172.21.15.191:6839/1977507302] conn(0x5653aef22c80 0x5653aef21d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:16:57.331 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.313+0000 7f63e09d4b80 4 client.15060 mds_command: new command op to 24508 tid=0[{"prefix": "get_command_descriptions"}] 2023-07-14T19:16:57.331 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.313+0000 7f63e09d4b80 1 -- 172.21.15.40:0/730678969 --> [v2:172.21.15.191:6838/1977507302,v1:172.21.15.191:6839/1977507302] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5653aeedcd50 con 0x5653aef22c80 2023-07-14T19:16:57.331 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.313+0000 7f63ce59c700 1 --2- 172.21.15.40:0/730678969 >> [v2:172.21.15.191:6838/1977507302,v1:172.21.15.191:6839/1977507302] conn(0x5653aef22c80 0x5653aef21d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:16:57.331 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.313+0000 7f63ce59c700 1 --2- 172.21.15.40:0/730678969 >> [v2:172.21.15.191:6838/1977507302,v1:172.21.15.191:6839/1977507302] conn(0x5653aef22c80 0x5653aef21d90 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=5222d3bd106c467b server_cookie=8f790780e7d58d00 in_seq=0 out_seq=0 2023-07-14T19:16:57.332 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.313+0000 7f63b5ffb700 10 client.15060 ms_handle_connect on v2:172.21.15.191:6838/1977507302 2023-07-14T19:16:57.332 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.314+0000 7f63b5ffb700 1 -- 172.21.15.40:0/730678969 <== mds.1 v2:172.21.15.191:6838/1977507302 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18017 (crc 0 0 0) 0x5653aeedcd50 con 0x5653aef22c80 2023-07-14T19:16:57.332 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.314+0000 7f63b5ffb700 10 client.15060 handle_command_reply: tid=0 2023-07-14T19:16:57.332 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.314+0000 7f63e09d4b80 1 client.15060 shutdown 2023-07-14T19:16:57.332 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.314+0000 7f63e09d4b80 1 -- 172.21.15.40:0/730678969 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5653aef0e860 msgr2=0x5653aef17eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:16:57.333 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.314+0000 7f63e09d4b80 1 --2- 172.21.15.40:0/730678969 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5653aef0e860 0x5653aef17eb0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f63b800b540 tx=0x7f63b80998c0 comp rx=0 tx=0).stop 2023-07-14T19:16:57.333 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.314+0000 7f63e09d4b80 1 -- 172.21.15.40:0/730678969 shutdown_connections 2023-07-14T19:16:57.333 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.314+0000 7f63e09d4b80 1 --2- 172.21.15.40:0/730678969 >> [v2:172.21.15.191:6838/1977507302,v1:172.21.15.191:6839/1977507302] conn(0x5653aef22c80 0x5653aef21d90 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:57.333 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.314+0000 7f63e09d4b80 1 --2- 172.21.15.40:0/730678969 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5653aef0f460 0x5653aef1eef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:57.333 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.314+0000 7f63e09d4b80 1 --2- 172.21.15.40:0/730678969 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5653aef0df10 0x5653aef17970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:57.333 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.314+0000 7f63e09d4b80 1 --2- 172.21.15.40:0/730678969 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5653aef0e860 0x5653aef17eb0 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:57.334 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.314+0000 7f63b5ffb700 0 client.15060 ms_handle_reset on v2:172.21.15.191:6838/1977507302 2023-07-14T19:16:57.334 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.314+0000 7f63e09d4b80 1 -- 172.21.15.40:0/730678969 >> 172.21.15.40:0/730678969 conn(0x5653aef03630 msgr2=0x5653aef07600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:16:57.334 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.315+0000 7f63e09d4b80 1 -- 172.21.15.40:0/730678969 shutdown_connections 2023-07-14T19:16:57.334 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.315+0000 7f63e09d4b80 1 -- 172.21.15.40:0/730678969 wait complete. 2023-07-14T19:16:57.334 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.315+0000 7f63e09d4b80 20 client.15060 trim_cache size 0 max 16384 2023-07-14T19:16:57.339 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.332+0000 7f63e09d4b80 1 Processor -- start 2023-07-14T19:16:57.339 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.332+0000 7f63e09d4b80 1 -- start start 2023-07-14T19:16:57.339 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.332+0000 7f63e09d4b80 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5653aef0df10 0x5653aef157f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:16:57.339 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.332+0000 7f63e09d4b80 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5653aef0e860 0x5653aef15d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:16:57.340 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.332+0000 7f63e09d4b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5653aef0f460 0x5653aef1eeb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:16:57.340 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.332+0000 7f63e09d4b80 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x5653aeedcd50 con 0x5653aef0df10 2023-07-14T19:16:57.340 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.332+0000 7f63e09d4b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5653aef09e30 con 0x5653aef0f460 2023-07-14T19:16:57.340 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.332+0000 7f63e09d4b80 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x5653aee08470 con 0x5653aef0e860 2023-07-14T19:16:57.340 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.333+0000 7f63ce59c700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5653aef0df10 0x5653aef157f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:16:57.340 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.333+0000 7f63ce59c700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5653aef0df10 0x5653aef157f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:33844/0 (socket says 172.21.15.40:33844) 2023-07-14T19:16:57.340 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.333+0000 7f63ce59c700 1 -- 172.21.15.40:0/459357888 learned_addr learned my addr 172.21.15.40:0/459357888 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-07-14T19:16:57.341 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.333+0000 7f63ce59c700 1 -- 172.21.15.40:0/459357888 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5653aef0e860 msgr2=0x5653aef15d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T19:16:57.341 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.333+0000 7f63ce59c700 1 --2- 172.21.15.40:0/459357888 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5653aef0e860 0x5653aef15d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:57.341 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.333+0000 7f63ce59c700 1 -- 172.21.15.40:0/459357888 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5653aef0f460 msgr2=0x5653aef1eeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T19:16:57.341 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.333+0000 7f63ce59c700 1 --2- 172.21.15.40:0/459357888 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5653aef0f460 0x5653aef1eeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:57.341 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.333+0000 7f63ce59c700 1 -- 172.21.15.40:0/459357888 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5653aecb3200 con 0x5653aef0df10 2023-07-14T19:16:57.341 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.333+0000 7f63ce59c700 1 --2- 172.21.15.40:0/459357888 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5653aef0df10 0x5653aef157f0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f63c0009990 tx=0x7f63c001f000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=685ec00aa7aa95d6 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:16:57.341 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.333+0000 7f63c57fa700 1 -- 172.21.15.40:0/459357888 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63c002d040 con 0x5653aef0df10 2023-07-14T19:16:57.341 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.333+0000 7f63c57fa700 1 -- 172.21.15.40:0/459357888 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f63c001e040 con 0x5653aef0df10 2023-07-14T19:16:57.342 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.333+0000 7f63c57fa700 1 -- 172.21.15.40:0/459357888 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63c0004a60 con 0x5653aef0df10 2023-07-14T19:16:57.342 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.334+0000 7f63e09d4b80 1 -- 172.21.15.40:0/459357888 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5653aef0df10 msgr2=0x5653aef157f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:16:57.342 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.334+0000 7f63e09d4b80 1 --2- 172.21.15.40:0/459357888 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5653aef0df10 0x5653aef157f0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f63c0009990 tx=0x7f63c001f000 comp rx=0 tx=0).stop 2023-07-14T19:16:57.342 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.334+0000 7f63e09d4b80 1 -- 172.21.15.40:0/459357888 shutdown_connections 2023-07-14T19:16:57.342 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.334+0000 7f63e09d4b80 1 --2- 172.21.15.40:0/459357888 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5653aef0f460 0x5653aef1eeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:57.342 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.334+0000 7f63e09d4b80 1 --2- 172.21.15.40:0/459357888 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5653aef0df10 0x5653aef157f0 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:57.342 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.334+0000 7f63e09d4b80 1 --2- 172.21.15.40:0/459357888 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5653aef0e860 0x5653aef15d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:57.343 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.334+0000 7f63e09d4b80 1 -- 172.21.15.40:0/459357888 >> 172.21.15.40:0/459357888 conn(0x5653aef03630 msgr2=0x5653aef16620 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:16:57.343 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.334+0000 7f63e09d4b80 1 -- 172.21.15.40:0/459357888 shutdown_connections 2023-07-14T19:16:57.343 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.334+0000 7f63e09d4b80 1 -- 172.21.15.40:0/459357888 wait complete. 2023-07-14T19:16:57.343 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.334+0000 7f63e09d4b80 1 -- start start 2023-07-14T19:16:57.343 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.334+0000 7f63e09d4b80 1 Processor -- start 2023-07-14T19:16:57.343 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.334+0000 7f63e09d4b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5653aef0df10 0x5653aef1d050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:16:57.343 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.334+0000 7f63e09d4b80 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5653aef0e860 0x5653aef1d590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:16:57.344 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.334+0000 7f63e09d4b80 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5653aef0f460 0x5653aef1dad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:16:57.344 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.334+0000 7f63e09d4b80 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x5653aef1e170 con 0x5653aef0e860 2023-07-14T19:16:57.344 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.334+0000 7f63e09d4b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5653aef1e2e0 con 0x5653aef0df10 2023-07-14T19:16:57.344 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.334+0000 7f63e09d4b80 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x5653aef1e4a0 con 0x5653aef0f460 2023-07-14T19:16:57.344 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.334+0000 7f63e09d4b80 10 client.0 _ll_register_callbacks cb 0x7f63c0005e80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-14T19:16:57.345 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.334+0000 7f63c7fff700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5653aef0e860 0x5653aef1d590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:16:57.345 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.334+0000 7f63c7fff700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5653aef0e860 0x5653aef1d590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:33852/0 (socket says 172.21.15.40:33852) 2023-07-14T19:16:57.345 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.334+0000 7f63c7fff700 1 -- 172.21.15.40:0/1949301324 learned_addr learned my addr 172.21.15.40:0/1949301324 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-07-14T19:16:57.345 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.335+0000 7f63ced9d700 1 --2- 172.21.15.40:0/1949301324 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5653aef0f460 0x5653aef1dad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:16:57.346 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.335+0000 7f63ce59c700 1 --2- 172.21.15.40:0/1949301324 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5653aef0df10 0x5653aef1d050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:16:57.346 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.335+0000 7f63c7fff700 1 -- 172.21.15.40:0/1949301324 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5653aef0f460 msgr2=0x5653aef1dad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T19:16:57.346 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.335+0000 7f63c7fff700 1 --2- 172.21.15.40:0/1949301324 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5653aef0f460 0x5653aef1dad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:57.347 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.335+0000 7f63c7fff700 1 -- 172.21.15.40:0/1949301324 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5653aef0df10 msgr2=0x5653aef1d050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T19:16:57.347 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.335+0000 7f63c7fff700 1 --2- 172.21.15.40:0/1949301324 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5653aef0df10 0x5653aef1d050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:57.347 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.335+0000 7f63c7fff700 1 -- 172.21.15.40:0/1949301324 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f63d0107880 con 0x5653aef0e860 2023-07-14T19:16:57.347 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.335+0000 7f63c7fff700 1 --2- 172.21.15.40:0/1949301324 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5653aef0e860 0x5653aef1d590 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f63b80966c0 tx=0x7f63b800b6e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5111c15985140a20 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:16:57.348 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.335+0000 7f63b5ffb700 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2023-07-14T19:16:57.348 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.335+0000 7f63b5ffb700 1 -- 172.21.15.40:0/1949301324 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63b800f440 con 0x5653aef0e860 2023-07-14T19:16:57.348 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.335+0000 7f63e09d4b80 1 -- 172.21.15.40:0/1949301324 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f63b80a20a0 con 0x5653aef0e860 2023-07-14T19:16:57.348 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.335+0000 7f63b5ffb700 1 -- 172.21.15.40:0/1949301324 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(83..83 src has 1..83) v4 ==== 8269+0+0 (secure 0 0 0) 0x7f63b809dc80 con 0x5653aef0e860 2023-07-14T19:16:57.349 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.335+0000 7f63b5ffb700 1 -- 172.21.15.40:0/1949301324 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7f63b8099790 con 0x5653aef0e860 2023-07-14T19:16:57.349 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.335+0000 7f63e09d4b80 10 client.15066 fetch_fsmap learned FSMap version 26 2023-07-14T19:16:57.349 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.335+0000 7f63e09d4b80 1 -- 172.21.15.40:0/1949301324 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) v3 -- 0x5653aec8b260 con 0x5653aef0e860 2023-07-14T19:16:57.350 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.335+0000 7f63b5ffb700 1 -- 172.21.15.40:0/1949301324 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f63b80a20a0 con 0x5653aef0e860 2023-07-14T19:16:57.350 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.335+0000 7f63b5ffb700 1 -- 172.21.15.40:0/1949301324 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 26) v1 ==== 7462+0+0 (secure 0 0 0) 0x7f63b8083070 con 0x5653aef0e860 2023-07-14T19:16:57.350 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.336+0000 7f63e09d4b80 10 client.15066 fetch_fsmap finished waiting for FSMap version 26 2023-07-14T19:16:57.350 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.336+0000 7f63e09d4b80 10 client.15066 resolve_mds: resolved name 'i' to daemon mds.i 2023-07-14T19:16:57.351 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.336+0000 7f63e09d4b80 20 client.15066 populate_metadata read hostname 'smithi040' 2023-07-14T19:16:57.351 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.336+0000 7f63e09d4b80 1 --2- 172.21.15.40:0/1949301324 >> [v2:172.21.15.191:6838/1977507302,v1:172.21.15.191:6839/1977507302] conn(0x5653aef7bba0 0x5653aef78630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:16:57.351 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.336+0000 7f63e09d4b80 4 client.15066 mds_command: new command op to 24508 tid=0[{"prefix": "status"}] 2023-07-14T19:16:57.351 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.336+0000 7f63e09d4b80 1 -- 172.21.15.40:0/1949301324 --> [v2:172.21.15.191:6838/1977507302,v1:172.21.15.191:6839/1977507302] -- command(tid 0: {"prefix": "status"}) v1 -- 0x5653aee57e60 con 0x5653aef7bba0 2023-07-14T19:16:57.352 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.336+0000 7f63ce59c700 1 --2- 172.21.15.40:0/1949301324 >> [v2:172.21.15.191:6838/1977507302,v1:172.21.15.191:6839/1977507302] conn(0x5653aef7bba0 0x5653aef78630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:16:57.352 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.336+0000 7f63b5ffb700 10 client.15066 ms_handle_connect on v2:172.21.15.191:6838/1977507302 2023-07-14T19:16:57.352 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.336+0000 7f63ce59c700 1 --2- 172.21.15.40:0/1949301324 >> [v2:172.21.15.191:6838/1977507302,v1:172.21.15.191:6839/1977507302] conn(0x5653aef7bba0 0x5653aef78630 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=99815b719131ec4f server_cookie=aac48bb5f05140f7 in_seq=0 out_seq=0 2023-07-14T19:16:57.352 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.337+0000 7f63b5ffb700 1 -- 172.21.15.40:0/1949301324 <== mds.1 v2:172.21.15.191:6838/1977507302 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+332 (crc 0 0 0) 0x5653aee57e60 con 0x5653aef7bba0 2023-07-14T19:16:57.353 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.337+0000 7f63b5ffb700 10 client.15066 handle_command_reply: tid=0 2023-07-14T19:16:57.353 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.337+0000 7f63e09d4b80 1 client.15066 shutdown 2023-07-14T19:16:57.353 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.337+0000 7f63e09d4b80 1 -- 172.21.15.40:0/1949301324 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5653aef0e860 msgr2=0x5653aef1d590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:16:57.354 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.337+0000 7f63e09d4b80 1 --2- 172.21.15.40:0/1949301324 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5653aef0e860 0x5653aef1d590 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f63b80966c0 tx=0x7f63b800b6e0 comp rx=0 tx=0).stop 2023-07-14T19:16:57.354 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.337+0000 7f63e09d4b80 1 -- 172.21.15.40:0/1949301324 shutdown_connections 2023-07-14T19:16:57.354 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.337+0000 7f63e09d4b80 1 --2- 172.21.15.40:0/1949301324 >> [v2:172.21.15.191:6838/1977507302,v1:172.21.15.191:6839/1977507302] conn(0x5653aef7bba0 0x5653aef78630 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:57.354 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.337+0000 7f63e09d4b80 1 --2- 172.21.15.40:0/1949301324 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5653aef0f460 0x5653aef1dad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:57.355 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.337+0000 7f63e09d4b80 1 --2- 172.21.15.40:0/1949301324 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5653aef0df10 0x5653aef1d050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:57.355 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.337+0000 7f63e09d4b80 1 --2- 172.21.15.40:0/1949301324 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5653aef0e860 0x5653aef1d590 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:57.355 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.337+0000 7f63b5ffb700 0 client.15066 ms_handle_reset on v2:172.21.15.191:6838/1977507302 2023-07-14T19:16:57.356 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.337+0000 7f63e09d4b80 1 -- 172.21.15.40:0/1949301324 >> 172.21.15.40:0/1949301324 conn(0x5653aef03630 msgr2=0x5653aef157f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:16:57.356 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.337+0000 7f63e09d4b80 1 -- 172.21.15.40:0/1949301324 shutdown_connections 2023-07-14T19:16:57.356 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.337+0000 7f63e09d4b80 1 -- 172.21.15.40:0/1949301324 wait complete. 2023-07-14T19:16:57.356 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.337+0000 7f63e09d4b80 20 client.15066 trim_cache size 0 max 16384 2023-07-14T19:16:57.357 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2023-07-14T19:16:57.357 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_fsid": "adc52f04-2279-11ee-9b33-001a4aab830c", 2023-07-14T19:16:57.358 INFO:teuthology.orchestra.run.smithi040.stdout: "whoami": 1, 2023-07-14T19:16:57.358 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 24508, 2023-07-14T19:16:57.358 INFO:teuthology.orchestra.run.smithi040.stdout: "want_state": "up:active", 2023-07-14T19:16:57.359 INFO:teuthology.orchestra.run.smithi040.stdout: "state": "up:active", 2023-07-14T19:16:57.359 INFO:teuthology.orchestra.run.smithi040.stdout: "fs_name": "cephfs", 2023-07-14T19:16:57.359 INFO:teuthology.orchestra.run.smithi040.stdout: "rank_uptime": 55.774403614999997, 2023-07-14T19:16:57.359 INFO:teuthology.orchestra.run.smithi040.stdout: "mdsmap_epoch": 26, 2023-07-14T19:16:57.359 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap_epoch": 83, 2023-07-14T19:16:57.359 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap_epoch_barrier": 81, 2023-07-14T19:16:57.359 INFO:teuthology.orchestra.run.smithi040.stdout: "uptime": 71.471808770999999 2023-07-14T19:16:57.360 INFO:teuthology.orchestra.run.smithi040.stdout:} 2023-07-14T19:16:57.361 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.337+0000 7f63d483c700 1 -- 172.21.15.40:0/1982785021 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f63bc0a1260 msgr2=0x7f63bc0a3710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:16:57.361 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.337+0000 7f63d483c700 1 --2- 172.21.15.40:0/1982785021 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f63bc0a1260 0x7f63bc0a3710 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f63c000b2e0 tx=0x7f63c0005fb0 comp rx=0 tx=0).stop 2023-07-14T19:16:57.361 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.337+0000 7f63d483c700 1 -- 172.21.15.40:0/1982785021 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f63d00ffd00 msgr2=0x7f63d019a710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:16:57.361 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.337+0000 7f63d483c700 1 --2- 172.21.15.40:0/1982785021 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f63d00ffd00 0x7f63d019a710 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f63b800d980 tx=0x7f63b800dd40 comp rx=0 tx=0).stop 2023-07-14T19:16:57.361 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.337+0000 7f63d483c700 1 -- 172.21.15.40:0/1982785021 shutdown_connections 2023-07-14T19:16:57.362 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.337+0000 7f63d483c700 1 --2- 172.21.15.40:0/1982785021 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f63bc0a1260 0x7f63bc0a3710 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:57.362 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.337+0000 7f63d483c700 1 --2- 172.21.15.40:0/1982785021 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f63d00feb10 0x7f63d0068310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:57.362 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.337+0000 7f63d483c700 1 --2- 172.21.15.40:0/1982785021 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f63d0100690 0x7f63d019cd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:57.362 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.337+0000 7f63d483c700 1 --2- 172.21.15.40:0/1982785021 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f63d00ffd00 0x7f63d019a710 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:57.362 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.337+0000 7f63d483c700 1 -- 172.21.15.40:0/1982785021 >> 172.21.15.40:0/1982785021 conn(0x7f63d00fa130 msgr2=0x7f63d00fc3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:16:57.362 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.338+0000 7f63d483c700 1 -- 172.21.15.40:0/1982785021 shutdown_connections 2023-07-14T19:16:57.362 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.338+0000 7f63d483c700 1 -- 172.21.15.40:0/1982785021 wait complete. 2023-07-14T19:16:57.363 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2023-07-14T19:16:57.492 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.488+0000 7f20518c3700 1 -- 172.21.15.40:0/2031733366 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f204c0ffd30 msgr2=0x7f204c100180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:16:57.493 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.488+0000 7f20518c3700 1 --2- 172.21.15.40:0/2031733366 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f204c0ffd30 0x7f204c100180 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f203c009990 tx=0x7f203c009ca0 comp rx=0 tx=0).stop 2023-07-14T19:16:57.493 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.488+0000 7f20518c3700 1 -- 172.21.15.40:0/2031733366 shutdown_connections 2023-07-14T19:16:57.493 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.488+0000 7f20518c3700 1 --2- 172.21.15.40:0/2031733366 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f204c0feb40 0x7f204c0fef50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:57.493 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.488+0000 7f20518c3700 1 --2- 172.21.15.40:0/2031733366 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f204c1006c0 0x7f204c106f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:57.494 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.488+0000 7f20518c3700 1 --2- 172.21.15.40:0/2031733366 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f204c0ffd30 0x7f204c100180 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:57.495 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.488+0000 7f20518c3700 1 -- 172.21.15.40:0/2031733366 >> 172.21.15.40:0/2031733366 conn(0x7f204c0fa160 msgr2=0x7f204c0fc590 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:16:57.495 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.488+0000 7f20518c3700 1 -- 172.21.15.40:0/2031733366 shutdown_connections 2023-07-14T19:16:57.496 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.488+0000 7f20518c3700 1 -- 172.21.15.40:0/2031733366 wait complete. 2023-07-14T19:16:57.496 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.488+0000 7f20518c3700 1 Processor -- start 2023-07-14T19:16:57.496 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.489+0000 7f20518c3700 1 -- start start 2023-07-14T19:16:57.496 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.489+0000 7f20518c3700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f204c0feb40 0x7f204c194870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:16:57.496 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.489+0000 7f20518c3700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f204c0ffd30 0x7f204c194db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:16:57.496 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.489+0000 7f20518c3700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f204c1006c0 0x7f204c19be20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:16:57.497 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.489+0000 7f20518c3700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f204c19c3f0 con 0x7f204c0ffd30 2023-07-14T19:16:57.497 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.489+0000 7f20518c3700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f204c19c530 con 0x7f204c0feb40 2023-07-14T19:16:57.497 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.489+0000 7f20518c3700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f204c19c6c0 con 0x7f204c1006c0 2023-07-14T19:16:57.497 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.489+0000 7f204a7fc700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f204c0ffd30 0x7f204c194db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:16:57.497 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.489+0000 7f204a7fc700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f204c0ffd30 0x7f204c194db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:33866/0 (socket says 172.21.15.40:33866) 2023-07-14T19:16:57.497 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.489+0000 7f204a7fc700 1 -- 172.21.15.40:0/2404362909 learned_addr learned my addr 172.21.15.40:0/2404362909 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-07-14T19:16:57.497 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.489+0000 7f204b7fe700 1 --2- 172.21.15.40:0/2404362909 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f204c1006c0 0x7f204c19be20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:16:57.497 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.489+0000 7f204affd700 1 --2- 172.21.15.40:0/2404362909 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f204c0feb40 0x7f204c194870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:16:57.498 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.489+0000 7f204a7fc700 1 -- 172.21.15.40:0/2404362909 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f204c1006c0 msgr2=0x7f204c19be20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:16:57.498 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.489+0000 7f204a7fc700 1 --2- 172.21.15.40:0/2404362909 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f204c1006c0 0x7f204c19be20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:57.498 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.489+0000 7f204a7fc700 1 -- 172.21.15.40:0/2404362909 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f204c0feb40 msgr2=0x7f204c194870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:16:57.498 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.489+0000 7f204a7fc700 1 --2- 172.21.15.40:0/2404362909 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f204c0feb40 0x7f204c194870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:57.498 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.489+0000 7f204a7fc700 1 -- 172.21.15.40:0/2404362909 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f203c0095f0 con 0x7f204c0ffd30 2023-07-14T19:16:57.498 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.490+0000 7f204a7fc700 1 --2- 172.21.15.40:0/2404362909 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f204c0ffd30 0x7f204c194db0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f203c004760 tx=0x7f203c004790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:16:57.498 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.490+0000 7f20508c1700 1 -- 172.21.15.40:0/2404362909 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f203c01d070 con 0x7f204c0ffd30 2023-07-14T19:16:57.499 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.490+0000 7f20508c1700 1 -- 172.21.15.40:0/2404362909 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f203c004830 con 0x7f204c0ffd30 2023-07-14T19:16:57.499 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.490+0000 7f20518c3700 1 -- 172.21.15.40:0/2404362909 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f204c19c940 con 0x7f204c0ffd30 2023-07-14T19:16:57.499 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.490+0000 7f20508c1700 1 -- 172.21.15.40:0/2404362909 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f203c005350 con 0x7f204c0ffd30 2023-07-14T19:16:57.499 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.490+0000 7f20518c3700 1 -- 172.21.15.40:0/2404362909 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f204c19cd00 con 0x7f204c0ffd30 2023-07-14T19:16:57.504 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.493+0000 7f205da5bb80 1 Processor -- start 2023-07-14T19:16:57.504 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.493+0000 7f205da5bb80 1 -- start start 2023-07-14T19:16:57.504 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.493+0000 7f205da5bb80 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564b933370d0 0x564b933374e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:16:57.504 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.493+0000 7f205da5bb80 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x564b93337a20 0x564b933380c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:16:57.505 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.493+0000 7f205da5bb80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564b93338650 0x564b93338ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:16:57.505 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.493+0000 7f205da5bb80 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x564b93239c60 con 0x564b933370d0 2023-07-14T19:16:57.505 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.493+0000 7f205da5bb80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x564b9328c9f0 con 0x564b93338650 2023-07-14T19:16:57.505 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.493+0000 7f205da5bb80 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x564b93285e80 con 0x564b93337a20 2023-07-14T19:16:57.505 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.493+0000 7f204b7fe700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564b933370d0 0x564b933374e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:16:57.505 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.493+0000 7f204b7fe700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564b933370d0 0x564b933374e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:33870/0 (socket says 172.21.15.40:33870) 2023-07-14T19:16:57.505 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.493+0000 7f204b7fe700 1 -- 172.21.15.40:0/2876076400 learned_addr learned my addr 172.21.15.40:0/2876076400 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-07-14T19:16:57.506 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.494+0000 7f204affd700 1 --2- 172.21.15.40:0/2876076400 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x564b93337a20 0x564b933380c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:16:57.506 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.494+0000 7f204a7fc700 1 --2- 172.21.15.40:0/2876076400 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564b93338650 0x564b93338ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:16:57.506 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.494+0000 7f204b7fe700 1 -- 172.21.15.40:0/2876076400 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x564b93337a20 msgr2=0x564b933380c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T19:16:57.506 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.494+0000 7f204b7fe700 1 --2- 172.21.15.40:0/2876076400 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x564b93337a20 0x564b933380c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:57.506 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.494+0000 7f204b7fe700 1 -- 172.21.15.40:0/2876076400 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564b93338650 msgr2=0x564b93338ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T19:16:57.506 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.494+0000 7f204b7fe700 1 --2- 172.21.15.40:0/2876076400 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564b93338650 0x564b93338ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:57.506 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.494+0000 7f204b7fe700 1 -- 172.21.15.40:0/2876076400 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x564b930b3d90 con 0x564b933370d0 2023-07-14T19:16:57.507 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.494+0000 7f204b7fe700 1 --2- 172.21.15.40:0/2876076400 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564b933370d0 0x564b933374e0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f2044011680 tx=0x7f2044009920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=356db0af131f59b0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:16:57.507 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.494+0000 7f2031ffb700 1 -- 172.21.15.40:0/2876076400 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f204400e040 con 0x564b933370d0 2023-07-14T19:16:57.507 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.494+0000 7f2031ffb700 1 -- 172.21.15.40:0/2876076400 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f204400f040 con 0x564b933370d0 2023-07-14T19:16:57.507 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.494+0000 7f2031ffb700 1 -- 172.21.15.40:0/2876076400 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2044004880 con 0x564b933370d0 2023-07-14T19:16:57.507 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.495+0000 7f205da5bb80 1 -- 172.21.15.40:0/2876076400 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564b933370d0 msgr2=0x564b933374e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:16:57.507 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.495+0000 7f205da5bb80 1 --2- 172.21.15.40:0/2876076400 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564b933370d0 0x564b933374e0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f2044011680 tx=0x7f2044009920 comp rx=0 tx=0).stop 2023-07-14T19:16:57.507 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.495+0000 7f205da5bb80 1 -- 172.21.15.40:0/2876076400 shutdown_connections 2023-07-14T19:16:57.507 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.495+0000 7f205da5bb80 1 --2- 172.21.15.40:0/2876076400 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564b93338650 0x564b93338ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:57.508 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.495+0000 7f205da5bb80 1 --2- 172.21.15.40:0/2876076400 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564b933370d0 0x564b933374e0 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:57.508 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.495+0000 7f205da5bb80 1 --2- 172.21.15.40:0/2876076400 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x564b93337a20 0x564b933380c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:57.508 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.495+0000 7f205da5bb80 1 -- 172.21.15.40:0/2876076400 >> 172.21.15.40:0/2876076400 conn(0x564b9332a630 msgr2=0x564b9332f980 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:16:57.508 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.495+0000 7f20508c1700 1 -- 172.21.15.40:0/2404362909 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7f203c005530 con 0x7f204c0ffd30 2023-07-14T19:16:57.508 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.495+0000 7f205da5bb80 1 -- 172.21.15.40:0/2876076400 shutdown_connections 2023-07-14T19:16:57.508 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.495+0000 7f205da5bb80 1 -- 172.21.15.40:0/2876076400 wait complete. 2023-07-14T19:16:57.508 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.495+0000 7f205da5bb80 1 -- start start 2023-07-14T19:16:57.509 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.496+0000 7f205da5bb80 1 Processor -- start 2023-07-14T19:16:57.509 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.496+0000 7f205da5bb80 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564b933370d0 0x564b93340ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:16:57.509 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.496+0000 7f205da5bb80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564b93337a20 0x564b933411e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:16:57.509 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.496+0000 7f205da5bb80 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x564b93346210 0x564b933485f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:16:57.509 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.496+0000 7f205da5bb80 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x564b93225a70 con 0x564b933370d0 2023-07-14T19:16:57.509 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.496+0000 7f20508c1700 1 --2- 172.21.15.40:0/2404362909 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f20340a10f0 0x7f20340a35a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:16:57.509 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.496+0000 7f205da5bb80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x564b93225770 con 0x564b93337a20 2023-07-14T19:16:57.509 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.496+0000 7f205da5bb80 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x564b931098d0 con 0x564b93346210 2023-07-14T19:16:57.510 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.496+0000 7f205da5bb80 10 client.0 _ll_register_callbacks cb 0x564b93223710 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-14T19:16:57.510 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.496+0000 7f204b7fe700 1 --2- 172.21.15.40:0/2404362909 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f20340a10f0 0x7f20340a35a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:16:57.510 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.496+0000 7f204affd700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564b93337a20 0x564b933411e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:16:57.510 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.496+0000 7f20508c1700 1 -- 172.21.15.40:0/2404362909 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 8269+0+0 (secure 0 0 0) 0x7f203c0d3530 con 0x7f204c0ffd30 2023-07-14T19:16:57.510 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.496+0000 7f204affd700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564b93337a20 0x564b933411e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.40:49036/0 (socket says 172.21.15.40:49036) 2023-07-14T19:16:57.510 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.496+0000 7f204affd700 1 -- 172.21.15.40:0/3318831674 learned_addr learned my addr 172.21.15.40:0/3318831674 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-07-14T19:16:57.511 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.496+0000 7f204a7fc700 1 --2- 172.21.15.40:0/3318831674 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x564b93346210 0x564b933485f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:16:57.511 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.496+0000 7f204b7fe700 1 --2- 172.21.15.40:0/3318831674 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564b933370d0 0x564b93340ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:16:57.511 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.496+0000 7f204affd700 1 -- 172.21.15.40:0/3318831674 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x564b93346210 msgr2=0x564b933485f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T19:16:57.511 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.496+0000 7f204affd700 1 --2- 172.21.15.40:0/3318831674 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x564b93346210 0x564b933485f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:57.511 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.496+0000 7f204affd700 1 -- 172.21.15.40:0/3318831674 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564b933370d0 msgr2=0x564b93340ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T19:16:57.511 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.496+0000 7f204affd700 1 --2- 172.21.15.40:0/3318831674 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564b933370d0 0x564b93340ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:57.511 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.496+0000 7f204affd700 1 -- 172.21.15.40:0/3318831674 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2044011330 con 0x564b93337a20 2023-07-14T19:16:57.512 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.496+0000 7f204b7fe700 1 --2- 172.21.15.40:0/2404362909 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f20340a10f0 0x7f20340a35a0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f2044009e40 tx=0x7f2044009dd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:16:57.512 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.497+0000 7f204affd700 1 --2- 172.21.15.40:0/3318831674 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564b93337a20 0x564b933411e0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f20400116f0 tx=0x7f2040011a00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f587d519fb53b581 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:16:57.512 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.497+0000 7f202affd700 10 client.0 ms_handle_connect on v2:172.21.15.130:3300/0 2023-07-14T19:16:57.512 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.497+0000 7f202affd700 1 -- 172.21.15.40:0/3318831674 <== mon.2 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2040016070 con 0x564b93337a20 2023-07-14T19:16:57.512 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.497+0000 7f205da5bb80 1 -- 172.21.15.40:0/3318831674 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x564b931df4a0 con 0x564b93337a20 2023-07-14T19:16:57.512 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.497+0000 7f202affd700 1 -- 172.21.15.40:0/3318831674 <== mon.2 v2:172.21.15.130:3300/0 2 ==== osd_map(83..83 src has 1..83) v4 ==== 8269+0+0 (secure 0 0 0) 0x7f2040004260 con 0x564b93337a20 2023-07-14T19:16:57.513 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.497+0000 7f202affd700 1 -- 172.21.15.40:0/3318831674 <== mon.2 v2:172.21.15.130:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2040004420 con 0x564b93337a20 2023-07-14T19:16:57.513 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.497+0000 7f205da5bb80 10 client.24902 fetch_fsmap learned FSMap version 26 2023-07-14T19:16:57.513 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.497+0000 7f205da5bb80 1 -- 172.21.15.40:0/3318831674 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) v3 -- 0x564b9309cc10 con 0x564b93337a20 2023-07-14T19:16:57.513 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.497+0000 7f202affd700 1 -- 172.21.15.40:0/3318831674 <== mon.2 v2:172.21.15.130:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f2040029080 con 0x564b93337a20 2023-07-14T19:16:57.513 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.498+0000 7f202affd700 1 -- 172.21.15.40:0/3318831674 <== mon.2 v2:172.21.15.130:3300/0 5 ==== fsmap(e 26) v1 ==== 7462+0+0 (secure 0 0 0) 0x7f2040023030 con 0x564b93337a20 2023-07-14T19:16:57.513 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.498+0000 7f205da5bb80 10 client.24902 fetch_fsmap finished waiting for FSMap version 26 2023-07-14T19:16:57.513 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.498+0000 7f205da5bb80 10 client.24902 resolve_mds: resolved name 'b' to daemon mds.b 2023-07-14T19:16:57.514 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.498+0000 7f205da5bb80 20 client.24902 populate_metadata read hostname 'smithi040' 2023-07-14T19:16:57.514 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.498+0000 7f205da5bb80 1 --2- 172.21.15.40:0/3318831674 >> [v2:172.21.15.130:6838/2145479535,v1:172.21.15.130:6839/2145479535] conn(0x564b9334c380 0x564b9334b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:16:57.514 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.498+0000 7f205da5bb80 4 client.24902 mds_command: new command op to 24539 tid=0[{"prefix": "get_command_descriptions"}] 2023-07-14T19:16:57.514 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.498+0000 7f205da5bb80 1 -- 172.21.15.40:0/3318831674 --> [v2:172.21.15.130:6838/2145479535,v1:172.21.15.130:6839/2145479535] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x564b93303d50 con 0x564b9334c380 2023-07-14T19:16:57.514 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.498+0000 7f204a7fc700 1 --2- 172.21.15.40:0/3318831674 >> [v2:172.21.15.130:6838/2145479535,v1:172.21.15.130:6839/2145479535] conn(0x564b9334c380 0x564b9334b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:16:57.514 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.498+0000 7f202affd700 10 client.24902 ms_handle_connect on v2:172.21.15.130:6838/2145479535 2023-07-14T19:16:57.514 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.498+0000 7f204a7fc700 1 --2- 172.21.15.40:0/3318831674 >> [v2:172.21.15.130:6838/2145479535,v1:172.21.15.130:6839/2145479535] conn(0x564b9334c380 0x564b9334b7b0 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f98a3f10603f7e5f server_cookie=9fe4e468d90e30e8 in_seq=0 out_seq=0 2023-07-14T19:16:57.514 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.499+0000 7f202affd700 1 -- 172.21.15.40:0/3318831674 <== mds.0 v2:172.21.15.130:6838/2145479535 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18017 (crc 0 0 0) 0x564b93303d50 con 0x564b9334c380 2023-07-14T19:16:57.515 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.499+0000 7f202affd700 10 client.24902 handle_command_reply: tid=0 2023-07-14T19:16:57.515 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.499+0000 7f205da5bb80 1 client.24902 shutdown 2023-07-14T19:16:57.515 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.499+0000 7f205da5bb80 1 -- 172.21.15.40:0/3318831674 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564b93337a20 msgr2=0x564b933411e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:16:57.515 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.499+0000 7f205da5bb80 1 --2- 172.21.15.40:0/3318831674 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564b93337a20 0x564b933411e0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f20400116f0 tx=0x7f2040011a00 comp rx=0 tx=0).stop 2023-07-14T19:16:57.515 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.499+0000 7f205da5bb80 1 -- 172.21.15.40:0/3318831674 shutdown_connections 2023-07-14T19:16:57.515 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.499+0000 7f205da5bb80 1 --2- 172.21.15.40:0/3318831674 >> [v2:172.21.15.130:6838/2145479535,v1:172.21.15.130:6839/2145479535] conn(0x564b9334c380 0x564b9334b7b0 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:57.515 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.499+0000 7f205da5bb80 1 --2- 172.21.15.40:0/3318831674 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x564b93346210 0x564b933485f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:57.516 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.499+0000 7f205da5bb80 1 --2- 172.21.15.40:0/3318831674 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564b933370d0 0x564b93340ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:57.516 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.499+0000 7f202affd700 0 client.24902 ms_handle_reset on v2:172.21.15.130:6838/2145479535 2023-07-14T19:16:57.516 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.499+0000 7f205da5bb80 1 --2- 172.21.15.40:0/3318831674 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564b93337a20 0x564b933411e0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:57.516 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.499+0000 7f205da5bb80 1 -- 172.21.15.40:0/3318831674 >> 172.21.15.40:0/3318831674 conn(0x564b9332a630 msgr2=0x564b9332e600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:16:57.516 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.499+0000 7f205da5bb80 1 -- 172.21.15.40:0/3318831674 shutdown_connections 2023-07-14T19:16:57.516 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.499+0000 7f205da5bb80 1 -- 172.21.15.40:0/3318831674 wait complete. 2023-07-14T19:16:57.516 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.500+0000 7f205da5bb80 20 client.24902 trim_cache size 0 max 16384 2023-07-14T19:16:57.526 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.517+0000 7f205da5bb80 1 Processor -- start 2023-07-14T19:16:57.526 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.518+0000 7f205da5bb80 1 -- start start 2023-07-14T19:16:57.527 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.518+0000 7f205da5bb80 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x564b933370d0 0x564b93348eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:16:57.527 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.518+0000 7f205da5bb80 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564b93337a20 0x564b933550f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:16:57.527 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.518+0000 7f205da5bb80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564b93346210 0x564b9334b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:16:57.527 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.518+0000 7f205da5bb80 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x564b93303d50 con 0x564b93337a20 2023-07-14T19:16:57.527 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.518+0000 7f205da5bb80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x564b93330880 con 0x564b93346210 2023-07-14T19:16:57.528 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.518+0000 7f205da5bb80 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x564b93331400 con 0x564b933370d0 2023-07-14T19:16:57.528 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.518+0000 7f204a7fc700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564b93337a20 0x564b933550f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:16:57.528 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.518+0000 7f204a7fc700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564b93337a20 0x564b933550f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:33888/0 (socket says 172.21.15.40:33888) 2023-07-14T19:16:57.528 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.518+0000 7f204a7fc700 1 -- 172.21.15.40:0/2312916167 learned_addr learned my addr 172.21.15.40:0/2312916167 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-07-14T19:16:57.528 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.518+0000 7f204affd700 1 --2- 172.21.15.40:0/2312916167 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x564b933370d0 0x564b93348eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:16:57.529 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.518+0000 7f204b7fe700 1 --2- 172.21.15.40:0/2312916167 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564b93346210 0x564b9334b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:16:57.529 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.518+0000 7f204a7fc700 1 -- 172.21.15.40:0/2312916167 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x564b933370d0 msgr2=0x564b93348eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T19:16:57.529 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.518+0000 7f204a7fc700 1 --2- 172.21.15.40:0/2312916167 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x564b933370d0 0x564b93348eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:57.529 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.518+0000 7f204a7fc700 1 -- 172.21.15.40:0/2312916167 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564b93346210 msgr2=0x564b9334b7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T19:16:57.529 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.518+0000 7f204a7fc700 1 --2- 172.21.15.40:0/2312916167 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564b93346210 0x564b9334b7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:57.529 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.518+0000 7f204a7fc700 1 -- 172.21.15.40:0/2312916167 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x564b930da200 con 0x564b93337a20 2023-07-14T19:16:57.530 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.518+0000 7f204b7fe700 1 --2- 172.21.15.40:0/2312916167 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564b93346210 0x564b9334b7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-14T19:16:57.530 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.518+0000 7f204a7fc700 1 --2- 172.21.15.40:0/2312916167 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564b93337a20 0x564b933550f0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f203c00b2e0 tx=0x7f203c02d000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=84e799c4eea495d1 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:16:57.530 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.518+0000 7f20327fc700 1 -- 172.21.15.40:0/2312916167 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f203c0a07c0 con 0x564b93337a20 2023-07-14T19:16:57.530 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.518+0000 7f20327fc700 1 -- 172.21.15.40:0/2312916167 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f203c0a47f0 con 0x564b93337a20 2023-07-14T19:16:57.530 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.518+0000 7f20327fc700 1 -- 172.21.15.40:0/2312916167 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f203c09fcb0 con 0x564b93337a20 2023-07-14T19:16:57.530 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.519+0000 7f205da5bb80 1 -- 172.21.15.40:0/2312916167 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564b93337a20 msgr2=0x564b933550f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:16:57.530 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.519+0000 7f205da5bb80 1 --2- 172.21.15.40:0/2312916167 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564b93337a20 0x564b933550f0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f203c00b2e0 tx=0x7f203c02d000 comp rx=0 tx=0).stop 2023-07-14T19:16:57.530 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.519+0000 7f205da5bb80 1 -- 172.21.15.40:0/2312916167 shutdown_connections 2023-07-14T19:16:57.531 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.519+0000 7f205da5bb80 1 --2- 172.21.15.40:0/2312916167 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x564b933370d0 0x564b93348eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:57.531 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.519+0000 7f205da5bb80 1 --2- 172.21.15.40:0/2312916167 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564b93346210 0x564b9334b7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:57.531 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.519+0000 7f205da5bb80 1 --2- 172.21.15.40:0/2312916167 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564b93337a20 0x564b933550f0 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:57.531 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.519+0000 7f205da5bb80 1 -- 172.21.15.40:0/2312916167 >> 172.21.15.40:0/2312916167 conn(0x564b9332a630 msgr2=0x564b9332f0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:16:57.531 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.519+0000 7f205da5bb80 1 -- 172.21.15.40:0/2312916167 shutdown_connections 2023-07-14T19:16:57.531 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.519+0000 7f205da5bb80 1 -- 172.21.15.40:0/2312916167 wait complete. 2023-07-14T19:16:57.531 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.519+0000 7f205da5bb80 1 -- start start 2023-07-14T19:16:57.532 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.520+0000 7f205da5bb80 1 Processor -- start 2023-07-14T19:16:57.532 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.520+0000 7f205da5bb80 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564b933370d0 0x564b93350360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:16:57.532 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.520+0000 7f205da5bb80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564b93337a20 0x564b933508a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:16:57.532 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.520+0000 7f205da5bb80 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x564b93346210 0x564b933a0000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:16:57.532 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.520+0000 7f205da5bb80 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x564b93350de0 con 0x564b933370d0 2023-07-14T19:16:57.532 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.520+0000 7f205da5bb80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x564b93350f50 con 0x564b93337a20 2023-07-14T19:16:57.532 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.520+0000 7f205da5bb80 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x564b933a0540 con 0x564b93346210 2023-07-14T19:16:57.533 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.520+0000 7f205da5bb80 10 client.0 _ll_register_callbacks cb 0x7f2040009d80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-14T19:16:57.533 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.520+0000 7f204affd700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564b933370d0 0x564b93350360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:16:57.533 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.520+0000 7f204affd700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564b933370d0 0x564b93350360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:33892/0 (socket says 172.21.15.40:33892) 2023-07-14T19:16:57.533 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.520+0000 7f204affd700 1 -- 172.21.15.40:0/519812260 learned_addr learned my addr 172.21.15.40:0/519812260 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-07-14T19:16:57.533 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.520+0000 7f204a7fc700 1 --2- 172.21.15.40:0/519812260 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564b93337a20 0x564b933508a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:16:57.534 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.520+0000 7f204b7fe700 1 --2- 172.21.15.40:0/519812260 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x564b93346210 0x564b933a0000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:16:57.534 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.520+0000 7f204affd700 1 -- 172.21.15.40:0/519812260 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x564b93346210 msgr2=0x564b933a0000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T19:16:57.534 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.520+0000 7f204affd700 1 --2- 172.21.15.40:0/519812260 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x564b93346210 0x564b933a0000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:57.534 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.520+0000 7f204affd700 1 -- 172.21.15.40:0/519812260 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564b93337a20 msgr2=0x564b933508a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T19:16:57.534 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.520+0000 7f204affd700 1 --2- 172.21.15.40:0/519812260 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564b93337a20 0x564b933508a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:57.534 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.520+0000 7f204affd700 1 -- 172.21.15.40:0/519812260 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f204c19cd00 con 0x564b933370d0 2023-07-14T19:16:57.534 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.520+0000 7f204a7fc700 1 --2- 172.21.15.40:0/519812260 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564b93337a20 0x564b933508a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T19:16:57.535 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.520+0000 7f204affd700 1 --2- 172.21.15.40:0/519812260 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564b933370d0 0x564b93350360 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f20400116c0 tx=0x7f2040005b40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b20cfb271c485219 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:16:57.535 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.520+0000 7f202affd700 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2023-07-14T19:16:57.535 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.521+0000 7f202affd700 1 -- 172.21.15.40:0/519812260 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2040016040 con 0x564b933370d0 2023-07-14T19:16:57.535 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.521+0000 7f205da5bb80 1 -- 172.21.15.40:0/519812260 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f204400f040 con 0x564b933370d0 2023-07-14T19:16:57.535 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.521+0000 7f202affd700 1 -- 172.21.15.40:0/519812260 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(83..83 src has 1..83) v4 ==== 8269+0+0 (secure 0 0 0) 0x7f20400054e0 con 0x564b933370d0 2023-07-14T19:16:57.535 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.521+0000 7f202affd700 1 -- 172.21.15.40:0/519812260 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7f204000fee0 con 0x564b933370d0 2023-07-14T19:16:57.535 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.521+0000 7f205da5bb80 10 client.15084 fetch_fsmap learned FSMap version 26 2023-07-14T19:16:57.535 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.521+0000 7f205da5bb80 1 -- 172.21.15.40:0/519812260 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) v3 -- 0x564b930b2260 con 0x564b933370d0 2023-07-14T19:16:57.536 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.521+0000 7f202affd700 1 -- 172.21.15.40:0/519812260 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f204400f040 con 0x564b933370d0 2023-07-14T19:16:57.536 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.521+0000 7f202affd700 1 -- 172.21.15.40:0/519812260 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 26) v1 ==== 7462+0+0 (secure 0 0 0) 0x7f2040011720 con 0x564b933370d0 2023-07-14T19:16:57.536 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.521+0000 7f205da5bb80 10 client.15084 fetch_fsmap finished waiting for FSMap version 26 2023-07-14T19:16:57.536 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.521+0000 7f205da5bb80 10 client.15084 resolve_mds: resolved name 'b' to daemon mds.b 2023-07-14T19:16:57.536 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.521+0000 7f205da5bb80 20 client.15084 populate_metadata read hostname 'smithi040' 2023-07-14T19:16:57.536 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.521+0000 7f205da5bb80 1 --2- 172.21.15.40:0/519812260 >> [v2:172.21.15.130:6838/2145479535,v1:172.21.15.130:6839/2145479535] conn(0x564b933a5240 0x564b933a1bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:16:57.536 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.521+0000 7f205da5bb80 4 client.15084 mds_command: new command op to 24539 tid=0[{"prefix": "status"}] 2023-07-14T19:16:57.537 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.521+0000 7f205da5bb80 1 -- 172.21.15.40:0/519812260 --> [v2:172.21.15.130:6838/2145479535,v1:172.21.15.130:6839/2145479535] -- command(tid 0: {"prefix": "status"}) v1 -- 0x564b9327ee60 con 0x564b933a5240 2023-07-14T19:16:57.537 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.521+0000 7f204a7fc700 1 --2- 172.21.15.40:0/519812260 >> [v2:172.21.15.130:6838/2145479535,v1:172.21.15.130:6839/2145479535] conn(0x564b933a5240 0x564b933a1bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:16:57.537 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.522+0000 7f202affd700 10 client.15084 ms_handle_connect on v2:172.21.15.130:6838/2145479535 2023-07-14T19:16:57.537 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.522+0000 7f204a7fc700 1 --2- 172.21.15.40:0/519812260 >> [v2:172.21.15.130:6838/2145479535,v1:172.21.15.130:6839/2145479535] conn(0x564b933a5240 0x564b933a1bb0 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=44b99434d47d01c4 server_cookie=c4df226d97215695 in_seq=0 out_seq=0 2023-07-14T19:16:57.537 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.522+0000 7f202affd700 1 -- 172.21.15.40:0/519812260 <== mds.0 v2:172.21.15.130:6838/2145479535 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+332 (crc 0 0 0) 0x564b9327ee60 con 0x564b933a5240 2023-07-14T19:16:57.537 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.522+0000 7f202affd700 10 client.15084 handle_command_reply: tid=0 2023-07-14T19:16:57.538 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.522+0000 7f205da5bb80 1 client.15084 shutdown 2023-07-14T19:16:57.538 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.522+0000 7f205da5bb80 1 -- 172.21.15.40:0/519812260 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564b933370d0 msgr2=0x564b93350360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:16:57.538 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.522+0000 7f205da5bb80 1 --2- 172.21.15.40:0/519812260 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564b933370d0 0x564b93350360 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f20400116c0 tx=0x7f2040005b40 comp rx=0 tx=0).stop 2023-07-14T19:16:57.538 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.522+0000 7f205da5bb80 1 -- 172.21.15.40:0/519812260 shutdown_connections 2023-07-14T19:16:57.538 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.522+0000 7f205da5bb80 1 --2- 172.21.15.40:0/519812260 >> [v2:172.21.15.130:6838/2145479535,v1:172.21.15.130:6839/2145479535] conn(0x564b933a5240 0x564b933a1bb0 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:57.538 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.522+0000 7f205da5bb80 1 --2- 172.21.15.40:0/519812260 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x564b93346210 0x564b933a0000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:57.538 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.522+0000 7f205da5bb80 1 --2- 172.21.15.40:0/519812260 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564b933370d0 0x564b93350360 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:57.539 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.522+0000 7f205da5bb80 1 --2- 172.21.15.40:0/519812260 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564b93337a20 0x564b933508a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:57.539 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.522+0000 7f202affd700 0 client.15084 ms_handle_reset on v2:172.21.15.130:6838/2145479535 2023-07-14T19:16:57.539 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.522+0000 7f205da5bb80 1 -- 172.21.15.40:0/519812260 >> 172.21.15.40:0/519812260 conn(0x564b9332a630 msgr2=0x564b9334b7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:16:57.539 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.522+0000 7f205da5bb80 1 -- 172.21.15.40:0/519812260 shutdown_connections 2023-07-14T19:16:57.539 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.522+0000 7f205da5bb80 1 -- 172.21.15.40:0/519812260 wait complete. 2023-07-14T19:16:57.540 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.522+0000 7f205da5bb80 20 client.15084 trim_cache size 0 max 16384 2023-07-14T19:16:57.540 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2023-07-14T19:16:57.540 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_fsid": "adc52f04-2279-11ee-9b33-001a4aab830c", 2023-07-14T19:16:57.540 INFO:teuthology.orchestra.run.smithi040.stdout: "whoami": 0, 2023-07-14T19:16:57.541 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 24539, 2023-07-14T19:16:57.541 INFO:teuthology.orchestra.run.smithi040.stdout: "want_state": "up:active", 2023-07-14T19:16:57.541 INFO:teuthology.orchestra.run.smithi040.stdout: "state": "up:active", 2023-07-14T19:16:57.541 INFO:teuthology.orchestra.run.smithi040.stdout: "fs_name": "cephfs", 2023-07-14T19:16:57.541 INFO:teuthology.orchestra.run.smithi040.stdout: "rank_uptime": 58.961001791000001, 2023-07-14T19:16:57.541 INFO:teuthology.orchestra.run.smithi040.stdout: "mdsmap_epoch": 26, 2023-07-14T19:16:57.541 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap_epoch": 83, 2023-07-14T19:16:57.542 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap_epoch_barrier": 81, 2023-07-14T19:16:57.542 INFO:teuthology.orchestra.run.smithi040.stdout: "uptime": 64.500830039999997 2023-07-14T19:16:57.542 INFO:teuthology.orchestra.run.smithi040.stdout:} 2023-07-14T19:16:57.543 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.523+0000 7f20518c3700 1 -- 172.21.15.40:0/2404362909 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f20340a10f0 msgr2=0x7f20340a35a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:16:57.543 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.523+0000 7f20518c3700 1 --2- 172.21.15.40:0/2404362909 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f20340a10f0 0x7f20340a35a0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f2044009e40 tx=0x7f2044009dd0 comp rx=0 tx=0).stop 2023-07-14T19:16:57.543 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.523+0000 7f20518c3700 1 -- 172.21.15.40:0/2404362909 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f204c0ffd30 msgr2=0x7f204c194db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:16:57.543 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.523+0000 7f20518c3700 1 --2- 172.21.15.40:0/2404362909 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f204c0ffd30 0x7f204c194db0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f203c004760 tx=0x7f203c004790 comp rx=0 tx=0).stop 2023-07-14T19:16:57.543 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.523+0000 7f20518c3700 1 -- 172.21.15.40:0/2404362909 shutdown_connections 2023-07-14T19:16:57.544 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.523+0000 7f20518c3700 1 --2- 172.21.15.40:0/2404362909 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f20340a10f0 0x7f20340a35a0 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:57.544 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.523+0000 7f20518c3700 1 --2- 172.21.15.40:0/2404362909 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f204c1006c0 0x7f204c19be20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:57.544 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.523+0000 7f20518c3700 1 --2- 172.21.15.40:0/2404362909 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f204c0feb40 0x7f204c194870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:57.544 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.523+0000 7f20518c3700 1 --2- 172.21.15.40:0/2404362909 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f204c0ffd30 0x7f204c194db0 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:57.544 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.523+0000 7f20518c3700 1 -- 172.21.15.40:0/2404362909 >> 172.21.15.40:0/2404362909 conn(0x7f204c0fa160 msgr2=0x7f204c0fc330 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:16:57.544 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.523+0000 7f20518c3700 1 -- 172.21.15.40:0/2404362909 shutdown_connections 2023-07-14T19:16:57.544 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:57.523+0000 7f20518c3700 1 -- 172.21.15.40:0/2404362909 wait complete. 2023-07-14T19:16:57.545 INFO:tasks.cephadm:Setup complete, yielding 2023-07-14T19:16:57.545 INFO:teuthology.run_tasks:Running task cephadm.shell... 2023-07-14T19:16:57.558 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi040.front.sepia.ceph.com 2023-07-14T19:16:57.558 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid adc52f04-2279-11ee-9b33-001a4aab830c -- bash -c 'ceph orch status' 2023-07-14T19:16:58.117 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/adc52f04-2279-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T19:16:58.971 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:16:58 smithi040 ceph-mon[96518]: pgmap v286: 97 pgs: 6 peering, 91 active+clean; 638 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 3 op/s; 83 B/s, 0 objects/s recovering 2023-07-14T19:16:59.222 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:16:58 smithi191 ceph-mon[98311]: pgmap v286: 97 pgs: 6 peering, 91 active+clean; 638 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 3 op/s; 83 B/s, 0 objects/s recovering 2023-07-14T19:16:59.242 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:16:58 smithi130 ceph-mon[97832]: pgmap v286: 97 pgs: 6 peering, 91 active+clean; 638 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 3 op/s; 83 B/s, 0 objects/s recovering 2023-07-14T19:16:59.363 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:59.360+0000 7f9445ec5700 1 -- 172.21.15.40:0/3519945323 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9440105a00 msgr2=0x7f9440107de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:16:59.363 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:59.360+0000 7f9445ec5700 1 --2- 172.21.15.40:0/3519945323 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9440105a00 0x7f9440107de0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f943800b0a0 tx=0x7f943800b3b0 comp rx=0 tx=0).stop 2023-07-14T19:16:59.363 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:59.360+0000 7f9445ec5700 1 -- 172.21.15.40:0/3519945323 shutdown_connections 2023-07-14T19:16:59.363 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:59.360+0000 7f9445ec5700 1 --2- 172.21.15.40:0/3519945323 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f94401007c0 0x7f9440102ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:59.363 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:59.360+0000 7f9445ec5700 1 --2- 172.21.15.40:0/3519945323 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9440105a00 0x7f9440107de0 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:59.364 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:59.360+0000 7f9445ec5700 1 --2- 172.21.15.40:0/3519945323 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f94401030e0 0x7f94401054c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:59.364 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:59.360+0000 7f9445ec5700 1 -- 172.21.15.40:0/3519945323 >> 172.21.15.40:0/3519945323 conn(0x7f94400fa1e0 msgr2=0x7f94400fc630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:16:59.364 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:59.360+0000 7f9445ec5700 1 -- 172.21.15.40:0/3519945323 shutdown_connections 2023-07-14T19:16:59.365 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:59.360+0000 7f9445ec5700 1 -- 172.21.15.40:0/3519945323 wait complete. 2023-07-14T19:16:59.365 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:59.361+0000 7f9445ec5700 1 Processor -- start 2023-07-14T19:16:59.365 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:59.361+0000 7f9445ec5700 1 -- start start 2023-07-14T19:16:59.365 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:59.361+0000 7f9445ec5700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f94401007c0 0x7f9440195b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:16:59.365 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:59.361+0000 7f9445ec5700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f94401030e0 0x7f9440196080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:16:59.365 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:59.361+0000 7f9445ec5700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9440105a00 0x7f944019d100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:16:59.365 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:59.361+0000 7f9445ec5700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f944019d6d0 con 0x7f9440105a00 2023-07-14T19:16:59.365 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:59.361+0000 7f9445ec5700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f944019d810 con 0x7f94401007c0 2023-07-14T19:16:59.366 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:59.361+0000 7f9445ec5700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f944019d9a0 con 0x7f94401030e0 2023-07-14T19:16:59.366 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:59.361+0000 7f943ffff700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9440105a00 0x7f944019d100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:16:59.366 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:59.361+0000 7f943ffff700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9440105a00 0x7f944019d100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:33908/0 (socket says 172.21.15.40:33908) 2023-07-14T19:16:59.366 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:59.361+0000 7f943ffff700 1 -- 172.21.15.40:0/2825746584 learned_addr learned my addr 172.21.15.40:0/2825746584 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-07-14T19:16:59.366 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:59.361+0000 7f943f7fe700 1 --2- 172.21.15.40:0/2825746584 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f94401007c0 0x7f9440195b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:16:59.366 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:59.361+0000 7f943effd700 1 --2- 172.21.15.40:0/2825746584 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f94401030e0 0x7f9440196080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:16:59.366 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:59.361+0000 7f943ffff700 1 -- 172.21.15.40:0/2825746584 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f94401030e0 msgr2=0x7f9440196080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:16:59.367 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:59.361+0000 7f943ffff700 1 --2- 172.21.15.40:0/2825746584 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f94401030e0 0x7f9440196080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:59.367 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:59.361+0000 7f943ffff700 1 -- 172.21.15.40:0/2825746584 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f94401007c0 msgr2=0x7f9440195b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:16:59.367 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:59.361+0000 7f943ffff700 1 --2- 172.21.15.40:0/2825746584 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f94401007c0 0x7f9440195b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:59.367 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:59.362+0000 7f943ffff700 1 -- 172.21.15.40:0/2825746584 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9438009d00 con 0x7f9440105a00 2023-07-14T19:16:59.367 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:59.362+0000 7f943effd700 1 --2- 172.21.15.40:0/2825746584 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f94401030e0 0x7f9440196080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T19:16:59.367 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:59.362+0000 7f943ffff700 1 --2- 172.21.15.40:0/2825746584 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9440105a00 0x7f944019d100 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f9438006070 tx=0x7f9438004d00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:16:59.367 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:59.362+0000 7f943cff9700 1 -- 172.21.15.40:0/2825746584 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f943800e040 con 0x7f9440105a00 2023-07-14T19:16:59.368 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:59.362+0000 7f9445ec5700 1 -- 172.21.15.40:0/2825746584 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f944019dc20 con 0x7f9440105a00 2023-07-14T19:16:59.368 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:59.362+0000 7f943cff9700 1 -- 172.21.15.40:0/2825746584 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f943800cea0 con 0x7f9440105a00 2023-07-14T19:16:59.368 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:59.362+0000 7f943cff9700 1 -- 172.21.15.40:0/2825746584 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9438004020 con 0x7f9440105a00 2023-07-14T19:16:59.368 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:59.362+0000 7f9445ec5700 1 -- 172.21.15.40:0/2825746584 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f944019e040 con 0x7f9440105a00 2023-07-14T19:16:59.369 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:59.363+0000 7f9445ec5700 1 -- 172.21.15.40:0/2825746584 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f94400fbd50 con 0x7f9440105a00 2023-07-14T19:16:59.369 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:59.364+0000 7f943cff9700 1 -- 172.21.15.40:0/2825746584 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7f943801d690 con 0x7f9440105a00 2023-07-14T19:16:59.369 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:59.364+0000 7f943cff9700 1 --2- 172.21.15.40:0/2825746584 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f94280a11e0 0x7f94280a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:16:59.369 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:59.365+0000 7f943f7fe700 1 --2- 172.21.15.40:0/2825746584 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f94280a11e0 0x7f94280a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:16:59.369 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:59.365+0000 7f943cff9700 1 -- 172.21.15.40:0/2825746584 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 8269+0+0 (secure 0 0 0) 0x7f94380ceda0 con 0x7f9440105a00 2023-07-14T19:16:59.370 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:59.365+0000 7f943f7fe700 1 --2- 172.21.15.40:0/2825746584 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f94280a11e0 0x7f94280a3690 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f9430009a10 tx=0x7f9430009190 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:16:59.370 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:59.368+0000 7f943cff9700 1 -- 172.21.15.40:0/2825746584 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f9438098400 con 0x7f9440105a00 2023-07-14T19:16:59.581 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:59.581+0000 7f9445ec5700 1 -- 172.21.15.40:0/2825746584 --> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] -- mgr_command(tid 0: {"prefix": "orch status", "target": ["mon-mgr", ""]}) v1 -- 0x7f9440060a80 con 0x7f94280a11e0 2023-07-14T19:16:59.582 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:59.582+0000 7f943cff9700 1 -- 172.21.15.40:0/2825746584 <== mgr.14152 v2:172.21.15.40:6800/1693027118 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+43 (secure 0 0 0) 0x7f9440060a80 con 0x7f94280a11e0 2023-07-14T19:16:59.582 INFO:teuthology.orchestra.run.smithi040.stdout:Backend: cephadm 2023-07-14T19:16:59.583 INFO:teuthology.orchestra.run.smithi040.stdout:Available: Yes 2023-07-14T19:16:59.583 INFO:teuthology.orchestra.run.smithi040.stdout:Paused: No 2023-07-14T19:16:59.585 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:59.584+0000 7f9445ec5700 1 -- 172.21.15.40:0/2825746584 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f94280a11e0 msgr2=0x7f94280a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:16:59.585 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:59.584+0000 7f9445ec5700 1 --2- 172.21.15.40:0/2825746584 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f94280a11e0 0x7f94280a3690 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f9430009a10 tx=0x7f9430009190 comp rx=0 tx=0).stop 2023-07-14T19:16:59.585 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:59.584+0000 7f9445ec5700 1 -- 172.21.15.40:0/2825746584 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9440105a00 msgr2=0x7f944019d100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:16:59.586 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:59.584+0000 7f9445ec5700 1 --2- 172.21.15.40:0/2825746584 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9440105a00 0x7f944019d100 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f9438006070 tx=0x7f9438004d00 comp rx=0 tx=0).stop 2023-07-14T19:16:59.586 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:59.584+0000 7f9445ec5700 1 -- 172.21.15.40:0/2825746584 shutdown_connections 2023-07-14T19:16:59.586 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:59.584+0000 7f9445ec5700 1 --2- 172.21.15.40:0/2825746584 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f94280a11e0 0x7f94280a3690 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:59.586 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:59.584+0000 7f9445ec5700 1 --2- 172.21.15.40:0/2825746584 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9440105a00 0x7f944019d100 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:59.586 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:59.584+0000 7f9445ec5700 1 --2- 172.21.15.40:0/2825746584 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f94401007c0 0x7f9440195b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:59.586 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:59.584+0000 7f9445ec5700 1 --2- 172.21.15.40:0/2825746584 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f94401030e0 0x7f9440196080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:59.586 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:59.584+0000 7f9445ec5700 1 -- 172.21.15.40:0/2825746584 >> 172.21.15.40:0/2825746584 conn(0x7f94400fa1e0 msgr2=0x7f9440101420 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:16:59.586 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:59.584+0000 7f9445ec5700 1 -- 172.21.15.40:0/2825746584 shutdown_connections 2023-07-14T19:16:59.587 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:16:59.584+0000 7f9445ec5700 1 -- 172.21.15.40:0/2825746584 wait complete. 2023-07-14T19:16:59.979 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid adc52f04-2279-11ee-9b33-001a4aab830c -- bash -c 'ceph orch ps' 2023-07-14T19:17:00.546 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/adc52f04-2279-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T19:17:01.146 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:17:00 smithi040 ceph-mon[96518]: pgmap v287: 97 pgs: 97 active+clean; 638 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 1 op/s; 224 B/s, 1 objects/s recovering 2023-07-14T19:17:01.146 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:17:00 smithi040 ceph-mon[96518]: from='client.15090 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T19:17:01.146 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:17:00 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T19:17:01.222 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:17:00 smithi191 ceph-mon[98311]: pgmap v287: 97 pgs: 97 active+clean; 638 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 1 op/s; 224 B/s, 1 objects/s recovering 2023-07-14T19:17:01.222 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:17:00 smithi191 ceph-mon[98311]: from='client.15090 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T19:17:01.222 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:17:00 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T19:17:01.242 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:17:00 smithi130 ceph-mon[97832]: pgmap v287: 97 pgs: 97 active+clean; 638 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 1 op/s; 224 B/s, 1 objects/s recovering 2023-07-14T19:17:01.242 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:17:00 smithi130 ceph-mon[97832]: from='client.15090 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T19:17:01.242 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:17:00 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T19:17:01.890 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:01.889+0000 7fa9b02eb700 1 -- 172.21.15.40:0/48820816 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa9a81030c0 msgr2=0x7fa9a81054a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:17:01.890 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:01.889+0000 7fa9b02eb700 1 --2- 172.21.15.40:0/48820816 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa9a81030c0 0x7fa9a81054a0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7fa998009990 tx=0x7fa998009ca0 comp rx=0 tx=0).stop 2023-07-14T19:17:01.891 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:01.889+0000 7fa9b02eb700 1 -- 172.21.15.40:0/48820816 shutdown_connections 2023-07-14T19:17:01.891 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:01.889+0000 7fa9b02eb700 1 --2- 172.21.15.40:0/48820816 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa9a81059e0 0x7fa9a8107de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:01.891 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:01.889+0000 7fa9b02eb700 1 --2- 172.21.15.40:0/48820816 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa9a81007a0 0x7fa9a8102b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:01.891 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:01.889+0000 7fa9b02eb700 1 --2- 172.21.15.40:0/48820816 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa9a81030c0 0x7fa9a81054a0 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:01.891 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:01.889+0000 7fa9b02eb700 1 -- 172.21.15.40:0/48820816 >> 172.21.15.40:0/48820816 conn(0x7fa9a80fa1e0 msgr2=0x7fa9a80fc630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:17:01.892 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:01.889+0000 7fa9b02eb700 1 -- 172.21.15.40:0/48820816 shutdown_connections 2023-07-14T19:17:01.892 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:01.889+0000 7fa9b02eb700 1 -- 172.21.15.40:0/48820816 wait complete. 2023-07-14T19:17:01.893 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:01.890+0000 7fa9b02eb700 1 Processor -- start 2023-07-14T19:17:01.894 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:01.890+0000 7fa9b02eb700 1 -- start start 2023-07-14T19:17:01.894 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:01.890+0000 7fa9b02eb700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa9a81007a0 0x7fa9a8100100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:17:01.894 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:01.890+0000 7fa9b02eb700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa9a81030c0 0x7fa9a80fe750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:17:01.894 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:01.890+0000 7fa9b02eb700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa9a81059e0 0x7fa9a80fec90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:17:01.894 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:01.890+0000 7fa9b02eb700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9a80ff410 con 0x7fa9a81059e0 2023-07-14T19:17:01.894 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:01.890+0000 7fa9b02eb700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9a80ff580 con 0x7fa9a81007a0 2023-07-14T19:17:01.895 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:01.890+0000 7fa9b02eb700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9a819fa80 con 0x7fa9a81030c0 2023-07-14T19:17:01.895 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:01.890+0000 7fa9ae888700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa9a81059e0 0x7fa9a80fec90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:17:01.895 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:01.890+0000 7fa9ae888700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa9a81059e0 0x7fa9a80fec90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:33920/0 (socket says 172.21.15.40:33920) 2023-07-14T19:17:01.895 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:01.890+0000 7fa9ae087700 1 --2- 172.21.15.40:0/1819887625 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa9a81007a0 0x7fa9a8100100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:17:01.895 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:01.890+0000 7fa9ae888700 1 -- 172.21.15.40:0/1819887625 learned_addr learned my addr 172.21.15.40:0/1819887625 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-07-14T19:17:01.895 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:01.890+0000 7fa9ad886700 1 --2- 172.21.15.40:0/1819887625 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa9a81030c0 0x7fa9a80fe750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:17:01.896 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:01.891+0000 7fa9ae888700 1 -- 172.21.15.40:0/1819887625 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa9a81030c0 msgr2=0x7fa9a80fe750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:17:01.896 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:01.891+0000 7fa9ae888700 1 --2- 172.21.15.40:0/1819887625 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa9a81030c0 0x7fa9a80fe750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:01.896 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:01.891+0000 7fa9ae888700 1 -- 172.21.15.40:0/1819887625 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa9a81007a0 msgr2=0x7fa9a8100100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:17:01.896 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:01.891+0000 7fa9ae888700 1 --2- 172.21.15.40:0/1819887625 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa9a81007a0 0x7fa9a8100100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:01.896 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:01.891+0000 7fa9ae888700 1 -- 172.21.15.40:0/1819887625 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9980095f0 con 0x7fa9a81059e0 2023-07-14T19:17:01.897 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:01.891+0000 7fa9ae888700 1 --2- 172.21.15.40:0/1819887625 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa9a81059e0 0x7fa9a80fec90 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7fa9a0007ec0 tx=0x7fa9a000d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:17:01.897 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:01.891+0000 7fa99f7fe700 1 -- 172.21.15.40:0/1819887625 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9a0017070 con 0x7fa9a81059e0 2023-07-14T19:17:01.897 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:01.891+0000 7fa99f7fe700 1 -- 172.21.15.40:0/1819887625 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa9a000f040 con 0x7fa9a81059e0 2023-07-14T19:17:01.897 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:01.891+0000 7fa99f7fe700 1 -- 172.21.15.40:0/1819887625 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9a0004020 con 0x7fa9a81059e0 2023-07-14T19:17:01.897 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:01.891+0000 7fa9b02eb700 1 -- 172.21.15.40:0/1819887625 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa9a819fd60 con 0x7fa9a81059e0 2023-07-14T19:17:01.897 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:01.891+0000 7fa9b02eb700 1 -- 172.21.15.40:0/1819887625 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa9a81a01b0 con 0x7fa9a81059e0 2023-07-14T19:17:01.898 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:01.892+0000 7fa9b02eb700 1 -- 172.21.15.40:0/1819887625 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa9a80fbd50 con 0x7fa9a81059e0 2023-07-14T19:17:01.898 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:01.893+0000 7fa99f7fe700 1 -- 172.21.15.40:0/1819887625 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7fa9a000db20 con 0x7fa9a81059e0 2023-07-14T19:17:01.899 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:01.894+0000 7fa99f7fe700 1 --2- 172.21.15.40:0/1819887625 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7fa9940a1330 0x7fa9940a37e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:17:01.899 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:01.894+0000 7fa9ae087700 1 --2- 172.21.15.40:0/1819887625 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7fa9940a1330 0x7fa9940a37e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:17:01.899 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:01.894+0000 7fa99f7fe700 1 -- 172.21.15.40:0/1819887625 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 8269+0+0 (secure 0 0 0) 0x7fa9a00cc490 con 0x7fa9a81059e0 2023-07-14T19:17:01.899 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:01.894+0000 7fa9ae087700 1 --2- 172.21.15.40:0/1819887625 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7fa9940a1330 0x7fa9940a37e0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fa9a4009c60 tx=0x7fa9a4009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:17:01.899 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:01.897+0000 7fa99f7fe700 1 -- 172.21.15.40:0/1819887625 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fa9a0095b70 con 0x7fa9a81059e0 2023-07-14T19:17:02.108 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:02.107+0000 7fa9b02eb700 1 -- 172.21.15.40:0/1819887625 --> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fa9a8060a80 con 0x7fa9940a1330 2023-07-14T19:17:02.129 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:02.128+0000 7fa99f7fe700 1 -- 172.21.15.40:0/1819887625 <== mgr.14152 v2:172.21.15.40:6800/1693027118 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+4185 (secure 0 0 0) 0x7fa9a8060a80 con 0x7fa9940a1330 2023-07-14T19:17:02.129 INFO:teuthology.orchestra.run.smithi040.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2023-07-14T19:17:02.130 INFO:teuthology.orchestra.run.smithi040.stdout:mds.a smithi040 running (71s) - 71s 14.4M - 18.0.0-4940-g81c9387d 219b180907a8 ff53cd0b9a41 2023-07-14T19:17:02.130 INFO:teuthology.orchestra.run.smithi040.stdout:mds.b smithi130 running (69s) - 68s 14.4M - 18.0.0-4940-g81c9387d 219b180907a8 7b43eb14e326 2023-07-14T19:17:02.130 INFO:teuthology.orchestra.run.smithi040.stdout:mds.c smithi191 running (81s) - 80s 14.4M - 18.0.0-4940-g81c9387d 219b180907a8 f77b1956565c 2023-07-14T19:17:02.130 INFO:teuthology.orchestra.run.smithi040.stdout:mds.d smithi040 running (87s) - 87s 14.4M - 18.0.0-4940-g81c9387d 219b180907a8 dfe3597584c7 2023-07-14T19:17:02.130 INFO:teuthology.orchestra.run.smithi040.stdout:mds.e smithi130 running (85s) - 85s 14.3M - 18.0.0-4940-g81c9387d 219b180907a8 696262f80d98 2023-07-14T19:17:02.130 INFO:teuthology.orchestra.run.smithi040.stdout:mds.f smithi191 running (83s) - 82s 15.7M - 18.0.0-4940-g81c9387d 219b180907a8 496a0d1d6338 2023-07-14T19:17:02.130 INFO:teuthology.orchestra.run.smithi040.stdout:mds.g smithi040 running (74s) - 73s 14.1M - 18.0.0-4940-g81c9387d 219b180907a8 8dba83225dcd 2023-07-14T19:17:02.131 INFO:teuthology.orchestra.run.smithi040.stdout:mds.h smithi130 running (78s) - 78s 16.6M - 18.0.0-4940-g81c9387d 219b180907a8 798bacc37193 2023-07-14T19:17:02.131 INFO:teuthology.orchestra.run.smithi040.stdout:mds.i smithi191 running (76s) - 76s 16.5M - 18.0.0-4940-g81c9387d 219b180907a8 3bccc6cb169a 2023-07-14T19:17:02.131 INFO:teuthology.orchestra.run.smithi040.stdout:mds.j smithi040 running (90s) - 89s 16.7M - 18.0.0-4940-g81c9387d 219b180907a8 0561434c66d8 2023-07-14T19:17:02.131 INFO:teuthology.orchestra.run.smithi040.stdout:mds.k smithi130 running (92s) - 91s 15.8M - 18.0.0-4940-g81c9387d 219b180907a8 fb7eb8e6ffbb 2023-07-14T19:17:02.131 INFO:teuthology.orchestra.run.smithi040.stdout:mds.l smithi191 running (94s) - 94s 14.4M - 18.0.0-4940-g81c9387d 219b180907a8 aba4b62e9f98 2023-07-14T19:17:02.131 INFO:teuthology.orchestra.run.smithi040.stdout:mgr.x smithi040 *:9283,8765 running (7m) - 7m 511M - 18.0.0-4940-g81c9387d 219b180907a8 51022a857c7c 2023-07-14T19:17:02.131 INFO:teuthology.orchestra.run.smithi040.stdout:mgr.y smithi130 *:8443,8765 running (6m) - 6m 447M - 18.0.0-4940-g81c9387d 219b180907a8 52d2499424a3 2023-07-14T19:17:02.131 INFO:teuthology.orchestra.run.smithi040.stdout:mgr.z smithi191 *:8443,8765 running (6m) - 6m 448M - 18.0.0-4940-g81c9387d 219b180907a8 d34005d85306 2023-07-14T19:17:02.132 INFO:teuthology.orchestra.run.smithi040.stdout:mon.a smithi040 running (8m) - 8m 88.8M 2048M 18.0.0-4940-g81c9387d 219b180907a8 84c8a307e84f 2023-07-14T19:17:02.132 INFO:teuthology.orchestra.run.smithi040.stdout:mon.b smithi130 running (6m) - 6m 59.1M 2048M 18.0.0-4940-g81c9387d 219b180907a8 5f00f7e99a36 2023-07-14T19:17:02.132 INFO:teuthology.orchestra.run.smithi040.stdout:mon.c smithi191 running (6m) - 6m 60.1M 2048M 18.0.0-4940-g81c9387d 219b180907a8 a35dab69edf6 2023-07-14T19:17:02.132 INFO:teuthology.orchestra.run.smithi040.stdout:osd.0 smithi040 running (5m) - 5m 59.8M 4293M 18.0.0-4940-g81c9387d 219b180907a8 773d715dee5c 2023-07-14T19:17:02.132 INFO:teuthology.orchestra.run.smithi040.stdout:osd.1 smithi130 running (5m) - 5m 64.3M 4292M 18.0.0-4940-g81c9387d 219b180907a8 fe0367c972a7 2023-07-14T19:17:02.132 INFO:teuthology.orchestra.run.smithi040.stdout:osd.2 smithi191 running (5m) - 5m 60.3M 4292M 18.0.0-4940-g81c9387d 219b180907a8 7eba60ddb4d5 2023-07-14T19:17:02.132 INFO:teuthology.orchestra.run.smithi040.stdout:osd.3 smithi040 running (4m) - 4m 57.2M 4293M 18.0.0-4940-g81c9387d 219b180907a8 bb203c31eb8a 2023-07-14T19:17:02.133 INFO:teuthology.orchestra.run.smithi040.stdout:osd.4 smithi130 running (4m) - 4m 52.1M 4292M 18.0.0-4940-g81c9387d 219b180907a8 445d9a984582 2023-07-14T19:17:02.133 INFO:teuthology.orchestra.run.smithi040.stdout:osd.5 smithi191 running (4m) - 4m 55.6M 4292M 18.0.0-4940-g81c9387d 219b180907a8 56d5657564c4 2023-07-14T19:17:02.133 INFO:teuthology.orchestra.run.smithi040.stdout:osd.6 smithi040 running (3m) - 3m 49.5M 4293M 18.0.0-4940-g81c9387d 219b180907a8 5f0439e3fd0b 2023-07-14T19:17:02.133 INFO:teuthology.orchestra.run.smithi040.stdout:osd.7 smithi130 running (3m) - 3m 55.2M 4292M 18.0.0-4940-g81c9387d 219b180907a8 97cd0d8f365f 2023-07-14T19:17:02.133 INFO:teuthology.orchestra.run.smithi040.stdout:osd.8 smithi191 running (2m) - 2m 52.2M 4292M 18.0.0-4940-g81c9387d 219b180907a8 e6f1db121cb0 2023-07-14T19:17:02.133 INFO:teuthology.orchestra.run.smithi040.stdout:osd.9 smithi040 running (2m) - 2m 49.8M 4293M 18.0.0-4940-g81c9387d 219b180907a8 9e0c3ccb6588 2023-07-14T19:17:02.133 INFO:teuthology.orchestra.run.smithi040.stdout:osd.10 smithi130 running (2m) - 2m 52.6M 4292M 18.0.0-4940-g81c9387d 219b180907a8 99ebcc9e0bb0 2023-07-14T19:17:02.133 INFO:teuthology.orchestra.run.smithi040.stdout:osd.11 smithi191 running (112s) - 112s 46.9M 4292M 18.0.0-4940-g81c9387d 219b180907a8 eeb75e1b8733 2023-07-14T19:17:02.134 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:02.131+0000 7fa9b02eb700 1 -- 172.21.15.40:0/1819887625 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7fa9940a1330 msgr2=0x7fa9940a37e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:17:02.134 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:02.131+0000 7fa9b02eb700 1 --2- 172.21.15.40:0/1819887625 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7fa9940a1330 0x7fa9940a37e0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fa9a4009c60 tx=0x7fa9a4009260 comp rx=0 tx=0).stop 2023-07-14T19:17:02.134 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:02.131+0000 7fa9b02eb700 1 -- 172.21.15.40:0/1819887625 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa9a81059e0 msgr2=0x7fa9a80fec90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:17:02.135 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:02.131+0000 7fa9b02eb700 1 --2- 172.21.15.40:0/1819887625 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa9a81059e0 0x7fa9a80fec90 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7fa9a0007ec0 tx=0x7fa9a000d3b0 comp rx=0 tx=0).stop 2023-07-14T19:17:02.135 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:02.131+0000 7fa9b02eb700 1 -- 172.21.15.40:0/1819887625 shutdown_connections 2023-07-14T19:17:02.135 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:02.131+0000 7fa9b02eb700 1 --2- 172.21.15.40:0/1819887625 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7fa9940a1330 0x7fa9940a37e0 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:02.135 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:02.131+0000 7fa9b02eb700 1 --2- 172.21.15.40:0/1819887625 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa9a81059e0 0x7fa9a80fec90 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:02.135 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:02.131+0000 7fa9b02eb700 1 --2- 172.21.15.40:0/1819887625 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa9a81007a0 0x7fa9a8100100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:02.135 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:02.131+0000 7fa9b02eb700 1 --2- 172.21.15.40:0/1819887625 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa9a81030c0 0x7fa9a80fe750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:02.135 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:02.131+0000 7fa9b02eb700 1 -- 172.21.15.40:0/1819887625 >> 172.21.15.40:0/1819887625 conn(0x7fa9a80fa1e0 msgr2=0x7fa9a8101420 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:17:02.136 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:02.131+0000 7fa9b02eb700 1 -- 172.21.15.40:0/1819887625 shutdown_connections 2023-07-14T19:17:02.136 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:02.131+0000 7fa9b02eb700 1 -- 172.21.15.40:0/1819887625 wait complete. 2023-07-14T19:17:02.711 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid adc52f04-2279-11ee-9b33-001a4aab830c -- bash -c 'ceph orch ls' 2023-07-14T19:17:02.971 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:17:02 smithi040 ceph-mon[96518]: pgmap v288: 97 pgs: 97 active+clean; 638 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 3 op/s; 197 B/s, 1 objects/s recovering 2023-07-14T19:17:03.177 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/adc52f04-2279-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T19:17:03.222 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:17:02 smithi191 ceph-mon[98311]: pgmap v288: 97 pgs: 97 active+clean; 638 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 3 op/s; 197 B/s, 1 objects/s recovering 2023-07-14T19:17:03.242 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:17:02 smithi130 ceph-mon[97832]: pgmap v288: 97 pgs: 97 active+clean; 638 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 3 op/s; 197 B/s, 1 objects/s recovering 2023-07-14T19:17:04.221 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:17:03 smithi040 ceph-mon[96518]: from='client.15096 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T19:17:04.221 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:17:03 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:17:04.221 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:17:03 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:17:04.221 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:17:03 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:17:04.221 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:17:03 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:17:04.222 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:17:03 smithi191 ceph-mon[98311]: from='client.15096 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T19:17:04.222 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:17:03 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:17:04.222 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:17:03 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:17:04.222 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:17:03 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:17:04.223 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:17:03 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:17:04.242 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:17:03 smithi130 ceph-mon[97832]: from='client.15096 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T19:17:04.242 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:17:03 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:17:04.243 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:17:03 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:17:04.243 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:17:03 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:17:04.243 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:17:03 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:17:04.520 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:04.518+0000 7f5f4ed5f700 1 -- 172.21.15.40:0/2716932636 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5f480fef00 msgr2=0x7f5f480ff370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:17:04.521 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:04.518+0000 7f5f4ed5f700 1 --2- 172.21.15.40:0/2716932636 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5f480fef00 0x7f5f480ff370 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f5f34009990 tx=0x7f5f34009ca0 comp rx=0 tx=0).stop 2023-07-14T19:17:04.521 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:04.518+0000 7f5f4ed5f700 1 -- 172.21.15.40:0/2716932636 shutdown_connections 2023-07-14T19:17:04.521 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:04.518+0000 7f5f4ed5f700 1 --2- 172.21.15.40:0/2716932636 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5f480ff8b0 0x7f5f480ffd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:04.521 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:04.518+0000 7f5f4ed5f700 1 --2- 172.21.15.40:0/2716932636 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5f480fe5b0 0x7f5f480fe9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:04.521 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:04.518+0000 7f5f4ed5f700 1 --2- 172.21.15.40:0/2716932636 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5f480fef00 0x7f5f480ff370 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:04.521 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:04.518+0000 7f5f4ed5f700 1 -- 172.21.15.40:0/2716932636 >> 172.21.15.40:0/2716932636 conn(0x7f5f480fa1e0 msgr2=0x7f5f480fc630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:17:04.522 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:04.518+0000 7f5f4ed5f700 1 -- 172.21.15.40:0/2716932636 shutdown_connections 2023-07-14T19:17:04.522 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:04.518+0000 7f5f4ed5f700 1 -- 172.21.15.40:0/2716932636 wait complete. 2023-07-14T19:17:04.522 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:04.519+0000 7f5f4ed5f700 1 Processor -- start 2023-07-14T19:17:04.522 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:04.519+0000 7f5f4ed5f700 1 -- start start 2023-07-14T19:17:04.522 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:04.519+0000 7f5f4ed5f700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5f480fe5b0 0x7f5f48197b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:17:04.522 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:04.519+0000 7f5f4ed5f700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5f480fef00 0x7f5f481980a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:17:04.522 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:04.519+0000 7f5f4ed5f700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5f480ff8b0 0x7f5f4819f120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:17:04.523 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:04.519+0000 7f5f4ed5f700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f4819f6f0 con 0x7f5f480fef00 2023-07-14T19:17:04.523 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:04.519+0000 7f5f4ed5f700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f4819f830 con 0x7f5f480fe5b0 2023-07-14T19:17:04.523 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:04.519+0000 7f5f4ed5f700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f4819f9c0 con 0x7f5f480ff8b0 2023-07-14T19:17:04.523 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:04.519+0000 7f5f47fff700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5f480fef00 0x7f5f481980a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:17:04.523 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:04.519+0000 7f5f47fff700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5f480fef00 0x7f5f481980a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:33932/0 (socket says 172.21.15.40:33932) 2023-07-14T19:17:04.523 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:04.519+0000 7f5f47fff700 1 -- 172.21.15.40:0/3163031035 learned_addr learned my addr 172.21.15.40:0/3163031035 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-07-14T19:17:04.523 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:04.519+0000 7f5f4d2fc700 1 --2- 172.21.15.40:0/3163031035 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5f480ff8b0 0x7f5f4819f120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:17:04.523 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:04.519+0000 7f5f4cafb700 1 --2- 172.21.15.40:0/3163031035 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5f480fe5b0 0x7f5f48197b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:17:04.524 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:04.520+0000 7f5f47fff700 1 -- 172.21.15.40:0/3163031035 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5f480ff8b0 msgr2=0x7f5f4819f120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:17:04.524 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:04.520+0000 7f5f47fff700 1 --2- 172.21.15.40:0/3163031035 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5f480ff8b0 0x7f5f4819f120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:04.524 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:04.520+0000 7f5f47fff700 1 -- 172.21.15.40:0/3163031035 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5f480fe5b0 msgr2=0x7f5f48197b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:17:04.524 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:04.520+0000 7f5f47fff700 1 --2- 172.21.15.40:0/3163031035 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5f480fe5b0 0x7f5f48197b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:04.524 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:04.520+0000 7f5f47fff700 1 -- 172.21.15.40:0/3163031035 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f340095f0 con 0x7f5f480fef00 2023-07-14T19:17:04.524 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:04.520+0000 7f5f4cafb700 1 --2- 172.21.15.40:0/3163031035 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5f480fe5b0 0x7f5f48197b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-14T19:17:04.524 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:04.520+0000 7f5f47fff700 1 --2- 172.21.15.40:0/3163031035 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5f480fef00 0x7f5f481980a0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f5f34005910 tx=0x7f5f3400f830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:17:04.525 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:04.520+0000 7f5f45ffb700 1 -- 172.21.15.40:0/3163031035 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f3401d070 con 0x7f5f480fef00 2023-07-14T19:17:04.525 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:04.520+0000 7f5f45ffb700 1 -- 172.21.15.40:0/3163031035 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f5f3400fd10 con 0x7f5f480fef00 2023-07-14T19:17:04.525 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:04.520+0000 7f5f45ffb700 1 -- 172.21.15.40:0/3163031035 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f340176e0 con 0x7f5f480fef00 2023-07-14T19:17:04.525 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:04.520+0000 7f5f4ed5f700 1 -- 172.21.15.40:0/3163031035 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5f4819fc40 con 0x7f5f480fef00 2023-07-14T19:17:04.526 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:04.520+0000 7f5f4ed5f700 1 -- 172.21.15.40:0/3163031035 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5f481a0090 con 0x7f5f480fef00 2023-07-14T19:17:04.526 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:04.521+0000 7f5f4ed5f700 1 -- 172.21.15.40:0/3163031035 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5f48109390 con 0x7f5f480fef00 2023-07-14T19:17:04.526 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:04.522+0000 7f5f45ffb700 1 -- 172.21.15.40:0/3163031035 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7f5f34022470 con 0x7f5f480fef00 2023-07-14T19:17:04.526 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:04.523+0000 7f5f45ffb700 1 --2- 172.21.15.40:0/3163031035 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f5f380a1260 0x7f5f380a3710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:17:04.526 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:04.523+0000 7f5f4cafb700 1 --2- 172.21.15.40:0/3163031035 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f5f380a1260 0x7f5f380a3710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:17:04.527 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:04.523+0000 7f5f45ffb700 1 -- 172.21.15.40:0/3163031035 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 8269+0+0 (secure 0 0 0) 0x7f5f340cded0 con 0x7f5f480fef00 2023-07-14T19:17:04.527 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:04.523+0000 7f5f4cafb700 1 --2- 172.21.15.40:0/3163031035 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f5f380a1260 0x7f5f380a3710 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f5f3c005fd0 tx=0x7f5f3c005e20 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:17:04.527 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:04.527+0000 7f5f45ffb700 1 -- 172.21.15.40:0/3163031035 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f5f3405c780 con 0x7f5f480fef00 2023-07-14T19:17:04.738 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:04.737+0000 7f5f4ed5f700 1 -- 172.21.15.40:0/3163031035 --> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] -- mgr_command(tid 0: {"prefix": "orch ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f5f48060a80 con 0x7f5f380a1260 2023-07-14T19:17:04.745 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:04.744+0000 7f5f45ffb700 1 -- 172.21.15.40:0/3163031035 <== mgr.14152 v2:172.21.15.40:6800/1693027118 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+980 (secure 0 0 0) 0x7f5f48060a80 con 0x7f5f380a1260 2023-07-14T19:17:04.745 INFO:teuthology.orchestra.run.smithi040.stdout:NAME PORTS RUNNING REFRESHED AGE PLACEMENT 2023-07-14T19:17:04.745 INFO:teuthology.orchestra.run.smithi040.stdout:mds.all 12/12 - 98s smithi040=a;smithi040=d;smithi040=g;smithi040=j;smithi130=b;smithi130=e;smithi130=h;smithi130=k;smithi191=c;smithi191=f;smithi191=i;smithi191=l;count:12 2023-07-14T19:17:04.745 INFO:teuthology.orchestra.run.smithi040.stdout:mgr 3/3 - 6m smithi040=x;smithi130=y;smithi191=z;count:3 2023-07-14T19:17:04.745 INFO:teuthology.orchestra.run.smithi040.stdout:mon 3/3 - 6m smithi040:172.21.15.40=a;smithi130:172.21.15.130=b;smithi191:172.21.15.191=c;count:3 2023-07-14T19:17:04.746 INFO:teuthology.orchestra.run.smithi040.stdout:osd 12 - - 2023-07-14T19:17:04.747 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:04.746+0000 7f5f4ed5f700 1 -- 172.21.15.40:0/3163031035 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f5f380a1260 msgr2=0x7f5f380a3710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:17:04.747 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:04.746+0000 7f5f4ed5f700 1 --2- 172.21.15.40:0/3163031035 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f5f380a1260 0x7f5f380a3710 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f5f3c005fd0 tx=0x7f5f3c005e20 comp rx=0 tx=0).stop 2023-07-14T19:17:04.747 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:04.746+0000 7f5f4ed5f700 1 -- 172.21.15.40:0/3163031035 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5f480fef00 msgr2=0x7f5f481980a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:17:04.747 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:04.746+0000 7f5f4ed5f700 1 --2- 172.21.15.40:0/3163031035 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5f480fef00 0x7f5f481980a0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f5f34005910 tx=0x7f5f3400f830 comp rx=0 tx=0).stop 2023-07-14T19:17:04.747 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:04.746+0000 7f5f4ed5f700 1 -- 172.21.15.40:0/3163031035 shutdown_connections 2023-07-14T19:17:04.748 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:04.746+0000 7f5f4ed5f700 1 --2- 172.21.15.40:0/3163031035 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f5f380a1260 0x7f5f380a3710 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:04.748 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:04.746+0000 7f5f4ed5f700 1 --2- 172.21.15.40:0/3163031035 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5f480ff8b0 0x7f5f4819f120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:04.748 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:04.746+0000 7f5f4ed5f700 1 --2- 172.21.15.40:0/3163031035 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5f480fe5b0 0x7f5f48197b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:04.748 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:04.746+0000 7f5f4ed5f700 1 --2- 172.21.15.40:0/3163031035 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5f480fef00 0x7f5f481980a0 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:04.748 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:04.746+0000 7f5f4ed5f700 1 -- 172.21.15.40:0/3163031035 >> 172.21.15.40:0/3163031035 conn(0x7f5f480fa1e0 msgr2=0x7f5f480fbdf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:17:04.748 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:04.746+0000 7f5f4ed5f700 1 -- 172.21.15.40:0/3163031035 shutdown_connections 2023-07-14T19:17:04.748 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:04.746+0000 7f5f4ed5f700 1 -- 172.21.15.40:0/3163031035 wait complete. 2023-07-14T19:17:05.201 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:17:04 smithi040 ceph-mon[96518]: pgmap v289: 97 pgs: 97 active+clean; 638 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 9 op/s; 178 B/s, 1 objects/s recovering 2023-07-14T19:17:05.201 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:17:04 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:17:05.202 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:17:04 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:17:05.202 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:17:04 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:17:05.202 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:17:04 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T19:17:05.202 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:17:04 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:17:05.222 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:17:04 smithi191 ceph-mon[98311]: pgmap v289: 97 pgs: 97 active+clean; 638 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 9 op/s; 178 B/s, 1 objects/s recovering 2023-07-14T19:17:05.222 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:17:04 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:17:05.222 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:17:04 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:17:05.222 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:17:04 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:17:05.222 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:17:04 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T19:17:05.223 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:17:04 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:17:05.234 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid adc52f04-2279-11ee-9b33-001a4aab830c -- bash -c 'ceph orch host ls' 2023-07-14T19:17:05.242 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:17:04 smithi130 ceph-mon[97832]: pgmap v289: 97 pgs: 97 active+clean; 638 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 9 op/s; 178 B/s, 1 objects/s recovering 2023-07-14T19:17:05.242 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:17:04 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:17:05.242 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:17:04 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:17:05.242 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:17:04 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:17:05.243 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:17:04 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T19:17:05.243 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:17:04 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' 2023-07-14T19:17:05.794 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/adc52f04-2279-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T19:17:06.056 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:17:05 smithi040 ceph-mon[96518]: from='client.15102 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T19:17:06.222 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:17:05 smithi191 ceph-mon[98311]: from='client.15102 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T19:17:06.242 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:17:05 smithi130 ceph-mon[97832]: from='client.15102 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T19:17:06.841 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:17:06 smithi040 ceph-mon[96518]: pgmap v290: 97 pgs: 97 active+clean; 638 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s; 149 B/s, 1 objects/s recovering 2023-07-14T19:17:06.988 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:06.987+0000 7fa5c37d1700 1 -- 172.21.15.40:0/1788219012 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa5bc100740 msgr2=0x7fa5bc107000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:17:06.988 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:06.987+0000 7fa5c37d1700 1 --2- 172.21.15.40:0/1788219012 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa5bc100740 0x7fa5bc107000 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7fa5b400b0a0 tx=0x7fa5b400b3b0 comp rx=0 tx=0).stop 2023-07-14T19:17:06.988 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:06.987+0000 7fa5c37d1700 1 -- 172.21.15.40:0/1788219012 shutdown_connections 2023-07-14T19:17:06.988 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:06.987+0000 7fa5c37d1700 1 --2- 172.21.15.40:0/1788219012 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa5bc0febb0 0x7fa5bc0fefc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:06.988 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:06.987+0000 7fa5c37d1700 1 --2- 172.21.15.40:0/1788219012 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa5bc100740 0x7fa5bc107000 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:06.989 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:06.987+0000 7fa5c37d1700 1 --2- 172.21.15.40:0/1788219012 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa5bc0ffdb0 0x7fa5bc100200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:06.989 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:06.987+0000 7fa5c37d1700 1 -- 172.21.15.40:0/1788219012 >> 172.21.15.40:0/1788219012 conn(0x7fa5bc0fa1e0 msgr2=0x7fa5bc0fc630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:17:06.989 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:06.987+0000 7fa5c37d1700 1 -- 172.21.15.40:0/1788219012 shutdown_connections 2023-07-14T19:17:06.989 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:06.987+0000 7fa5c37d1700 1 -- 172.21.15.40:0/1788219012 wait complete. 2023-07-14T19:17:06.990 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:06.988+0000 7fa5c37d1700 1 Processor -- start 2023-07-14T19:17:06.991 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:06.988+0000 7fa5c37d1700 1 -- start start 2023-07-14T19:17:06.991 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:06.988+0000 7fa5c37d1700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa5bc0febb0 0x7fa5bc195a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:17:06.991 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:06.988+0000 7fa5c37d1700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa5bc0ffdb0 0x7fa5bc195f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:17:06.991 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:06.988+0000 7fa5c37d1700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa5bc100740 0x7fa5bc19d010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:17:06.991 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:06.988+0000 7fa5c37d1700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5bc19d5e0 con 0x7fa5bc0febb0 2023-07-14T19:17:06.992 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:06.988+0000 7fa5c37d1700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5bc19d720 con 0x7fa5bc0ffdb0 2023-07-14T19:17:06.992 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:06.988+0000 7fa5c37d1700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5bc19d8b0 con 0x7fa5bc100740 2023-07-14T19:17:06.992 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:06.988+0000 7fa5c156d700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa5bc0febb0 0x7fa5bc195a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:17:06.992 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:06.988+0000 7fa5c156d700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa5bc0febb0 0x7fa5bc195a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:33940/0 (socket says 172.21.15.40:33940) 2023-07-14T19:17:06.992 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:06.988+0000 7fa5c156d700 1 -- 172.21.15.40:0/919781709 learned_addr learned my addr 172.21.15.40:0/919781709 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-07-14T19:17:06.992 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:06.988+0000 7fa5c0d6c700 1 --2- 172.21.15.40:0/919781709 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa5bc0ffdb0 0x7fa5bc195f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:17:06.992 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:06.988+0000 7fa5c1d6e700 1 --2- 172.21.15.40:0/919781709 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa5bc100740 0x7fa5bc19d010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:17:06.993 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:06.988+0000 7fa5c156d700 1 -- 172.21.15.40:0/919781709 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa5bc100740 msgr2=0x7fa5bc19d010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:17:06.993 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:06.988+0000 7fa5c156d700 1 --2- 172.21.15.40:0/919781709 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa5bc100740 0x7fa5bc19d010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:06.993 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:06.988+0000 7fa5c156d700 1 -- 172.21.15.40:0/919781709 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa5bc0ffdb0 msgr2=0x7fa5bc195f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:17:06.993 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:06.988+0000 7fa5c156d700 1 --2- 172.21.15.40:0/919781709 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa5bc0ffdb0 0x7fa5bc195f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:06.993 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:06.988+0000 7fa5c156d700 1 -- 172.21.15.40:0/919781709 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5b4009d00 con 0x7fa5bc0febb0 2023-07-14T19:17:06.993 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:06.989+0000 7fa5c156d700 1 --2- 172.21.15.40:0/919781709 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa5bc0febb0 0x7fa5bc195a50 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7fa5b800c9a0 tx=0x7fa5b800ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:17:06.993 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:06.989+0000 7fa5b27fc700 1 -- 172.21.15.40:0/919781709 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5b8004220 con 0x7fa5bc0febb0 2023-07-14T19:17:06.994 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:06.989+0000 7fa5b27fc700 1 -- 172.21.15.40:0/919781709 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa5b8007d20 con 0x7fa5bc0febb0 2023-07-14T19:17:06.994 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:06.989+0000 7fa5c37d1700 1 -- 172.21.15.40:0/919781709 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa5bc19db90 con 0x7fa5bc0febb0 2023-07-14T19:17:06.994 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:06.989+0000 7fa5b27fc700 1 -- 172.21.15.40:0/919781709 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5b8003c00 con 0x7fa5bc0febb0 2023-07-14T19:17:06.994 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:06.989+0000 7fa5c37d1700 1 -- 172.21.15.40:0/919781709 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa5bc19e0e0 con 0x7fa5bc0febb0 2023-07-14T19:17:06.994 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:06.990+0000 7fa5c37d1700 1 -- 172.21.15.40:0/919781709 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa5bc10a380 con 0x7fa5bc0febb0 2023-07-14T19:17:06.995 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:06.991+0000 7fa5b27fc700 1 -- 172.21.15.40:0/919781709 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7fa5b8007860 con 0x7fa5bc0febb0 2023-07-14T19:17:06.995 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:06.991+0000 7fa5b27fc700 1 --2- 172.21.15.40:0/919781709 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7fa5a80a1260 0x7fa5a80a3710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:17:06.995 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:06.992+0000 7fa5c0d6c700 1 --2- 172.21.15.40:0/919781709 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7fa5a80a1260 0x7fa5a80a3710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:17:06.995 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:06.992+0000 7fa5b27fc700 1 -- 172.21.15.40:0/919781709 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 8269+0+0 (secure 0 0 0) 0x7fa5b80cca10 con 0x7fa5bc0febb0 2023-07-14T19:17:06.995 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:06.992+0000 7fa5c0d6c700 1 --2- 172.21.15.40:0/919781709 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7fa5a80a1260 0x7fa5a80a3710 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fa5ac00abd0 tx=0x7fa5ac00a380 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:17:06.996 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:06.995+0000 7fa5b27fc700 1 -- 172.21.15.40:0/919781709 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fa5b8096060 con 0x7fa5bc0febb0 2023-07-14T19:17:07.199 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:07.199+0000 7fa5c37d1700 1 -- 172.21.15.40:0/919781709 --> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""]}) v1 -- 0x7fa5bc068810 con 0x7fa5a80a1260 2023-07-14T19:17:07.201 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:07.200+0000 7fa5b27fc700 1 -- 172.21.15.40:0/919781709 <== mgr.14152 v2:172.21.15.40:6800/1693027118 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+191 (secure 0 0 0) 0x7fa5bc068810 con 0x7fa5a80a1260 2023-07-14T19:17:07.201 INFO:teuthology.orchestra.run.smithi040.stdout:HOST ADDR LABELS STATUS 2023-07-14T19:17:07.201 INFO:teuthology.orchestra.run.smithi040.stdout:smithi040 172.21.15.40 2023-07-14T19:17:07.201 INFO:teuthology.orchestra.run.smithi040.stdout:smithi130 172.21.15.130 2023-07-14T19:17:07.202 INFO:teuthology.orchestra.run.smithi040.stdout:smithi191 172.21.15.191 2023-07-14T19:17:07.202 INFO:teuthology.orchestra.run.smithi040.stdout:3 hosts in cluster 2023-07-14T19:17:07.203 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:07.202+0000 7fa5c37d1700 1 -- 172.21.15.40:0/919781709 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7fa5a80a1260 msgr2=0x7fa5a80a3710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:17:07.203 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:07.202+0000 7fa5c37d1700 1 --2- 172.21.15.40:0/919781709 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7fa5a80a1260 0x7fa5a80a3710 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fa5ac00abd0 tx=0x7fa5ac00a380 comp rx=0 tx=0).stop 2023-07-14T19:17:07.204 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:07.202+0000 7fa5c37d1700 1 -- 172.21.15.40:0/919781709 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa5bc0febb0 msgr2=0x7fa5bc195a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:17:07.204 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:07.202+0000 7fa5c37d1700 1 --2- 172.21.15.40:0/919781709 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa5bc0febb0 0x7fa5bc195a50 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7fa5b800c9a0 tx=0x7fa5b800ccb0 comp rx=0 tx=0).stop 2023-07-14T19:17:07.204 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:07.203+0000 7fa5c37d1700 1 -- 172.21.15.40:0/919781709 shutdown_connections 2023-07-14T19:17:07.204 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:07.203+0000 7fa5c37d1700 1 --2- 172.21.15.40:0/919781709 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7fa5a80a1260 0x7fa5a80a3710 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:07.204 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:07.203+0000 7fa5c37d1700 1 --2- 172.21.15.40:0/919781709 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa5bc100740 0x7fa5bc19d010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:07.204 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:07.203+0000 7fa5c37d1700 1 --2- 172.21.15.40:0/919781709 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa5bc0febb0 0x7fa5bc195a50 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:07.205 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:07.203+0000 7fa5c37d1700 1 --2- 172.21.15.40:0/919781709 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa5bc0ffdb0 0x7fa5bc195f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:07.205 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:07.203+0000 7fa5c37d1700 1 -- 172.21.15.40:0/919781709 >> 172.21.15.40:0/919781709 conn(0x7fa5bc0fa1e0 msgr2=0x7fa5bc068100 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:17:07.205 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:07.203+0000 7fa5c37d1700 1 -- 172.21.15.40:0/919781709 shutdown_connections 2023-07-14T19:17:07.205 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:07.203+0000 7fa5c37d1700 1 -- 172.21.15.40:0/919781709 wait complete. 2023-07-14T19:17:07.222 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:17:06 smithi191 ceph-mon[98311]: pgmap v290: 97 pgs: 97 active+clean; 638 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s; 149 B/s, 1 objects/s recovering 2023-07-14T19:17:07.242 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:17:06 smithi130 ceph-mon[97832]: pgmap v290: 97 pgs: 97 active+clean; 638 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s; 149 B/s, 1 objects/s recovering 2023-07-14T19:17:07.744 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid adc52f04-2279-11ee-9b33-001a4aab830c -- bash -c 'ceph orch device ls' 2023-07-14T19:17:08.304 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/adc52f04-2279-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T19:17:08.946 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:17:08 smithi040 ceph-mon[96518]: pgmap v291: 97 pgs: 97 active+clean; 638 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s; 93 B/s, 0 objects/s recovering 2023-07-14T19:17:08.946 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:17:08 smithi040 ceph-mon[96518]: from='client.15108 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T19:17:09.222 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:17:08 smithi191 ceph-mon[98311]: pgmap v291: 97 pgs: 97 active+clean; 638 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s; 93 B/s, 0 objects/s recovering 2023-07-14T19:17:09.222 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:17:08 smithi191 ceph-mon[98311]: from='client.15108 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T19:17:09.242 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:17:08 smithi130 ceph-mon[97832]: pgmap v291: 97 pgs: 97 active+clean; 638 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s; 93 B/s, 0 objects/s recovering 2023-07-14T19:17:09.242 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:17:08 smithi130 ceph-mon[97832]: from='client.15108 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T19:17:09.722 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:09.719+0000 7f5f8ba61700 1 -- 172.21.15.40:0/3346954788 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5f840ff3e0 msgr2=0x7f5f841017c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:17:09.723 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:09.719+0000 7f5f8ba61700 1 --2- 172.21.15.40:0/3346954788 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5f840ff3e0 0x7f5f841017c0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f5f74009990 tx=0x7f5f74009ca0 comp rx=0 tx=0).stop 2023-07-14T19:17:09.723 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:09.719+0000 7f5f8ba61700 1 -- 172.21.15.40:0/3346954788 shutdown_connections 2023-07-14T19:17:09.723 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:09.719+0000 7f5f8ba61700 1 --2- 172.21.15.40:0/3346954788 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5f84104830 0x7f5f84106ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:09.724 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:09.719+0000 7f5f8ba61700 1 --2- 172.21.15.40:0/3346954788 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5f840ff3e0 0x7f5f841017c0 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:09.724 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:09.719+0000 7f5f8ba61700 1 --2- 172.21.15.40:0/3346954788 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5f84101d90 0x7f5f84104170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:09.724 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:09.719+0000 7f5f8ba61700 1 -- 172.21.15.40:0/3346954788 >> 172.21.15.40:0/3346954788 conn(0x7f5f840f9500 msgr2=0x7f5f840fb970 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:17:09.724 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:09.719+0000 7f5f8ba61700 1 -- 172.21.15.40:0/3346954788 shutdown_connections 2023-07-14T19:17:09.725 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:09.719+0000 7f5f8ba61700 1 -- 172.21.15.40:0/3346954788 wait complete. 2023-07-14T19:17:09.725 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:09.720+0000 7f5f8ba61700 1 Processor -- start 2023-07-14T19:17:09.725 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:09.720+0000 7f5f8ba61700 1 -- start start 2023-07-14T19:17:09.725 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:09.720+0000 7f5f8ba61700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5f840ff3e0 0x7f5f84197b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:17:09.726 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:09.720+0000 7f5f8ba61700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5f84101d90 0x7f5f841980d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:17:09.726 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:09.720+0000 7f5f8ba61700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5f84104830 0x7f5f8419f150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:17:09.726 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:09.720+0000 7f5f8ba61700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f8419f720 con 0x7f5f840ff3e0 2023-07-14T19:17:09.726 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:09.720+0000 7f5f8ba61700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f8419f860 con 0x7f5f84104830 2023-07-14T19:17:09.727 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:09.720+0000 7f5f8ba61700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f8419f9f0 con 0x7f5f84101d90 2023-07-14T19:17:09.727 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:09.720+0000 7f5f897fd700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5f840ff3e0 0x7f5f84197b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:17:09.727 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:09.720+0000 7f5f897fd700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5f840ff3e0 0x7f5f84197b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:33954/0 (socket says 172.21.15.40:33954) 2023-07-14T19:17:09.728 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:09.720+0000 7f5f897fd700 1 -- 172.21.15.40:0/4014213829 learned_addr learned my addr 172.21.15.40:0/4014213829 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-07-14T19:17:09.728 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:09.720+0000 7f5f88ffc700 1 --2- 172.21.15.40:0/4014213829 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5f84101d90 0x7f5f841980d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:17:09.728 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:09.720+0000 7f5f897fd700 1 -- 172.21.15.40:0/4014213829 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5f84101d90 msgr2=0x7f5f841980d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:17:09.729 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:09.720+0000 7f5f897fd700 1 --2- 172.21.15.40:0/4014213829 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5f84101d90 0x7f5f841980d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:09.729 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:09.720+0000 7f5f89ffe700 1 --2- 172.21.15.40:0/4014213829 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5f84104830 0x7f5f8419f150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:17:09.729 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:09.720+0000 7f5f897fd700 1 -- 172.21.15.40:0/4014213829 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5f84104830 msgr2=0x7f5f8419f150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:17:09.729 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:09.720+0000 7f5f897fd700 1 --2- 172.21.15.40:0/4014213829 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5f84104830 0x7f5f8419f150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:09.730 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:09.720+0000 7f5f897fd700 1 -- 172.21.15.40:0/4014213829 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f740095f0 con 0x7f5f840ff3e0 2023-07-14T19:17:09.730 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:09.721+0000 7f5f897fd700 1 --2- 172.21.15.40:0/4014213829 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5f840ff3e0 0x7f5f84197b90 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f5f740059d0 tx=0x7f5f7400f740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:17:09.730 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:09.721+0000 7f5f7a7fc700 1 -- 172.21.15.40:0/4014213829 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f7401d070 con 0x7f5f840ff3e0 2023-07-14T19:17:09.730 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:09.721+0000 7f5f7a7fc700 1 -- 172.21.15.40:0/4014213829 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f5f7400fbf0 con 0x7f5f840ff3e0 2023-07-14T19:17:09.731 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:09.721+0000 7f5f7a7fc700 1 -- 172.21.15.40:0/4014213829 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f74017650 con 0x7f5f840ff3e0 2023-07-14T19:17:09.731 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:09.721+0000 7f5f8ba61700 1 -- 172.21.15.40:0/4014213829 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5f8419fc70 con 0x7f5f840ff3e0 2023-07-14T19:17:09.731 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:09.721+0000 7f5f8ba61700 1 -- 172.21.15.40:0/4014213829 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5f841a0030 con 0x7f5f840ff3e0 2023-07-14T19:17:09.731 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:09.722+0000 7f5f8ba61700 1 -- 172.21.15.40:0/4014213829 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5f8410a390 con 0x7f5f840ff3e0 2023-07-14T19:17:09.733 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:09.723+0000 7f5f7a7fc700 1 -- 172.21.15.40:0/4014213829 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7f5f740050d0 con 0x7f5f840ff3e0 2023-07-14T19:17:09.733 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:09.724+0000 7f5f7a7fc700 1 --2- 172.21.15.40:0/4014213829 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f5f700a11e0 0x7f5f700a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:17:09.733 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:09.724+0000 7f5f88ffc700 1 --2- 172.21.15.40:0/4014213829 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f5f700a11e0 0x7f5f700a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:17:09.733 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:09.724+0000 7f5f7a7fc700 1 -- 172.21.15.40:0/4014213829 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 8269+0+0 (secure 0 0 0) 0x7f5f740ce320 con 0x7f5f840ff3e0 2023-07-14T19:17:09.733 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:09.724+0000 7f5f88ffc700 1 --2- 172.21.15.40:0/4014213829 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f5f700a11e0 0x7f5f700a3690 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f5f80005950 tx=0x7f5f800058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:17:09.734 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:09.728+0000 7f5f7a7fc700 1 -- 172.21.15.40:0/4014213829 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f5f74097980 con 0x7f5f840ff3e0 2023-07-14T19:17:09.933 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:09.932+0000 7f5f8ba61700 1 -- 172.21.15.40:0/4014213829 --> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] -- mgr_command(tid 0: {"prefix": "orch device ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f5f84105ce0 con 0x7f5f700a11e0 2023-07-14T19:17:09.935 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:09.934+0000 7f5f7a7fc700 1 -- 172.21.15.40:0/4014213829 <== mgr.14152 v2:172.21.15.40:6800/1693027118 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+492 (secure 0 0 0) 0x7f5f84105ce0 con 0x7f5f700a11e0 2023-07-14T19:17:09.935 INFO:teuthology.orchestra.run.smithi040.stdout:HOST PATH TYPE DEVICE ID SIZE AVAILABLE REFRESHED REJECT REASONS 2023-07-14T19:17:09.935 INFO:teuthology.orchestra.run.smithi040.stdout:smithi040 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_CVFT5330009J400BGN 372G No 2m ago LVM detected, locked 2023-07-14T19:17:09.936 INFO:teuthology.orchestra.run.smithi040.stdout:smithi130 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_PHFT620500Q1400BGN 372G No 2m ago LVM detected, locked 2023-07-14T19:17:09.936 INFO:teuthology.orchestra.run.smithi040.stdout:smithi191 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_CVFT6303008M400BGN 372G No 114s ago LVM detected, locked 2023-07-14T19:17:09.937 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:09.937+0000 7f5f8ba61700 1 -- 172.21.15.40:0/4014213829 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f5f700a11e0 msgr2=0x7f5f700a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:17:09.938 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:09.937+0000 7f5f8ba61700 1 --2- 172.21.15.40:0/4014213829 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f5f700a11e0 0x7f5f700a3690 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f5f80005950 tx=0x7f5f800058e0 comp rx=0 tx=0).stop 2023-07-14T19:17:09.938 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:09.937+0000 7f5f8ba61700 1 -- 172.21.15.40:0/4014213829 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5f840ff3e0 msgr2=0x7f5f84197b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:17:09.938 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:09.937+0000 7f5f8ba61700 1 --2- 172.21.15.40:0/4014213829 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5f840ff3e0 0x7f5f84197b90 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f5f740059d0 tx=0x7f5f7400f740 comp rx=0 tx=0).stop 2023-07-14T19:17:09.938 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:09.937+0000 7f5f8ba61700 1 -- 172.21.15.40:0/4014213829 shutdown_connections 2023-07-14T19:17:09.938 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:09.937+0000 7f5f8ba61700 1 --2- 172.21.15.40:0/4014213829 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f5f700a11e0 0x7f5f700a3690 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:09.938 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:09.937+0000 7f5f8ba61700 1 --2- 172.21.15.40:0/4014213829 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5f84104830 0x7f5f8419f150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:09.938 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:09.937+0000 7f5f8ba61700 1 --2- 172.21.15.40:0/4014213829 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5f840ff3e0 0x7f5f84197b90 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:09.939 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:09.937+0000 7f5f8ba61700 1 --2- 172.21.15.40:0/4014213829 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f5f84101d90 0x7f5f841980d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:09.939 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:09.937+0000 7f5f8ba61700 1 -- 172.21.15.40:0/4014213829 >> 172.21.15.40:0/4014213829 conn(0x7f5f840f9500 msgr2=0x7f5f840fe990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:17:09.939 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:09.937+0000 7f5f8ba61700 1 -- 172.21.15.40:0/4014213829 shutdown_connections 2023-07-14T19:17:09.939 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:09.937+0000 7f5f8ba61700 1 -- 172.21.15.40:0/4014213829 wait complete. 2023-07-14T19:17:10.565 INFO:teuthology.run_tasks:Running task cephadm.shell... 2023-07-14T19:17:10.577 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi040.front.sepia.ceph.com 2023-07-14T19:17:10.578 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid adc52f04-2279-11ee-9b33-001a4aab830c -- bash -c 'ceph fs dump' 2023-07-14T19:17:11.141 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/adc52f04-2279-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T19:17:11.221 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:17:10 smithi040 ceph-mon[96518]: pgmap v292: 97 pgs: 97 active+clean; 638 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s; 93 B/s, 0 objects/s recovering 2023-07-14T19:17:11.221 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:17:10 smithi191 ceph-mon[98311]: pgmap v292: 97 pgs: 97 active+clean; 638 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s; 93 B/s, 0 objects/s recovering 2023-07-14T19:17:11.242 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:17:10 smithi130 ceph-mon[97832]: pgmap v292: 97 pgs: 97 active+clean; 638 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s; 93 B/s, 0 objects/s recovering 2023-07-14T19:17:11.971 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:17:11 smithi040 ceph-mon[96518]: from='client.15114 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T19:17:12.222 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:17:11 smithi191 ceph-mon[98311]: from='client.15114 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T19:17:12.242 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:17:11 smithi130 ceph-mon[97832]: from='client.15114 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T19:17:12.495 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:12.494+0000 7fa5ddef0700 1 -- 172.21.15.40:0/3390978929 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa5d80ffdb0 msgr2=0x7fa5d8100200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:17:12.496 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:12.494+0000 7fa5ddef0700 1 --2- 172.21.15.40:0/3390978929 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa5d80ffdb0 0x7fa5d8100200 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7fa5cc009990 tx=0x7fa5cc009ca0 comp rx=0 tx=0).stop 2023-07-14T19:17:12.496 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:12.494+0000 7fa5ddef0700 1 -- 172.21.15.40:0/3390978929 shutdown_connections 2023-07-14T19:17:12.496 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:12.494+0000 7fa5ddef0700 1 --2- 172.21.15.40:0/3390978929 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa5d80febb0 0x7fa5d80fefc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:12.497 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:12.494+0000 7fa5ddef0700 1 --2- 172.21.15.40:0/3390978929 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa5d8100740 0x7fa5d8107000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:12.497 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:12.494+0000 7fa5ddef0700 1 --2- 172.21.15.40:0/3390978929 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa5d80ffdb0 0x7fa5d8100200 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:12.497 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:12.494+0000 7fa5ddef0700 1 -- 172.21.15.40:0/3390978929 >> 172.21.15.40:0/3390978929 conn(0x7fa5d80fa1e0 msgr2=0x7fa5d80fc630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:17:12.497 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:12.494+0000 7fa5ddef0700 1 -- 172.21.15.40:0/3390978929 shutdown_connections 2023-07-14T19:17:12.497 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:12.494+0000 7fa5ddef0700 1 -- 172.21.15.40:0/3390978929 wait complete. 2023-07-14T19:17:12.497 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:12.495+0000 7fa5ddef0700 1 Processor -- start 2023-07-14T19:17:12.497 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:12.495+0000 7fa5ddef0700 1 -- start start 2023-07-14T19:17:12.499 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:12.495+0000 7fa5ddef0700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa5d80febb0 0x7fa5d8068c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:17:12.499 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:12.495+0000 7fa5ddef0700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa5d80ffdb0 0x7fa5d819a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:17:12.499 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:12.495+0000 7fa5ddef0700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa5d8100740 0x7fa5d819ce10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:17:12.499 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:12.495+0000 7fa5ddef0700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5d819d5b0 con 0x7fa5d80ffdb0 2023-07-14T19:17:12.500 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:12.495+0000 7fa5ddef0700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5d819d720 con 0x7fa5d80febb0 2023-07-14T19:17:12.500 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:12.495+0000 7fa5ddef0700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5d819d8e0 con 0x7fa5d8100740 2023-07-14T19:17:12.500 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:12.495+0000 7fa5d6ffd700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa5d80ffdb0 0x7fa5d819a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:17:12.500 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:12.496+0000 7fa5d6ffd700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa5d80ffdb0 0x7fa5d819a7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:33968/0 (socket says 172.21.15.40:33968) 2023-07-14T19:17:12.500 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:12.496+0000 7fa5d6ffd700 1 -- 172.21.15.40:0/1292635947 learned_addr learned my addr 172.21.15.40:0/1292635947 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-07-14T19:17:12.500 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:12.496+0000 7fa5d77fe700 1 --2- 172.21.15.40:0/1292635947 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa5d80febb0 0x7fa5d8068c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:17:12.501 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:12.496+0000 7fa5d7fff700 1 --2- 172.21.15.40:0/1292635947 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa5d8100740 0x7fa5d819ce10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:17:12.501 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:12.496+0000 7fa5d6ffd700 1 -- 172.21.15.40:0/1292635947 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa5d8100740 msgr2=0x7fa5d819ce10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:17:12.501 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:12.496+0000 7fa5d6ffd700 1 --2- 172.21.15.40:0/1292635947 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa5d8100740 0x7fa5d819ce10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:12.501 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:12.496+0000 7fa5d6ffd700 1 -- 172.21.15.40:0/1292635947 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa5d80febb0 msgr2=0x7fa5d8068c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:17:12.502 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:12.496+0000 7fa5d6ffd700 1 --2- 172.21.15.40:0/1292635947 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa5d80febb0 0x7fa5d8068c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:12.502 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:12.496+0000 7fa5d6ffd700 1 -- 172.21.15.40:0/1292635947 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5cc0095f0 con 0x7fa5d80ffdb0 2023-07-14T19:17:12.502 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:12.496+0000 7fa5d6ffd700 1 --2- 172.21.15.40:0/1292635947 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa5d80ffdb0 0x7fa5d819a7d0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7fa5cc000c00 tx=0x7fa5cc00f830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:17:12.502 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:12.496+0000 7fa5d4ff9700 1 -- 172.21.15.40:0/1292635947 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5cc01d070 con 0x7fa5d80ffdb0 2023-07-14T19:17:12.502 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:12.496+0000 7fa5d4ff9700 1 -- 172.21.15.40:0/1292635947 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa5cc00fce0 con 0x7fa5d80ffdb0 2023-07-14T19:17:12.502 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:12.496+0000 7fa5ddef0700 1 -- 172.21.15.40:0/1292635947 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa5d819db60 con 0x7fa5d80ffdb0 2023-07-14T19:17:12.502 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:12.496+0000 7fa5d4ff9700 1 -- 172.21.15.40:0/1292635947 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5cc0176e0 con 0x7fa5d80ffdb0 2023-07-14T19:17:12.503 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:12.496+0000 7fa5ddef0700 1 -- 172.21.15.40:0/1292635947 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa5d819e000 con 0x7fa5d80ffdb0 2023-07-14T19:17:12.503 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:12.497+0000 7fa5ddef0700 1 -- 172.21.15.40:0/1292635947 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa5d81061a0 con 0x7fa5d80ffdb0 2023-07-14T19:17:12.503 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:12.498+0000 7fa5d4ff9700 1 -- 172.21.15.40:0/1292635947 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7fa5cc022470 con 0x7fa5d80ffdb0 2023-07-14T19:17:12.504 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:12.499+0000 7fa5d4ff9700 1 --2- 172.21.15.40:0/1292635947 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7fa5c00a11e0 0x7fa5c00a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:17:12.504 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:12.499+0000 7fa5d4ff9700 1 -- 172.21.15.40:0/1292635947 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 8269+0+0 (secure 0 0 0) 0x7fa5cc0ce390 con 0x7fa5d80ffdb0 2023-07-14T19:17:12.504 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:12.499+0000 7fa5d77fe700 1 --2- 172.21.15.40:0/1292635947 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7fa5c00a11e0 0x7fa5c00a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:17:12.504 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:12.500+0000 7fa5d77fe700 1 --2- 172.21.15.40:0/1292635947 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7fa5c00a11e0 0x7fa5c00a3690 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fa5c8006fd0 tx=0x7fa5c8008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:17:12.505 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:12.502+0000 7fa5d4ff9700 1 -- 172.21.15.40:0/1292635947 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fa5cc0979f0 con 0x7fa5d80ffdb0 2023-07-14T19:17:12.748 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:12.747+0000 7fa5ddef0700 1 -- 172.21.15.40:0/1292635947 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7fa5d802c8e0 con 0x7fa5d80ffdb0 2023-07-14T19:17:12.749 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:12.747+0000 7fa5d4ff9700 1 -- 172.21.15.40:0/1292635947 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 26 v26) v1 ==== 76+0+2925 (secure 0 0 0) 0x7fa5cc09b010 con 0x7fa5d80ffdb0 2023-07-14T19:17:12.749 INFO:teuthology.orchestra.run.smithi040.stdout:e26 2023-07-14T19:17:12.749 INFO:teuthology.orchestra.run.smithi040.stdout:enable_multiple, ever_enabled_multiple: 1,1 2023-07-14T19:17:12.750 INFO:teuthology.orchestra.run.smithi040.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2023-07-14T19:17:12.750 INFO:teuthology.orchestra.run.smithi040.stdout:legacy client fscid: 1 2023-07-14T19:17:12.750 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-07-14T19:17:12.750 INFO:teuthology.orchestra.run.smithi040.stdout:Filesystem 'cephfs' (1) 2023-07-14T19:17:12.750 INFO:teuthology.orchestra.run.smithi040.stdout:fs_name cephfs 2023-07-14T19:17:12.750 INFO:teuthology.orchestra.run.smithi040.stdout:epoch 26 2023-07-14T19:17:12.750 INFO:teuthology.orchestra.run.smithi040.stdout:flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2023-07-14T19:17:12.751 INFO:teuthology.orchestra.run.smithi040.stdout:created 2023-07-14T19:15:57.650710+0000 2023-07-14T19:17:12.751 INFO:teuthology.orchestra.run.smithi040.stdout:modified 2023-07-14T19:16:05.624568+0000 2023-07-14T19:17:12.751 INFO:teuthology.orchestra.run.smithi040.stdout:tableserver 0 2023-07-14T19:17:12.751 INFO:teuthology.orchestra.run.smithi040.stdout:root 0 2023-07-14T19:17:12.751 INFO:teuthology.orchestra.run.smithi040.stdout:session_timeout 300 2023-07-14T19:17:12.751 INFO:teuthology.orchestra.run.smithi040.stdout:session_autoclose 300 2023-07-14T19:17:12.751 INFO:teuthology.orchestra.run.smithi040.stdout:max_file_size 1099511627776 2023-07-14T19:17:12.752 INFO:teuthology.orchestra.run.smithi040.stdout:max_xattr_size 65536 2023-07-14T19:17:12.752 INFO:teuthology.orchestra.run.smithi040.stdout:required_client_features {} 2023-07-14T19:17:12.752 INFO:teuthology.orchestra.run.smithi040.stdout:last_failure 0 2023-07-14T19:17:12.752 INFO:teuthology.orchestra.run.smithi040.stdout:last_failure_osd_epoch 0 2023-07-14T19:17:12.752 INFO:teuthology.orchestra.run.smithi040.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2023-07-14T19:17:12.752 INFO:teuthology.orchestra.run.smithi040.stdout:max_mds 5 2023-07-14T19:17:12.752 INFO:teuthology.orchestra.run.smithi040.stdout:in 0,1,2,3,4 2023-07-14T19:17:12.752 INFO:teuthology.orchestra.run.smithi040.stdout:up {0=24539,1=24508,2=24506,3=24490,4=24482} 2023-07-14T19:17:12.753 INFO:teuthology.orchestra.run.smithi040.stdout:failed 2023-07-14T19:17:12.753 INFO:teuthology.orchestra.run.smithi040.stdout:damaged 2023-07-14T19:17:12.753 INFO:teuthology.orchestra.run.smithi040.stdout:stopped 2023-07-14T19:17:12.753 INFO:teuthology.orchestra.run.smithi040.stdout:data_pools [3] 2023-07-14T19:17:12.753 INFO:teuthology.orchestra.run.smithi040.stdout:metadata_pool 2 2023-07-14T19:17:12.753 INFO:teuthology.orchestra.run.smithi040.stdout:inline_data disabled 2023-07-14T19:17:12.753 INFO:teuthology.orchestra.run.smithi040.stdout:balancer 2023-07-14T19:17:12.754 INFO:teuthology.orchestra.run.smithi040.stdout:bal_rank_mask -1 2023-07-14T19:17:12.754 INFO:teuthology.orchestra.run.smithi040.stdout:standby_count_wanted 0 2023-07-14T19:17:12.754 INFO:teuthology.orchestra.run.smithi040.stdout:[mds.b{0:24539} state up:active seq 3 addr [v2:172.21.15.130:6838/2145479535,v1:172.21.15.130:6839/2145479535] compat {c=[1],r=[1],i=[7ff]}] 2023-07-14T19:17:12.754 INFO:teuthology.orchestra.run.smithi040.stdout:[mds.f{0:24478} state up:standby-replay seq 1 addr [v2:172.21.15.191:6834/3712674634,v1:172.21.15.191:6835/3712674634] compat {c=[1],r=[1],i=[7ff]}] 2023-07-14T19:17:12.754 INFO:teuthology.orchestra.run.smithi040.stdout:[mds.i{1:24508} state up:active seq 5 addr [v2:172.21.15.191:6838/1977507302,v1:172.21.15.191:6839/1977507302] compat {c=[1],r=[1],i=[7ff]}] 2023-07-14T19:17:12.754 INFO:teuthology.orchestra.run.smithi040.stdout:[mds.k{1:24452} state up:standby-replay seq 1 addr [v2:172.21.15.130:6832/3677452688,v1:172.21.15.130:6833/3677452688] compat {c=[1],r=[1],i=[7ff]}] 2023-07-14T19:17:12.754 INFO:teuthology.orchestra.run.smithi040.stdout:[mds.h{2:24506} state up:active seq 6 addr [v2:172.21.15.130:6836/2604019369,v1:172.21.15.130:6837/2604019369] compat {c=[1],r=[1],i=[7ff]}] 2023-07-14T19:17:12.754 INFO:teuthology.orchestra.run.smithi040.stdout:[mds.l{2:24433} state up:standby-replay seq 1 addr [v2:172.21.15.191:6832/31509862,v1:172.21.15.191:6833/31509862] compat {c=[1],r=[1],i=[7ff]}] 2023-07-14T19:17:12.755 INFO:teuthology.orchestra.run.smithi040.stdout:[mds.c{3:24490} state up:active seq 7 addr [v2:172.21.15.191:6836/2406892975,v1:172.21.15.191:6837/2406892975] compat {c=[1],r=[1],i=[7ff]}] 2023-07-14T19:17:12.755 INFO:teuthology.orchestra.run.smithi040.stdout:[mds.a{3:14598} state up:standby-replay seq 1 addr [v2:172.21.15.40:6840/1590486077,v1:172.21.15.40:6841/1590486077] compat {c=[1],r=[1],i=[7ff]}] 2023-07-14T19:17:12.755 INFO:teuthology.orchestra.run.smithi040.stdout:[mds.e{4:24482} state up:active seq 8 addr [v2:172.21.15.130:6834/3088490808,v1:172.21.15.130:6835/3088490808] compat {c=[1],r=[1],i=[7ff]}] 2023-07-14T19:17:12.755 INFO:teuthology.orchestra.run.smithi040.stdout:[mds.g{4:14586} state up:standby-replay seq 1 addr [v2:172.21.15.40:6838/959937545,v1:172.21.15.40:6839/959937545] compat {c=[1],r=[1],i=[7ff]}] 2023-07-14T19:17:12.755 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-07-14T19:17:12.755 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-07-14T19:17:12.755 INFO:teuthology.orchestra.run.smithi040.stdout:Standby daemons: 2023-07-14T19:17:12.755 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-07-14T19:17:12.756 INFO:teuthology.orchestra.run.smithi040.stdout:[mds.j{-1:14523} state up:standby seq 1 addr [v2:172.21.15.40:6834/501143718,v1:172.21.15.40:6835/501143718] compat {c=[1],r=[1],i=[7ff]}] 2023-07-14T19:17:12.756 INFO:teuthology.orchestra.run.smithi040.stdout:[mds.d{-1:14535} state up:standby seq 1 addr [v2:172.21.15.40:6836/2634476117,v1:172.21.15.40:6837/2634476117] compat {c=[1],r=[1],i=[7ff]}] 2023-07-14T19:17:12.756 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:12.749+0000 7fa5ddef0700 1 -- 172.21.15.40:0/1292635947 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7fa5c00a11e0 msgr2=0x7fa5c00a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:17:12.757 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:12.749+0000 7fa5ddef0700 1 --2- 172.21.15.40:0/1292635947 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7fa5c00a11e0 0x7fa5c00a3690 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fa5c8006fd0 tx=0x7fa5c8008040 comp rx=0 tx=0).stop 2023-07-14T19:17:12.757 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:12.749+0000 7fa5ddef0700 1 -- 172.21.15.40:0/1292635947 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa5d80ffdb0 msgr2=0x7fa5d819a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:17:12.757 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:12.749+0000 7fa5ddef0700 1 --2- 172.21.15.40:0/1292635947 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa5d80ffdb0 0x7fa5d819a7d0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7fa5cc000c00 tx=0x7fa5cc00f830 comp rx=0 tx=0).stop 2023-07-14T19:17:12.757 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:12.749+0000 7fa5ddef0700 1 -- 172.21.15.40:0/1292635947 shutdown_connections 2023-07-14T19:17:12.757 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:12.749+0000 7fa5ddef0700 1 --2- 172.21.15.40:0/1292635947 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7fa5c00a11e0 0x7fa5c00a3690 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:12.757 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:12.749+0000 7fa5ddef0700 1 --2- 172.21.15.40:0/1292635947 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fa5d8100740 0x7fa5d819ce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:12.758 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:12.749+0000 7fa5ddef0700 1 --2- 172.21.15.40:0/1292635947 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa5d80febb0 0x7fa5d8068c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:12.758 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:12.749+0000 7fa5ddef0700 1 --2- 172.21.15.40:0/1292635947 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa5d80ffdb0 0x7fa5d819a7d0 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:12.758 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:12.749+0000 7fa5ddef0700 1 -- 172.21.15.40:0/1292635947 >> 172.21.15.40:0/1292635947 conn(0x7fa5d80fa1e0 msgr2=0x7fa5d81032e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:17:12.758 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:12.749+0000 7fa5ddef0700 1 -- 172.21.15.40:0/1292635947 shutdown_connections 2023-07-14T19:17:12.758 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:12.750+0000 7fa5ddef0700 1 -- 172.21.15.40:0/1292635947 wait complete. 2023-07-14T19:17:12.758 INFO:teuthology.orchestra.run.smithi040.stderr:dumped fsmap epoch 26 2023-07-14T19:17:13.182 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:17:12 smithi040 ceph-mon[96518]: pgmap v293: 97 pgs: 97 active+clean; 638 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 10 op/s 2023-07-14T19:17:13.183 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:17:12 smithi040 ceph-mon[96518]: from='client.? 172.21.15.40:0/1292635947' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2023-07-14T19:17:13.194 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid adc52f04-2279-11ee-9b33-001a4aab830c -- bash -c 'ceph osd dump' 2023-07-14T19:17:13.222 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:17:12 smithi191 ceph-mon[98311]: pgmap v293: 97 pgs: 97 active+clean; 638 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 10 op/s 2023-07-14T19:17:13.227 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:17:12 smithi191 ceph-mon[98311]: from='client.? 172.21.15.40:0/1292635947' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2023-07-14T19:17:13.242 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:17:12 smithi130 ceph-mon[97832]: pgmap v293: 97 pgs: 97 active+clean; 638 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 10 op/s 2023-07-14T19:17:13.242 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:17:12 smithi130 ceph-mon[97832]: from='client.? 172.21.15.40:0/1292635947' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2023-07-14T19:17:13.754 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/adc52f04-2279-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T19:17:14.210 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:17:13 smithi040 ceph-mon[96518]: pgmap v294: 97 pgs: 97 active+clean; 638 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2023-07-14T19:17:14.222 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:17:13 smithi191 ceph-mon[98311]: pgmap v294: 97 pgs: 97 active+clean; 638 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2023-07-14T19:17:14.242 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:17:13 smithi130 ceph-mon[97832]: pgmap v294: 97 pgs: 97 active+clean; 638 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2023-07-14T19:17:15.108 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:15.104+0000 7f67a6453700 1 -- 172.21.15.40:0/2709725128 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f67a0105980 msgr2=0x7f67a0107de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:17:15.108 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:15.104+0000 7f67a6453700 1 --2- 172.21.15.40:0/2709725128 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f67a0105980 0x7f67a0107de0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f679800b0a0 tx=0x7f679800b3b0 comp rx=0 tx=0).stop 2023-07-14T19:17:15.108 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:15.104+0000 7f67a6453700 1 -- 172.21.15.40:0/2709725128 shutdown_connections 2023-07-14T19:17:15.108 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:15.104+0000 7f67a6453700 1 --2- 172.21.15.40:0/2709725128 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f67a0100740 0x7f67a0102b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:15.109 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:15.104+0000 7f67a6453700 1 --2- 172.21.15.40:0/2709725128 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f67a0105980 0x7f67a0107de0 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:15.109 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:15.104+0000 7f67a6453700 1 --2- 172.21.15.40:0/2709725128 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f67a0103060 0x7f67a0105440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:15.109 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:15.104+0000 7f67a6453700 1 -- 172.21.15.40:0/2709725128 >> 172.21.15.40:0/2709725128 conn(0x7f67a00fa1e0 msgr2=0x7f67a00fc630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:17:15.110 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:15.104+0000 7f67a6453700 1 -- 172.21.15.40:0/2709725128 shutdown_connections 2023-07-14T19:17:15.110 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:15.104+0000 7f67a6453700 1 -- 172.21.15.40:0/2709725128 wait complete. 2023-07-14T19:17:15.110 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:15.105+0000 7f67a6453700 1 Processor -- start 2023-07-14T19:17:15.111 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:15.105+0000 7f67a6453700 1 -- start start 2023-07-14T19:17:15.111 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:15.105+0000 7f67a6453700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f67a0100740 0x7f67a01000a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:17:15.111 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:15.105+0000 7f67a6453700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f67a0103060 0x7f67a00fe6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:17:15.112 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:15.105+0000 7f67a6453700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f67a0105980 0x7f67a00fec30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:17:15.112 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:15.105+0000 7f67a6453700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67a00ff3b0 con 0x7f67a0100740 2023-07-14T19:17:15.112 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:15.105+0000 7f67a6453700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67a00ff520 con 0x7f67a0105980 2023-07-14T19:17:15.113 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:15.105+0000 7f67a6453700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67a019fb20 con 0x7f67a0103060 2023-07-14T19:17:15.113 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:15.105+0000 7f679ffff700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f67a0100740 0x7f67a01000a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:17:15.113 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:15.105+0000 7f679ffff700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f67a0100740 0x7f67a01000a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:33980/0 (socket says 172.21.15.40:33980) 2023-07-14T19:17:15.113 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:15.105+0000 7f679ffff700 1 -- 172.21.15.40:0/291202405 learned_addr learned my addr 172.21.15.40:0/291202405 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-07-14T19:17:15.114 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:15.105+0000 7f679f7fe700 1 --2- 172.21.15.40:0/291202405 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f67a0103060 0x7f67a00fe6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:17:15.114 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:15.105+0000 7f67a49f0700 1 --2- 172.21.15.40:0/291202405 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f67a0105980 0x7f67a00fec30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:17:15.114 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:15.105+0000 7f679ffff700 1 -- 172.21.15.40:0/291202405 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f67a0103060 msgr2=0x7f67a00fe6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:17:15.115 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:15.105+0000 7f679ffff700 1 --2- 172.21.15.40:0/291202405 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f67a0103060 0x7f67a00fe6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:15.115 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:15.105+0000 7f679ffff700 1 -- 172.21.15.40:0/291202405 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f67a0105980 msgr2=0x7f67a00fec30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:17:15.115 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:15.105+0000 7f679ffff700 1 --2- 172.21.15.40:0/291202405 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f67a0105980 0x7f67a00fec30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:15.116 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:15.105+0000 7f679ffff700 1 -- 172.21.15.40:0/291202405 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6798009d00 con 0x7f67a0100740 2023-07-14T19:17:15.117 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:15.106+0000 7f679ffff700 1 --2- 172.21.15.40:0/291202405 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f67a0100740 0x7f67a01000a0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f679000d950 tx=0x7f679000dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:17:15.117 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:15.106+0000 7f679d7fa700 1 -- 172.21.15.40:0/291202405 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f679000f890 con 0x7f67a0100740 2023-07-14T19:17:15.117 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:15.106+0000 7f679d7fa700 1 -- 172.21.15.40:0/291202405 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f6790010460 con 0x7f67a0100740 2023-07-14T19:17:15.117 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:15.106+0000 7f679d7fa700 1 -- 172.21.15.40:0/291202405 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f679000e610 con 0x7f67a0100740 2023-07-14T19:17:15.118 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:15.106+0000 7f67a6453700 1 -- 172.21.15.40:0/291202405 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f67a019fe00 con 0x7f67a0100740 2023-07-14T19:17:15.118 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:15.106+0000 7f67a6453700 1 -- 172.21.15.40:0/291202405 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f67a01a01c0 con 0x7f67a0100740 2023-07-14T19:17:15.120 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:15.107+0000 7f67a6453700 1 -- 172.21.15.40:0/291202405 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f67a00fbd50 con 0x7f67a0100740 2023-07-14T19:17:15.120 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:15.108+0000 7f679d7fa700 1 -- 172.21.15.40:0/291202405 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7f67900108d0 con 0x7f67a0100740 2023-07-14T19:17:15.121 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:15.108+0000 7f679d7fa700 1 --2- 172.21.15.40:0/291202405 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f67880a11e0 0x7f67880a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:17:15.121 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:15.109+0000 7f679f7fe700 1 --2- 172.21.15.40:0/291202405 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f67880a11e0 0x7f67880a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:17:15.121 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:15.109+0000 7f679d7fa700 1 -- 172.21.15.40:0/291202405 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 8269+0+0 (secure 0 0 0) 0x7f67900cc870 con 0x7f67a0100740 2023-07-14T19:17:15.121 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:15.109+0000 7f679f7fe700 1 --2- 172.21.15.40:0/291202405 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f67880a11e0 0x7f67880a3690 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f6794005950 tx=0x7f679400b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:17:15.122 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:15.112+0000 7f679d7fa700 1 -- 172.21.15.40:0/291202405 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f6790095de0 con 0x7f67a0100740 2023-07-14T19:17:15.312 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:15.310+0000 7f67a6453700 1 -- 172.21.15.40:0/291202405 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd dump"} v 0) v1 -- 0x7f67a002c8e0 con 0x7f67a0100740 2023-07-14T19:17:15.312 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:15.311+0000 7f679d7fa700 1 -- 172.21.15.40:0/291202405 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump"}]=0 v83) v1 ==== 56+0+5335 (secure 0 0 0) 0x7f6790099400 con 0x7f67a0100740 2023-07-14T19:17:15.313 INFO:teuthology.orchestra.run.smithi040.stdout:epoch 83 2023-07-14T19:17:15.313 INFO:teuthology.orchestra.run.smithi040.stdout:fsid adc52f04-2279-11ee-9b33-001a4aab830c 2023-07-14T19:17:15.313 INFO:teuthology.orchestra.run.smithi040.stdout:created 2023-07-14T19:08:56.893730+0000 2023-07-14T19:17:15.313 INFO:teuthology.orchestra.run.smithi040.stdout:modified 2023-07-14T19:16:52.763512+0000 2023-07-14T19:17:15.314 INFO:teuthology.orchestra.run.smithi040.stdout:flags sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit 2023-07-14T19:17:15.314 INFO:teuthology.orchestra.run.smithi040.stdout:crush_version 26 2023-07-14T19:17:15.314 INFO:teuthology.orchestra.run.smithi040.stdout:full_ratio 0.95 2023-07-14T19:17:15.314 INFO:teuthology.orchestra.run.smithi040.stdout:backfillfull_ratio 0.9 2023-07-14T19:17:15.314 INFO:teuthology.orchestra.run.smithi040.stdout:nearfull_ratio 0.85 2023-07-14T19:17:15.314 INFO:teuthology.orchestra.run.smithi040.stdout:require_min_compat_client luminous 2023-07-14T19:17:15.314 INFO:teuthology.orchestra.run.smithi040.stdout:min_compat_client luminous 2023-07-14T19:17:15.315 INFO:teuthology.orchestra.run.smithi040.stdout:require_osd_release reef 2023-07-14T19:17:15.315 INFO:teuthology.orchestra.run.smithi040.stdout:stretch_mode_enabled false 2023-07-14T19:17:15.315 INFO:teuthology.orchestra.run.smithi040.stdout:pool 1 '.mgr' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode off last_change 21 flags hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr read_balance_score 12.00 2023-07-14T19:17:15.315 INFO:teuthology.orchestra.run.smithi040.stdout:pool 2 'cephfs_metadata' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 80 flags hashpspool stripe_width 0 pg_autoscale_bias 4 pg_num_min 16 recovery_priority 5 application cephfs read_balance_score 1.88 2023-07-14T19:17:15.315 INFO:teuthology.orchestra.run.smithi040.stdout:pool 3 'cephfs_data' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 64 pgp_num 64 autoscale_mode off last_change 81 flags hashpspool stripe_width 0 pg_num_min 64 target_size_ratio 0.9 application cephfs read_balance_score 1.50 2023-07-14T19:17:15.316 INFO:teuthology.orchestra.run.smithi040.stdout:max_osd 12 2023-07-14T19:17:15.316 INFO:teuthology.orchestra.run.smithi040.stdout:osd.0 up in weight 1 up_from 8 up_thru 82 down_at 0 last_clean_interval [0,0) [v2:172.21.15.40:6802/3599703125,v1:172.21.15.40:6803/3599703125] [v2:172.21.15.40:6804/3599703125,v1:172.21.15.40:6805/3599703125] exists,up 0d79356a-95d2-49ad-857b-2092ae9f7e13 2023-07-14T19:17:15.316 INFO:teuthology.orchestra.run.smithi040.stdout:osd.1 up in weight 1 up_from 13 up_thru 79 down_at 0 last_clean_interval [0,0) [v2:172.21.15.130:6800/2512760702,v1:172.21.15.130:6801/2512760702] [v2:172.21.15.130:6802/2512760702,v1:172.21.15.130:6803/2512760702] exists,up 40f8a3d1-a5e4-473c-b9f2-7600ec22be0d 2023-07-14T19:17:15.316 INFO:teuthology.orchestra.run.smithi040.stdout:osd.2 up in weight 1 up_from 18 up_thru 79 down_at 0 last_clean_interval [0,0) [v2:172.21.15.191:6800/589569135,v1:172.21.15.191:6801/589569135] [v2:172.21.15.191:6802/589569135,v1:172.21.15.191:6803/589569135] exists,up 5a157891-cb7a-4ea6-9239-f48a7c67c003 2023-07-14T19:17:15.316 INFO:teuthology.orchestra.run.smithi040.stdout:osd.3 up in weight 1 up_from 25 up_thru 79 down_at 0 last_clean_interval [0,0) [v2:172.21.15.40:6810/996287860,v1:172.21.15.40:6811/996287860] [v2:172.21.15.40:6812/996287860,v1:172.21.15.40:6813/996287860] exists,up 8020e7dc-ac40-487f-857a-8da40bccb459 2023-07-14T19:17:15.316 INFO:teuthology.orchestra.run.smithi040.stdout:osd.4 up in weight 1 up_from 32 up_thru 82 down_at 0 last_clean_interval [0,0) [v2:172.21.15.130:6808/3507448720,v1:172.21.15.130:6809/3507448720] [v2:172.21.15.130:6810/3507448720,v1:172.21.15.130:6811/3507448720] exists,up c5501b35-f62d-482d-8d22-115c88f0298e 2023-07-14T19:17:15.317 INFO:teuthology.orchestra.run.smithi040.stdout:osd.5 up in weight 1 up_from 37 up_thru 79 down_at 0 last_clean_interval [0,0) [v2:172.21.15.191:6808/1781944663,v1:172.21.15.191:6809/1781944663] [v2:172.21.15.191:6810/1781944663,v1:172.21.15.191:6811/1781944663] exists,up 7770b6d6-5874-493a-986a-84db482cee42 2023-07-14T19:17:15.317 INFO:teuthology.orchestra.run.smithi040.stdout:osd.6 up in weight 1 up_from 45 up_thru 79 down_at 0 last_clean_interval [0,0) [v2:172.21.15.40:6818/693186773,v1:172.21.15.40:6819/693186773] [v2:172.21.15.40:6820/693186773,v1:172.21.15.40:6821/693186773] exists,up ef65ad00-dfee-4300-9961-dfc8573aab06 2023-07-14T19:17:15.317 INFO:teuthology.orchestra.run.smithi040.stdout:osd.7 up in weight 1 up_from 50 up_thru 79 down_at 0 last_clean_interval [0,0) [v2:172.21.15.130:6816/2384019528,v1:172.21.15.130:6817/2384019528] [v2:172.21.15.130:6818/2384019528,v1:172.21.15.130:6819/2384019528] exists,up 1700fbfa-cbfc-4f76-97be-727a3b100740 2023-07-14T19:17:15.317 INFO:teuthology.orchestra.run.smithi040.stdout:osd.8 up in weight 1 up_from 57 up_thru 82 down_at 0 last_clean_interval [0,0) [v2:172.21.15.191:6816/2386818417,v1:172.21.15.191:6817/2386818417] [v2:172.21.15.191:6818/2386818417,v1:172.21.15.191:6819/2386818417] exists,up b76d4033-c1e2-457b-8571-c8a49fe443f0 2023-07-14T19:17:15.317 INFO:teuthology.orchestra.run.smithi040.stdout:osd.9 up in weight 1 up_from 64 up_thru 82 down_at 0 last_clean_interval [0,0) [v2:172.21.15.40:6826/2228380263,v1:172.21.15.40:6827/2228380263] [v2:172.21.15.40:6828/2228380263,v1:172.21.15.40:6829/2228380263] exists,up df31bf2f-e30e-41b6-b2b9-0e26a07a88bd 2023-07-14T19:17:15.318 INFO:teuthology.orchestra.run.smithi040.stdout:osd.10 up in weight 1 up_from 69 up_thru 79 down_at 0 last_clean_interval [0,0) [v2:172.21.15.130:6824/2417067798,v1:172.21.15.130:6825/2417067798] [v2:172.21.15.130:6826/2417067798,v1:172.21.15.130:6827/2417067798] exists,up 4a93d621-49ee-4ad7-9e49-750614faf912 2023-07-14T19:17:15.318 INFO:teuthology.orchestra.run.smithi040.stdout:osd.11 up in weight 1 up_from 76 up_thru 82 down_at 0 last_clean_interval [0,0) [v2:172.21.15.191:6824/2886059446,v1:172.21.15.191:6825/2886059446] [v2:172.21.15.191:6826/2886059446,v1:172.21.15.191:6827/2886059446] exists,up f43a2a6f-82f3-42ac-b77c-4f1b34da6918 2023-07-14T19:17:15.318 INFO:teuthology.orchestra.run.smithi040.stdout:pg_upmap_items 2.a [2,11] 2023-07-14T19:17:15.318 INFO:teuthology.orchestra.run.smithi040.stdout:pg_upmap_items 2.b [8,11] 2023-07-14T19:17:15.318 INFO:teuthology.orchestra.run.smithi040.stdout:pg_upmap_items 2.c [2,4] 2023-07-14T19:17:15.318 INFO:teuthology.orchestra.run.smithi040.stdout:pg_upmap_items 2.1f [8,1] 2023-07-14T19:17:15.318 INFO:teuthology.orchestra.run.smithi040.stdout:pg_upmap_items 3.e [4,7] 2023-07-14T19:17:15.319 INFO:teuthology.orchestra.run.smithi040.stdout:pg_upmap_items 3.17 [4,8] 2023-07-14T19:17:15.319 INFO:teuthology.orchestra.run.smithi040.stdout:pg_upmap_items 3.1a [5,6] 2023-07-14T19:17:15.319 INFO:teuthology.orchestra.run.smithi040.stdout:pg_upmap_items 3.20 [5,9] 2023-07-14T19:17:15.319 INFO:teuthology.orchestra.run.smithi040.stdout:pg_upmap_items 3.27 [3,9] 2023-07-14T19:17:15.319 INFO:teuthology.orchestra.run.smithi040.stdout:pg_upmap_items 3.2c [4,9] 2023-07-14T19:17:15.320 INFO:teuthology.orchestra.run.smithi040.stdout:blocklist 172.21.15.40:6801/3698757264 expires 2023-07-15T19:09:50.732458+0000 2023-07-14T19:17:15.320 INFO:teuthology.orchestra.run.smithi040.stdout:blocklist 172.21.15.40:6800/3698757264 expires 2023-07-15T19:09:50.732458+0000 2023-07-14T19:17:15.320 INFO:teuthology.orchestra.run.smithi040.stdout:blocklist 172.21.15.40:6800/2569015667 expires 2023-07-15T19:09:24.441543+0000 2023-07-14T19:17:15.320 INFO:teuthology.orchestra.run.smithi040.stdout:blocklist 172.21.15.40:0/3653961003 expires 2023-07-15T19:09:24.441543+0000 2023-07-14T19:17:15.320 INFO:teuthology.orchestra.run.smithi040.stdout:blocklist 172.21.15.40:0/3568861680 expires 2023-07-15T19:09:24.441543+0000 2023-07-14T19:17:15.321 INFO:teuthology.orchestra.run.smithi040.stdout:blocklist 172.21.15.40:0/3249773172 expires 2023-07-15T19:09:50.732458+0000 2023-07-14T19:17:15.321 INFO:teuthology.orchestra.run.smithi040.stdout:blocklist 172.21.15.40:0/4084215480 expires 2023-07-15T19:09:24.441543+0000 2023-07-14T19:17:15.321 INFO:teuthology.orchestra.run.smithi040.stdout:blocklist 172.21.15.40:0/1163941760 expires 2023-07-15T19:09:50.732458+0000 2023-07-14T19:17:15.321 INFO:teuthology.orchestra.run.smithi040.stdout:blocklist 172.21.15.40:6801/2569015667 expires 2023-07-15T19:09:24.441543+0000 2023-07-14T19:17:15.321 INFO:teuthology.orchestra.run.smithi040.stdout:blocklist 172.21.15.40:0/2326639960 expires 2023-07-15T19:09:50.732458+0000 2023-07-14T19:17:15.322 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:15.313+0000 7f67a6453700 1 -- 172.21.15.40:0/291202405 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f67880a11e0 msgr2=0x7f67880a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:17:15.322 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:15.313+0000 7f67a6453700 1 --2- 172.21.15.40:0/291202405 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f67880a11e0 0x7f67880a3690 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f6794005950 tx=0x7f679400b410 comp rx=0 tx=0).stop 2023-07-14T19:17:15.322 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:15.313+0000 7f67a6453700 1 -- 172.21.15.40:0/291202405 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f67a0100740 msgr2=0x7f67a01000a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:17:15.323 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:15.313+0000 7f67a6453700 1 --2- 172.21.15.40:0/291202405 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f67a0100740 0x7f67a01000a0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f679000d950 tx=0x7f679000dc60 comp rx=0 tx=0).stop 2023-07-14T19:17:15.323 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:15.313+0000 7f67a6453700 1 -- 172.21.15.40:0/291202405 shutdown_connections 2023-07-14T19:17:15.323 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:15.313+0000 7f67a6453700 1 --2- 172.21.15.40:0/291202405 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f67880a11e0 0x7f67880a3690 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:15.323 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:15.313+0000 7f67a6453700 1 --2- 172.21.15.40:0/291202405 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f67a0105980 0x7f67a00fec30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:15.324 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:15.313+0000 7f67a6453700 1 --2- 172.21.15.40:0/291202405 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f67a0100740 0x7f67a01000a0 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:15.324 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:15.313+0000 7f67a6453700 1 --2- 172.21.15.40:0/291202405 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f67a0103060 0x7f67a00fe6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:15.324 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:15.313+0000 7f67a6453700 1 -- 172.21.15.40:0/291202405 >> 172.21.15.40:0/291202405 conn(0x7f67a00fa1e0 msgr2=0x7f67a01013a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:17:15.324 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:15.313+0000 7f67a6453700 1 -- 172.21.15.40:0/291202405 shutdown_connections 2023-07-14T19:17:15.325 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:15.313+0000 7f67a6453700 1 -- 172.21.15.40:0/291202405 wait complete. 2023-07-14T19:17:15.716 INFO:teuthology.run_tasks:Running task fs.ready... 2023-07-14T19:17:15.731 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-07-14T19:17:15.865 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:15.860+0000 7ff18877a700 1 -- 172.21.15.40:0/3099610684 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff1800feb00 msgr2=0x7ff1800fef10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:17:15.865 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:15.860+0000 7ff18877a700 1 --2- 172.21.15.40:0/3099610684 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff1800feb00 0x7ff1800fef10 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7ff17c009990 tx=0x7ff17c009ca0 comp rx=0 tx=0).stop 2023-07-14T19:17:15.865 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:15.860+0000 7ff18877a700 1 -- 172.21.15.40:0/3099610684 shutdown_connections 2023-07-14T19:17:15.865 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:15.860+0000 7ff18877a700 1 --2- 172.21.15.40:0/3099610684 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff180100680 0x7ff180106f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:15.866 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:15.860+0000 7ff18877a700 1 --2- 172.21.15.40:0/3099610684 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff1800feb00 0x7ff1800fef10 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:15.866 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:15.860+0000 7ff18877a700 1 --2- 172.21.15.40:0/3099610684 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff1800ffcf0 0x7ff180100140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:15.866 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:15.860+0000 7ff18877a700 1 -- 172.21.15.40:0/3099610684 >> 172.21.15.40:0/3099610684 conn(0x7ff1800fa100 msgr2=0x7ff1800fc550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:17:15.866 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:15.860+0000 7ff18877a700 1 -- 172.21.15.40:0/3099610684 shutdown_connections 2023-07-14T19:17:15.866 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:15.860+0000 7ff18877a700 1 -- 172.21.15.40:0/3099610684 wait complete. 2023-07-14T19:17:15.866 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:15.861+0000 7ff18877a700 1 Processor -- start 2023-07-14T19:17:15.866 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:15.861+0000 7ff18877a700 1 -- start start 2023-07-14T19:17:15.866 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:15.861+0000 7ff18877a700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff1800feb00 0x7ff180068370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:17:15.867 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:15.861+0000 7ff18877a700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff1800ffcf0 0x7ff18019ca50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:17:15.867 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:15.861+0000 7ff18877a700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff180100680 0x7ff18019efa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:17:15.867 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:15.861+0000 7ff18877a700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff18019f680 con 0x7ff1800feb00 2023-07-14T19:17:15.867 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:15.861+0000 7ff18877a700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff18019f7f0 con 0x7ff180100680 2023-07-14T19:17:15.867 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:15.861+0000 7ff18877a700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff18019f9b0 con 0x7ff1800ffcf0 2023-07-14T19:17:15.867 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:15.861+0000 7ff186516700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff1800feb00 0x7ff180068370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:17:15.867 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:15.861+0000 7ff186516700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff1800feb00 0x7ff180068370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:33992/0 (socket says 172.21.15.40:33992) 2023-07-14T19:17:15.868 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:15.861+0000 7ff186516700 1 -- 172.21.15.40:0/491575093 learned_addr learned my addr 172.21.15.40:0/491575093 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-07-14T19:17:15.868 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:15.861+0000 7ff186d17700 1 --2- 172.21.15.40:0/491575093 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff180100680 0x7ff18019efa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:17:15.868 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:15.861+0000 7ff185d15700 1 --2- 172.21.15.40:0/491575093 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff1800ffcf0 0x7ff18019ca50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:17:15.868 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:15.861+0000 7ff186516700 1 -- 172.21.15.40:0/491575093 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff1800ffcf0 msgr2=0x7ff18019ca50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:17:15.868 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:15.861+0000 7ff186516700 1 --2- 172.21.15.40:0/491575093 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff1800ffcf0 0x7ff18019ca50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:15.868 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:15.861+0000 7ff186516700 1 -- 172.21.15.40:0/491575093 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff180100680 msgr2=0x7ff18019efa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:17:15.868 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:15.861+0000 7ff186516700 1 --2- 172.21.15.40:0/491575093 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff180100680 0x7ff18019efa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:15.869 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:15.861+0000 7ff186516700 1 -- 172.21.15.40:0/491575093 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff170009520 con 0x7ff1800feb00 2023-07-14T19:17:15.869 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:15.861+0000 7ff186d17700 1 --2- 172.21.15.40:0/491575093 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff180100680 0x7ff18019efa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T19:17:15.869 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:15.862+0000 7ff186516700 1 --2- 172.21.15.40:0/491575093 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff1800feb00 0x7ff180068370 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7ff17c004730 tx=0x7ff17c009510 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:17:15.869 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:15.862+0000 7ff1777fe700 1 -- 172.21.15.40:0/491575093 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff17c01d070 con 0x7ff1800feb00 2023-07-14T19:17:15.869 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:15.862+0000 7ff1777fe700 1 -- 172.21.15.40:0/491575093 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff17c004cd0 con 0x7ff1800feb00 2023-07-14T19:17:15.869 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:15.862+0000 7ff18877a700 1 -- 172.21.15.40:0/491575093 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff17c0095f0 con 0x7ff1800feb00 2023-07-14T19:17:15.869 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:15.862+0000 7ff1777fe700 1 -- 172.21.15.40:0/491575093 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff17c005530 con 0x7ff1800feb00 2023-07-14T19:17:15.869 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:15.862+0000 7ff18877a700 1 -- 172.21.15.40:0/491575093 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff18019fec0 con 0x7ff1800feb00 2023-07-14T19:17:15.870 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:15.863+0000 7ff18877a700 1 -- 172.21.15.40:0/491575093 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff1801060a0 con 0x7ff1800feb00 2023-07-14T19:17:15.870 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:15.864+0000 7ff1777fe700 1 -- 172.21.15.40:0/491575093 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7ff17c00fbc0 con 0x7ff1800feb00 2023-07-14T19:17:15.871 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:15.864+0000 7ff1777fe700 1 --2- 172.21.15.40:0/491575093 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7ff16c0a12a0 0x7ff16c0a3750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:17:15.871 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:15.865+0000 7ff1777fe700 1 -- 172.21.15.40:0/491575093 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 8269+0+0 (secure 0 0 0) 0x7ff17c0ce170 con 0x7ff1800feb00 2023-07-14T19:17:15.871 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:15.865+0000 7ff185d15700 1 --2- 172.21.15.40:0/491575093 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7ff16c0a12a0 0x7ff16c0a3750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:17:15.871 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:15.865+0000 7ff185d15700 1 --2- 172.21.15.40:0/491575093 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7ff16c0a12a0 0x7ff16c0a3750 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7ff170009d80 tx=0x7ff170009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:17:15.871 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:15.868+0000 7ff1777fe700 1 -- 172.21.15.40:0/491575093 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7ff17c0977d0 con 0x7ff1800feb00 2023-07-14T19:17:15.971 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:17:15 smithi040 ceph-mon[96518]: pgmap v295: 97 pgs: 97 active+clean; 638 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2023-07-14T19:17:15.971 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:17:15 smithi040 ceph-mon[96518]: from='client.? 172.21.15.40:0/291202405' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2023-07-14T19:17:16.072 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:16.070+0000 7ff18877a700 1 -- 172.21.15.40:0/491575093 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff18002cb30 con 0x7ff1800feb00 2023-07-14T19:17:16.073 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:16.072+0000 7ff1777fe700 1 -- 172.21.15.40:0/491575093 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v83) v1 ==== 74+0+22647 (secure 0 0 0) 0x7ff17c09adf0 con 0x7ff1800feb00 2023-07-14T19:17:16.074 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-07-14T19:17:16.074 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":83,"fsid":"adc52f04-2279-11ee-9b33-001a4aab830c","created":"2023-07-14T19:08:56.893730+0000","modified":"2023-07-14T19:16:52.763512+0000","last_up_change":"2023-07-14T19:15:16.632979+0000","last_in_change":"2023-07-14T19:14:58.961702+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-07-14T19:12:03.271333+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-07-14T19:15:56.073464+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-07-14T19:15:57.018726+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"0d79356a-95d2-49ad-857b-2092ae9f7e13","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":3599703125},{"type":"v1","addr":"172.21.15.40:6803","nonce":3599703125}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":3599703125},{"type":"v1","addr":"172.21.15.40:6805","nonce":3599703125}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":3599703125},{"type":"v1","addr":"172.21.15.40:6809","nonce":3599703125}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":3599703125},{"type":"v1","addr":"172.21.15.40:6807","nonce":3599703125}]},"public_addr":"172.21.15.40:6803/3599703125","cluster_addr":"172.21.15.40:6805/3599703125","heartbeat_back_addr":"172.21.15.40:6809/3599703125","heartbeat_front_addr":"172.21.15.40:6807/3599703125","state":["exists","up"]},{"osd":1,"uuid":"40f8a3d1-a5e4-473c-b9f2-7600ec22be0d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6800","nonce":2512760702},{"type":"v1","addr":"172.21.15.130:6801","nonce":2512760702}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6802","nonce":2512760702},{"type":"v1","addr":"172.21.15.130:6803","nonce":2512760702}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6806","nonce":2512760702},{"type":"v1","addr":"172.21.15.130:6807","nonce":2512760702}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6804","nonce":2512760702},{"type":"v1","addr":"172.21.15.130:6805","nonce":2512760702}]},"public_addr":"172.21.15.130:6801/2512760702","cluster_addr":"172.21.15.130:6803/2512760702","heartbeat_back_addr":"172.21.15.130:6807/2512760702","heartbeat_front_addr":"172.21.15.130:6805/2512760702","state":["exists","up"]},{"osd":2,"uuid":"5a157891-cb7a-4ea6-9239-f48a7c67c003","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6800","nonce":589569135},{"type":"v1","addr":"172.21.15.191:6801","nonce":589569135}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6802","nonce":589569135},{"type":"v1","addr":"172.21.15.191:6803","nonce":589569135}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6806","nonce":589569135},{"type":"v1","addr":"172.21.15.191:6807","nonce":589569135}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6804","nonce":589569135},{"type":"v1","addr":"172.21.15.191:6805","nonce":589569135}]},"public_addr":"172.21.15.191:6801/589569135","cluster_addr":"172.21.15.191:6803/589569135","heartbeat_back_addr":"172.21.15.191:6807/589569135","heartbeat_front_addr":"172.21.15.191:6805/589569135","state":["exists","up"]},{"osd":3,"uuid":"8020e7dc-ac40-487f-857a-8da40bccb459","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":996287860},{"type":"v1","addr":"172.21.15.40:6811","nonce":996287860}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6812","nonce":996287860},{"type":"v1","addr":"172.21.15.40:6813","nonce":996287860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6816","nonce":996287860},{"type":"v1","addr":"172.21.15.40:6817","nonce":996287860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":996287860},{"type":"v1","addr":"172.21.15.40:6815","nonce":996287860}]},"public_addr":"172.21.15.40:6811/996287860","cluster_addr":"172.21.15.40:6813/996287860","heartbeat_back_addr":"172.21.15.40:6817/996287860","heartbeat_front_addr":"172.21.15.40:6815/996287860","state":["exists","up"]},{"osd":4,"uuid":"c5501b35-f62d-482d-8d22-115c88f0298e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6808","nonce":3507448720},{"type":"v1","addr":"172.21.15.130:6809","nonce":3507448720}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6810","nonce":3507448720},{"type":"v1","addr":"172.21.15.130:6811","nonce":3507448720}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6814","nonce":3507448720},{"type":"v1","addr":"172.21.15.130:6815","nonce":3507448720}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6812","nonce":3507448720},{"type":"v1","addr":"172.21.15.130:6813","nonce":3507448720}]},"public_addr":"172.21.15.130:6809/3507448720","cluster_addr":"172.21.15.130:6811/3507448720","heartbeat_back_addr":"172.21.15.130:6815/3507448720","heartbeat_front_addr":"172.21.15.130:6813/3507448720","state":["exists","up"]},{"osd":5,"uuid":"7770b6d6-5874-493a-986a-84db482cee42","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6808","nonce":1781944663},{"type":"v1","addr":"172.21.15.191:6809","nonce":1781944663}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6810","nonce":1781944663},{"type":"v1","addr":"172.21.15.191:6811","nonce":1781944663}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6814","nonce":1781944663},{"type":"v1","addr":"172.21.15.191:6815","nonce":1781944663}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6812","nonce":1781944663},{"type":"v1","addr":"172.21.15.191:6813","nonce":1781944663}]},"public_addr":"172.21.15.191:6809/1781944663","cluster_addr":"172.21.15.191:6811/1781944663","heartbeat_back_addr":"172.21.15.191:6815/1781944663","heartbeat_front_addr":"172.21.15.191:6813/1781944663","state":["exists","up"]},{"osd":6,"uuid":"ef65ad00-dfee-4300-9961-dfc8573aab06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6818","nonce":693186773},{"type":"v1","addr":"172.21.15.40:6819","nonce":693186773}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6820","nonce":693186773},{"type":"v1","addr":"172.21.15.40:6821","nonce":693186773}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6824","nonce":693186773},{"type":"v1","addr":"172.21.15.40:6825","nonce":693186773}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6822","nonce":693186773},{"type":"v1","addr":"172.21.15.40:6823","nonce":693186773}]},"public_addr":"172.21.15.40:6819/693186773","cluster_addr":"172.21.15.40:6821/693186773","heartbeat_back_addr":"172.21.15.40:6825/693186773","heartbeat_front_addr":"172.21.15.40:6823/693186773","state":["exists","up"]},{"osd":7,"uuid":"1700fbfa-cbfc-4f76-97be-727a3b100740","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":50,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6816","nonce":2384019528},{"type":"v1","addr":"172.21.15.130:6817","nonce":2384019528}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6818","nonce":2384019528},{"type":"v1","addr":"172.21.15.130:6819","nonce":2384019528}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6822","nonce":2384019528},{"type":"v1","addr":"172.21.15.130:6823","nonce":2384019528}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6820","nonce":2384019528},{"type":"v1","addr":"172.21.15.130:6821","nonce":2384019528}]},"public_addr":"172.21.15.130:6817/2384019528","cluster_addr":"172.21.15.130:6819/2384019528","heartbeat_back_addr":"172.21.15.130:6823/2384019528","heartbeat_front_addr":"172.21.15.130:6821/2384019528","state":["exists","up"]},{"osd":8,"uuid":"b76d4033-c1e2-457b-8571-c8a49fe443f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":57,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6816","nonce":2386818417},{"type":"v1","addr":"172.21.15.191:6817","nonce":2386818417}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6818","nonce":2386818417},{"type":"v1","addr":"172.21.15.191:6819","nonce":2386818417}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6822","nonce":2386818417},{"type":"v1","addr":"172.21.15.191:6823","nonce":2386818417}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6820","nonce":2386818417},{"type":"v1","addr":"172.21.15.191:6821","nonce":2386818417}]},"public_addr":"172.21.15.191:6817/2386818417","cluster_addr":"172.21.15.191:6819/2386818417","heartbeat_back_addr":"172.21.15.191:6823/2386818417","heartbeat_front_addr":"172.21.15.191:6821/2386818417","state":["exists","up"]},{"osd":9,"uuid":"df31bf2f-e30e-41b6-b2b9-0e26a07a88bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":64,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6826","nonce":2228380263},{"type":"v1","addr":"172.21.15.40:6827","nonce":2228380263}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6828","nonce":2228380263},{"type":"v1","addr":"172.21.15.40:6829","nonce":2228380263}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":2228380263},{"type":"v1","addr":"172.21.15.40:6833","nonce":2228380263}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6830","nonce":2228380263},{"type":"v1","addr":"172.21.15.40:6831","nonce":2228380263}]},"public_addr":"172.21.15.40:6827/2228380263","cluster_addr":"172.21.15.40:6829/2228380263","heartbeat_back_addr":"172.21.15.40:6833/2228380263","heartbeat_front_addr":"172.21.15.40:6831/2228380263","state":["exists","up"]},{"osd":10,"uuid":"4a93d621-49ee-4ad7-9e49-750614faf912","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6824","nonce":2417067798},{"type":"v1","addr":"172.21.15.130:6825","nonce":2417067798}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6826","nonce":2417067798},{"type":"v1","addr":"172.21.15.130:6827","nonce":2417067798}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6830","nonce":2417067798},{"type":"v1","addr":"172.21.15.130:6831","nonce":2417067798}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6828","nonce":2417067798},{"type":"v1","addr":"172.21.15.130:6829","nonce":2417067798}]},"public_addr":"172.21.15.130:6825/2417067798","cluster_addr":"172.21.15.130:6827/2417067798","heartbeat_back_addr":"172.21.15.130:6831/2417067798","heartbeat_front_addr":"172.21.15.130:6829/2417067798","state":["exists","up"]},{"osd":11,"uuid":"f43a2a6f-82f3-42ac-b77c-4f1b34da6918","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":76,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6824","nonce":2886059446},{"type":"v1","addr":"172.21.15.191:6825","nonce":2886059446}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6826","nonce":2886059446},{"type":"v1","addr":"172.21.15.191:6827","nonce":2886059446}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6830","nonce":2886059446},{"type":"v1","addr":"172.21.15.191:6831","nonce":2886059446}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6828","nonce":2886059446},{"type":"v1","addr":"172.21.15.191:6829","nonce":2886059446}]},"public_addr":"172.21.15.191:6825/2886059446","cluster_addr":"172.21.15.191:6827/2886059446","heartbeat_back_addr":"172.21.15.191:6831/2886059446","heartbeat_front_addr":"172.21.15.191:6829/2886059446","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:11:18.320495+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:11:39.901199+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:12:00.869042+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:12:23.292742+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:12:44.587019+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:13:06.494893+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:13:28.378374+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:13:50.033491+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:14:10.937062+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:14:32.358498+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:14:53.666584+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:15:14.837566+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"2.b","mappings":[{"from":8,"to":11}]},{"pgid":"2.c","mappings":[{"from":2,"to":4}]},{"pgid":"2.1f","mappings":[{"from":8,"to":1}]},{"pgid":"3.e","mappings":[{"from":4,"to":7}]},{"pgid":"3.17","mappings":[{"from":4,"to":8}]},{"pgid":"3.1a","mappings":[{"from":5,"to":6}]},{"pgid":"3.20","mappings":[{"from":5,"to":9}]},{"pgid":"3.27","mappings":[{"from":3,"to":9}]},{"pgid":"3.2c","mappings":[{"from":4,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.40:6801/3698757264":"2023-07-15T19:09:50.732458+0000","172.21.15.40:6800/3698757264":"2023-07-15T19:09:50.732458+0000","172.21.15.40:6800/2569015667":"2023-07-15T19:09:24.441543+0000","172.21.15.40:0/3653961003":"2023-07-15T19:09:24.441543+0000","172.21.15.40:0/3568861680":"2023-07-15T19:09:24.441543+0000","172.21.15.40:0/3249773172":"2023-07-15T19:09:50.732458+0000","172.21.15.40:0/4084215480":"2023-07-15T19:09:24.441543+0000","172.21.15.40:0/1163941760":"2023-07-15T19:09:50.732458+0000","172.21.15.40:6801/2569015667":"2023-07-15T19:09:24.441543+0000","172.21.15.40:0/2326639960":"2023-07-15T19:09:50.732458+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-07-14T19:17:16.076 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:16.074+0000 7ff18877a700 1 -- 172.21.15.40:0/491575093 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7ff16c0a12a0 msgr2=0x7ff16c0a3750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:17:16.076 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:16.074+0000 7ff18877a700 1 --2- 172.21.15.40:0/491575093 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7ff16c0a12a0 0x7ff16c0a3750 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7ff170009d80 tx=0x7ff170009260 comp rx=0 tx=0).stop 2023-07-14T19:17:16.076 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:16.074+0000 7ff18877a700 1 -- 172.21.15.40:0/491575093 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff1800feb00 msgr2=0x7ff180068370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:17:16.076 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:16.074+0000 7ff18877a700 1 --2- 172.21.15.40:0/491575093 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff1800feb00 0x7ff180068370 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7ff17c004730 tx=0x7ff17c009510 comp rx=0 tx=0).stop 2023-07-14T19:17:16.076 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:16.074+0000 7ff18877a700 1 -- 172.21.15.40:0/491575093 shutdown_connections 2023-07-14T19:17:16.076 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:16.074+0000 7ff18877a700 1 --2- 172.21.15.40:0/491575093 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7ff16c0a12a0 0x7ff16c0a3750 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:16.077 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:16.074+0000 7ff18877a700 1 --2- 172.21.15.40:0/491575093 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff180100680 0x7ff18019efa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:16.077 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:16.074+0000 7ff18877a700 1 --2- 172.21.15.40:0/491575093 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff1800feb00 0x7ff180068370 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:16.077 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:16.074+0000 7ff18877a700 1 --2- 172.21.15.40:0/491575093 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff1800ffcf0 0x7ff18019ca50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:16.077 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:16.074+0000 7ff18877a700 1 -- 172.21.15.40:0/491575093 >> 172.21.15.40:0/491575093 conn(0x7ff1800fa100 msgr2=0x7ff1801031e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:17:16.077 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:16.074+0000 7ff18877a700 1 -- 172.21.15.40:0/491575093 shutdown_connections 2023-07-14T19:17:16.077 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:16.074+0000 7ff18877a700 1 -- 172.21.15.40:0/491575093 wait complete. 2023-07-14T19:17:16.088 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-07-14T19:12:03.271333+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-07-14T19:15:56.073464+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '80', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-07-14T19:15:57.018726+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '81', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-07-14T19:17:16.089 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-07-14T19:17:16.219 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:16.217+0000 7efcf07e5700 1 -- 172.21.15.40:0/833481219 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efce8100690 msgr2=0x7efce8106f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:17:16.220 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:16.217+0000 7efcf07e5700 1 --2- 172.21.15.40:0/833481219 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efce8100690 0x7efce8106f50 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7efce000b0a0 tx=0x7efce000b3b0 comp rx=0 tx=0).stop 2023-07-14T19:17:16.220 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:16.217+0000 7efcf07e5700 1 -- 172.21.15.40:0/833481219 shutdown_connections 2023-07-14T19:17:16.220 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:16.217+0000 7efcf07e5700 1 --2- 172.21.15.40:0/833481219 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7efce80feb10 0x7efce80fef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:16.220 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:16.217+0000 7efcf07e5700 1 --2- 172.21.15.40:0/833481219 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efce8100690 0x7efce8106f50 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:16.220 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:16.217+0000 7efcf07e5700 1 --2- 172.21.15.40:0/833481219 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7efce80ffd00 0x7efce8100150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:16.220 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:16.217+0000 7efcf07e5700 1 -- 172.21.15.40:0/833481219 >> 172.21.15.40:0/833481219 conn(0x7efce80fa130 msgr2=0x7efce80fc560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:17:16.220 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:16.218+0000 7efcf07e5700 1 -- 172.21.15.40:0/833481219 shutdown_connections 2023-07-14T19:17:16.221 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:16.218+0000 7efcf07e5700 1 -- 172.21.15.40:0/833481219 wait complete. 2023-07-14T19:17:16.223 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:16.218+0000 7efcf07e5700 1 Processor -- start 2023-07-14T19:17:16.223 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:16.218+0000 7efcf07e5700 1 -- start start 2023-07-14T19:17:16.223 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:16.218+0000 7efcf07e5700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efce80feb10 0x7efce81959d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:17:16.223 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:16.218+0000 7efcf07e5700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7efce80ffd00 0x7efce8195f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:17:16.223 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:16.218+0000 7efcf07e5700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7efce8100690 0x7efce819cf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:17:16.223 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:16.218+0000 7efcf07e5700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efce819d550 con 0x7efce80feb10 2023-07-14T19:17:16.224 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:16.218+0000 7efcf07e5700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efce819d690 con 0x7efce8100690 2023-07-14T19:17:16.224 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:16.218+0000 7efcf07e5700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efce819d820 con 0x7efce80ffd00 2023-07-14T19:17:16.224 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:16.218+0000 7efcee581700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efce80feb10 0x7efce81959d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:17:16.224 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:16.218+0000 7efcee581700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efce80feb10 0x7efce81959d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:34006/0 (socket says 172.21.15.40:34006) 2023-07-14T19:17:16.224 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:16.218+0000 7efcee581700 1 -- 172.21.15.40:0/1105510431 learned_addr learned my addr 172.21.15.40:0/1105510431 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-07-14T19:17:16.224 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:16.219+0000 7efceed82700 1 --2- 172.21.15.40:0/1105510431 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7efce8100690 0x7efce819cf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:17:16.224 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:16.219+0000 7efcedd80700 1 --2- 172.21.15.40:0/1105510431 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7efce80ffd00 0x7efce8195f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:17:16.225 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:16.219+0000 7efcee581700 1 -- 172.21.15.40:0/1105510431 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7efce80ffd00 msgr2=0x7efce8195f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:17:16.225 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:16.219+0000 7efcee581700 1 --2- 172.21.15.40:0/1105510431 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7efce80ffd00 0x7efce8195f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:16.225 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:16.219+0000 7efcee581700 1 -- 172.21.15.40:0/1105510431 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7efce8100690 msgr2=0x7efce819cf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:17:16.225 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:16.219+0000 7efcee581700 1 --2- 172.21.15.40:0/1105510431 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7efce8100690 0x7efce819cf80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:16.225 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:16.219+0000 7efcee581700 1 -- 172.21.15.40:0/1105510431 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efce0009d00 con 0x7efce80feb10 2023-07-14T19:17:16.225 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:16.219+0000 7efceed82700 1 --2- 172.21.15.40:0/1105510431 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7efce8100690 0x7efce819cf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T19:17:16.225 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:16.219+0000 7efcee581700 1 --2- 172.21.15.40:0/1105510431 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efce80feb10 0x7efce81959d0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7efcd800d980 tx=0x7efcd800dd40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:17:16.226 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:16.219+0000 7efcdf7fe700 1 -- 172.21.15.40:0/1105510431 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcd800de60 con 0x7efce80feb10 2023-07-14T19:17:16.226 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:16.219+0000 7efcdf7fe700 1 -- 172.21.15.40:0/1105510431 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7efcd80044e0 con 0x7efce80feb10 2023-07-14T19:17:16.226 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:16.219+0000 7efcf07e5700 1 -- 172.21.15.40:0/1105510431 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efce819db00 con 0x7efce80feb10 2023-07-14T19:17:16.226 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:16.219+0000 7efcdf7fe700 1 -- 172.21.15.40:0/1105510431 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcd8003be0 con 0x7efce80feb10 2023-07-14T19:17:16.226 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:16.219+0000 7efcf07e5700 1 -- 172.21.15.40:0/1105510431 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efce819e060 con 0x7efce80feb10 2023-07-14T19:17:16.227 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:16.220+0000 7efcf07e5700 1 -- 172.21.15.40:0/1105510431 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efce8105f30 con 0x7efce80feb10 2023-07-14T19:17:16.227 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:16.221+0000 7efcdf7fe700 1 -- 172.21.15.40:0/1105510431 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7efcd8009bb0 con 0x7efce80feb10 2023-07-14T19:17:16.228 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:17:15 smithi191 ceph-mon[98311]: pgmap v295: 97 pgs: 97 active+clean; 638 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2023-07-14T19:17:16.228 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:17:15 smithi191 ceph-mon[98311]: from='client.? 172.21.15.40:0/291202405' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2023-07-14T19:17:16.228 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:16.222+0000 7efcdf7fe700 1 --2- 172.21.15.40:0/1105510431 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7efcd40a1260 0x7efcd40a3710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:17:16.229 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:16.222+0000 7efcedd80700 1 --2- 172.21.15.40:0/1105510431 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7efcd40a1260 0x7efcd40a3710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:17:16.229 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:16.222+0000 7efcdf7fe700 1 -- 172.21.15.40:0/1105510431 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 8269+0+0 (secure 0 0 0) 0x7efcd80ccb60 con 0x7efce80feb10 2023-07-14T19:17:16.229 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:16.222+0000 7efcedd80700 1 --2- 172.21.15.40:0/1105510431 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7efcd40a1260 0x7efcd40a3710 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7efce4005ea0 tx=0x7efce4005e30 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:17:16.229 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:16.225+0000 7efcdf7fe700 1 -- 172.21.15.40:0/1105510431 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7efcd80961b0 con 0x7efce80feb10 2023-07-14T19:17:16.242 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:17:15 smithi130 ceph-mon[97832]: pgmap v295: 97 pgs: 97 active+clean; 638 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2023-07-14T19:17:16.242 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:17:15 smithi130 ceph-mon[97832]: from='client.? 172.21.15.40:0/291202405' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2023-07-14T19:17:16.427 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:16.426+0000 7efcf07e5700 1 -- 172.21.15.40:0/1105510431 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7efce802cb90 con 0x7efce80feb10 2023-07-14T19:17:16.428 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:16.426+0000 7efcdf7fe700 1 -- 172.21.15.40:0/1105510431 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v83) v1 ==== 93+0+10 (secure 0 0 0) 0x7efcd8009940 con 0x7efce80feb10 2023-07-14T19:17:16.429 INFO:teuthology.orchestra.run.smithi040.stdout:pg_num: 1 2023-07-14T19:17:16.430 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:16.428+0000 7efcf07e5700 1 -- 172.21.15.40:0/1105510431 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7efcd40a1260 msgr2=0x7efcd40a3710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:17:16.430 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:16.428+0000 7efcf07e5700 1 --2- 172.21.15.40:0/1105510431 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7efcd40a1260 0x7efcd40a3710 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7efce4005ea0 tx=0x7efce4005e30 comp rx=0 tx=0).stop 2023-07-14T19:17:16.430 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:16.428+0000 7efcf07e5700 1 -- 172.21.15.40:0/1105510431 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efce80feb10 msgr2=0x7efce81959d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:17:16.430 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:16.428+0000 7efcf07e5700 1 --2- 172.21.15.40:0/1105510431 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efce80feb10 0x7efce81959d0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7efcd800d980 tx=0x7efcd800dd40 comp rx=0 tx=0).stop 2023-07-14T19:17:16.431 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:16.428+0000 7efcf07e5700 1 -- 172.21.15.40:0/1105510431 shutdown_connections 2023-07-14T19:17:16.431 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:16.428+0000 7efcf07e5700 1 --2- 172.21.15.40:0/1105510431 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7efcd40a1260 0x7efcd40a3710 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:16.431 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:16.428+0000 7efcf07e5700 1 --2- 172.21.15.40:0/1105510431 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7efce8100690 0x7efce819cf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:16.431 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:16.428+0000 7efcf07e5700 1 --2- 172.21.15.40:0/1105510431 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efce80feb10 0x7efce81959d0 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:16.431 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:16.428+0000 7efcf07e5700 1 --2- 172.21.15.40:0/1105510431 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7efce80ffd00 0x7efce8195f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:16.431 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:16.428+0000 7efcf07e5700 1 -- 172.21.15.40:0/1105510431 >> 172.21.15.40:0/1105510431 conn(0x7efce80fa130 msgr2=0x7efce8103070 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:17:16.431 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:16.428+0000 7efcf07e5700 1 -- 172.21.15.40:0/1105510431 shutdown_connections 2023-07-14T19:17:16.432 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:16.428+0000 7efcf07e5700 1 -- 172.21.15.40:0/1105510431 wait complete. 2023-07-14T19:17:16.442 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-07-14T19:17:16.571 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:16.569+0000 7f18df892700 1 -- 172.21.15.40:0/3738530437 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f18d8100690 msgr2=0x7f18d8102a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:17:16.572 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:16.569+0000 7f18df892700 1 --2- 172.21.15.40:0/3738530437 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f18d8100690 0x7f18d8102a70 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f18d4009990 tx=0x7f18d4009ca0 comp rx=0 tx=0).stop 2023-07-14T19:17:16.572 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:16.569+0000 7f18df892700 1 -- 172.21.15.40:0/3738530437 shutdown_connections 2023-07-14T19:17:16.572 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:16.569+0000 7f18df892700 1 --2- 172.21.15.40:0/3738530437 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f18d81058d0 0x7f18d8107d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:16.572 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:16.569+0000 7f18df892700 1 --2- 172.21.15.40:0/3738530437 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f18d8100690 0x7f18d8102a70 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:16.572 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:16.569+0000 7f18df892700 1 --2- 172.21.15.40:0/3738530437 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f18d8102fb0 0x7f18d8105390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:16.572 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:16.569+0000 7f18df892700 1 -- 172.21.15.40:0/3738530437 >> 172.21.15.40:0/3738530437 conn(0x7f18d80fa130 msgr2=0x7f18d80fc560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:17:16.573 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:16.570+0000 7f18df892700 1 -- 172.21.15.40:0/3738530437 shutdown_connections 2023-07-14T19:17:16.573 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:16.570+0000 7f18df892700 1 -- 172.21.15.40:0/3738530437 wait complete. 2023-07-14T19:17:16.574 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:16.570+0000 7f18df892700 1 Processor -- start 2023-07-14T19:17:16.574 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:16.570+0000 7f18df892700 1 -- start start 2023-07-14T19:17:16.575 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:16.570+0000 7f18df892700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f18d8100690 0x7f18d8193770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:17:16.575 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:16.570+0000 7f18df892700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f18d8102fb0 0x7f18d8193cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:17:16.575 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:16.570+0000 7f18df892700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f18d81058d0 0x7f18d819ad20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:17:16.575 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:16.570+0000 7f18df892700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18d819b2f0 con 0x7f18d8100690 2023-07-14T19:17:16.575 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:16.570+0000 7f18df892700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18d819b430 con 0x7f18d81058d0 2023-07-14T19:17:16.575 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:16.570+0000 7f18df892700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18d819b5c0 con 0x7f18d8102fb0 2023-07-14T19:17:16.575 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:16.570+0000 7f18dd62e700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f18d8100690 0x7f18d8193770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:17:16.576 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:16.570+0000 7f18dd62e700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f18d8100690 0x7f18d8193770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:34020/0 (socket says 172.21.15.40:34020) 2023-07-14T19:17:16.576 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:16.570+0000 7f18dd62e700 1 -- 172.21.15.40:0/3928327744 learned_addr learned my addr 172.21.15.40:0/3928327744 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-07-14T19:17:16.576 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:16.571+0000 7f18dde2f700 1 --2- 172.21.15.40:0/3928327744 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f18d81058d0 0x7f18d819ad20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:17:16.576 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:16.571+0000 7f18dce2d700 1 --2- 172.21.15.40:0/3928327744 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f18d8102fb0 0x7f18d8193cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:17:16.576 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:16.571+0000 7f18dd62e700 1 -- 172.21.15.40:0/3928327744 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f18d8102fb0 msgr2=0x7f18d8193cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:17:16.576 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:16.571+0000 7f18dd62e700 1 --2- 172.21.15.40:0/3928327744 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f18d8102fb0 0x7f18d8193cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:16.576 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:16.571+0000 7f18dd62e700 1 -- 172.21.15.40:0/3928327744 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f18d81058d0 msgr2=0x7f18d819ad20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:17:16.577 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:16.571+0000 7f18dd62e700 1 --2- 172.21.15.40:0/3928327744 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f18d81058d0 0x7f18d819ad20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:16.577 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:16.571+0000 7f18dd62e700 1 -- 172.21.15.40:0/3928327744 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18d40095f0 con 0x7f18d8100690 2023-07-14T19:17:16.577 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:16.571+0000 7f18dd62e700 1 --2- 172.21.15.40:0/3928327744 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f18d8100690 0x7f18d8193770 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f18d4000c00 tx=0x7f18d400f710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:17:16.577 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:16.571+0000 7f18ce7fc700 1 -- 172.21.15.40:0/3928327744 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18d401d070 con 0x7f18d8100690 2023-07-14T19:17:16.577 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:16.571+0000 7f18ce7fc700 1 -- 172.21.15.40:0/3928327744 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f18d400fb50 con 0x7f18d8100690 2023-07-14T19:17:16.577 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:16.571+0000 7f18df892700 1 -- 172.21.15.40:0/3928327744 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f18d819b840 con 0x7f18d8100690 2023-07-14T19:17:16.577 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:16.571+0000 7f18ce7fc700 1 -- 172.21.15.40:0/3928327744 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18d40173f0 con 0x7f18d8100690 2023-07-14T19:17:16.577 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:16.571+0000 7f18df892700 1 -- 172.21.15.40:0/3928327744 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f18d819bcf0 con 0x7f18d8100690 2023-07-14T19:17:16.578 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:16.572+0000 7f18df892700 1 -- 172.21.15.40:0/3928327744 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f18d80fbca0 con 0x7f18d8100690 2023-07-14T19:17:16.578 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:16.573+0000 7f18ce7fc700 1 -- 172.21.15.40:0/3928327744 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7f18d400fcc0 con 0x7f18d8100690 2023-07-14T19:17:16.579 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:16.574+0000 7f18ce7fc700 1 --2- 172.21.15.40:0/3928327744 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f18c40a55f0 0x7f18c40a7aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:17:16.579 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:16.574+0000 7f18dce2d700 1 --2- 172.21.15.40:0/3928327744 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f18c40a55f0 0x7f18c40a7aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:17:16.579 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:16.574+0000 7f18ce7fc700 1 -- 172.21.15.40:0/3928327744 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 8269+0+0 (secure 0 0 0) 0x7f18d40cdd00 con 0x7f18d8100690 2023-07-14T19:17:16.579 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:16.574+0000 7f18dce2d700 1 --2- 172.21.15.40:0/3928327744 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f18c40a55f0 0x7f18c40a7aa0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f18c8005950 tx=0x7f18c800b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:17:16.579 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:16.578+0000 7f18ce7fc700 1 -- 172.21.15.40:0/3928327744 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f18d4097300 con 0x7f18d8100690 2023-07-14T19:17:16.780 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:16.778+0000 7f18df892700 1 -- 172.21.15.40:0/3928327744 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f18d802cb00 con 0x7f18d8100690 2023-07-14T19:17:16.780 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:16.778+0000 7f18ce7fc700 1 -- 172.21.15.40:0/3928327744 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v83) v1 ==== 104+0+11 (secure 0 0 0) 0x7f18d4027020 con 0x7f18d8100690 2023-07-14T19:17:16.780 INFO:teuthology.orchestra.run.smithi040.stdout:pg_num: 32 2023-07-14T19:17:16.782 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:16.780+0000 7f18df892700 1 -- 172.21.15.40:0/3928327744 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f18c40a55f0 msgr2=0x7f18c40a7aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:17:16.782 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:16.780+0000 7f18df892700 1 --2- 172.21.15.40:0/3928327744 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f18c40a55f0 0x7f18c40a7aa0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f18c8005950 tx=0x7f18c800b410 comp rx=0 tx=0).stop 2023-07-14T19:17:16.782 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:16.780+0000 7f18df892700 1 -- 172.21.15.40:0/3928327744 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f18d8100690 msgr2=0x7f18d8193770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:17:16.782 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:16.780+0000 7f18df892700 1 --2- 172.21.15.40:0/3928327744 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f18d8100690 0x7f18d8193770 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f18d4000c00 tx=0x7f18d400f710 comp rx=0 tx=0).stop 2023-07-14T19:17:16.783 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:16.780+0000 7f18df892700 1 -- 172.21.15.40:0/3928327744 shutdown_connections 2023-07-14T19:17:16.783 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:16.780+0000 7f18df892700 1 --2- 172.21.15.40:0/3928327744 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f18c40a55f0 0x7f18c40a7aa0 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:16.783 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:16.780+0000 7f18df892700 1 --2- 172.21.15.40:0/3928327744 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f18d81058d0 0x7f18d819ad20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:16.783 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:16.780+0000 7f18df892700 1 --2- 172.21.15.40:0/3928327744 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f18d8100690 0x7f18d8193770 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:16.783 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:16.780+0000 7f18df892700 1 --2- 172.21.15.40:0/3928327744 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f18d8102fb0 0x7f18d8193cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:16.783 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:16.780+0000 7f18df892700 1 -- 172.21.15.40:0/3928327744 >> 172.21.15.40:0/3928327744 conn(0x7f18d80fa130 msgr2=0x7f18d81012f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:17:16.783 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:16.780+0000 7f18df892700 1 -- 172.21.15.40:0/3928327744 shutdown_connections 2023-07-14T19:17:16.784 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:16.780+0000 7f18df892700 1 -- 172.21.15.40:0/3928327744 wait complete. 2023-07-14T19:17:16.794 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-07-14T19:17:16.927 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:16.923+0000 7f43a6f2b700 1 -- 172.21.15.40:0/4198400486 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f43a00ffd00 msgr2=0x7f43a0100150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:17:16.927 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:16.923+0000 7f43a6f2b700 1 --2- 172.21.15.40:0/4198400486 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f43a00ffd00 0x7f43a0100150 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f438c009990 tx=0x7f438c009ca0 comp rx=0 tx=0).stop 2023-07-14T19:17:16.927 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:16.923+0000 7f43a6f2b700 1 -- 172.21.15.40:0/4198400486 shutdown_connections 2023-07-14T19:17:16.927 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:16.923+0000 7f43a6f2b700 1 --2- 172.21.15.40:0/4198400486 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f43a00feb10 0x7f43a00fef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:16.928 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:16.923+0000 7f43a6f2b700 1 --2- 172.21.15.40:0/4198400486 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f43a0100690 0x7f43a0106f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:16.928 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:16.923+0000 7f43a6f2b700 1 --2- 172.21.15.40:0/4198400486 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f43a00ffd00 0x7f43a0100150 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:16.928 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:16.923+0000 7f43a6f2b700 1 -- 172.21.15.40:0/4198400486 >> 172.21.15.40:0/4198400486 conn(0x7f43a00fa130 msgr2=0x7f43a00fc560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:17:16.928 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:16.923+0000 7f43a6f2b700 1 -- 172.21.15.40:0/4198400486 shutdown_connections 2023-07-14T19:17:16.928 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:16.923+0000 7f43a6f2b700 1 -- 172.21.15.40:0/4198400486 wait complete. 2023-07-14T19:17:16.928 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:16.923+0000 7f43a6f2b700 1 Processor -- start 2023-07-14T19:17:16.928 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:16.923+0000 7f43a6f2b700 1 -- start start 2023-07-14T19:17:16.929 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:16.923+0000 7f43a6f2b700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f43a00feb10 0x7f43a0068370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:17:16.929 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:16.923+0000 7f43a6f2b700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f43a00ffd00 0x7f43a019a5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:17:16.929 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:16.923+0000 7f43a6f2b700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f43a0100690 0x7f43a019cc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:17:16.929 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:16.923+0000 7f43a6f2b700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43a019d410 con 0x7f43a00ffd00 2023-07-14T19:17:16.929 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:16.923+0000 7f43a6f2b700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43a019d580 con 0x7f43a00feb10 2023-07-14T19:17:16.929 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:16.923+0000 7f43a6f2b700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43a019d740 con 0x7f43a0100690 2023-07-14T19:17:16.929 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:16.924+0000 7f439ffff700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f43a00ffd00 0x7f43a019a5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:17:16.929 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:16.924+0000 7f439ffff700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f43a00ffd00 0x7f43a019a5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:34036/0 (socket says 172.21.15.40:34036) 2023-07-14T19:17:16.930 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:16.924+0000 7f439ffff700 1 -- 172.21.15.40:0/4043650530 learned_addr learned my addr 172.21.15.40:0/4043650530 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-07-14T19:17:16.930 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:16.924+0000 7f43a54c8700 1 --2- 172.21.15.40:0/4043650530 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f43a0100690 0x7f43a019cc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:17:16.930 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:16.924+0000 7f43a4cc7700 1 --2- 172.21.15.40:0/4043650530 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f43a00feb10 0x7f43a0068370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:17:16.930 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:16.924+0000 7f439ffff700 1 -- 172.21.15.40:0/4043650530 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f43a0100690 msgr2=0x7f43a019cc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:17:16.930 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:16.924+0000 7f439ffff700 1 --2- 172.21.15.40:0/4043650530 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f43a0100690 0x7f43a019cc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:16.930 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:16.924+0000 7f439ffff700 1 -- 172.21.15.40:0/4043650530 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f43a00feb10 msgr2=0x7f43a0068370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:17:16.930 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:16.924+0000 7f439ffff700 1 --2- 172.21.15.40:0/4043650530 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f43a00feb10 0x7f43a0068370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:16.931 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:16.924+0000 7f439ffff700 1 -- 172.21.15.40:0/4043650530 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f438c0095f0 con 0x7f43a00ffd00 2023-07-14T19:17:16.931 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:16.924+0000 7f43a54c8700 1 --2- 172.21.15.40:0/4043650530 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f43a0100690 0x7f43a019cc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T19:17:16.931 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:16.924+0000 7f439ffff700 1 --2- 172.21.15.40:0/4043650530 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f43a00ffd00 0x7f43a019a5e0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f438c000c00 tx=0x7f438c00f510 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:17:16.931 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:16.924+0000 7f439dffb700 1 -- 172.21.15.40:0/4043650530 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f438c01d070 con 0x7f43a00ffd00 2023-07-14T19:17:16.931 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:16.924+0000 7f43a6f2b700 1 -- 172.21.15.40:0/4043650530 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f43a019d9c0 con 0x7f43a00ffd00 2023-07-14T19:17:16.931 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:16.924+0000 7f439dffb700 1 -- 172.21.15.40:0/4043650530 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f438c00fc60 con 0x7f43a00ffd00 2023-07-14T19:17:16.931 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:16.924+0000 7f439dffb700 1 -- 172.21.15.40:0/4043650530 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f438c017650 con 0x7f43a00ffd00 2023-07-14T19:17:16.931 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:16.924+0000 7f43a6f2b700 1 -- 172.21.15.40:0/4043650530 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f43a019dde0 con 0x7f43a00ffd00 2023-07-14T19:17:16.932 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:16.925+0000 7f43a6f2b700 1 -- 172.21.15.40:0/4043650530 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f43a01059f0 con 0x7f43a00ffd00 2023-07-14T19:17:16.932 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:16.927+0000 7f439dffb700 1 -- 172.21.15.40:0/4043650530 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7f438c004830 con 0x7f43a00ffd00 2023-07-14T19:17:16.933 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:16.927+0000 7f439dffb700 1 --2- 172.21.15.40:0/4043650530 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f43900a11e0 0x7f43900a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:17:16.933 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:16.927+0000 7f439dffb700 1 -- 172.21.15.40:0/4043650530 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 8269+0+0 (secure 0 0 0) 0x7f438c0ce5c0 con 0x7f43a00ffd00 2023-07-14T19:17:16.933 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:16.927+0000 7f43a4cc7700 1 --2- 172.21.15.40:0/4043650530 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f43900a11e0 0x7f43900a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:17:16.933 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:16.928+0000 7f43a4cc7700 1 --2- 172.21.15.40:0/4043650530 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f43900a11e0 0x7f43900a3690 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f43a00ffb60 tx=0x7f4394009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:17:16.933 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:16.930+0000 7f439dffb700 1 -- 172.21.15.40:0/4043650530 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f438c097c20 con 0x7f43a00ffd00 2023-07-14T19:17:17.133 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:17.131+0000 7f43a6f2b700 1 -- 172.21.15.40:0/4043650530 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f43a002ca70 con 0x7f43a00ffd00 2023-07-14T19:17:17.134 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:17.132+0000 7f439dffb700 1 -- 172.21.15.40:0/4043650530 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v83) v1 ==== 100+0+11 (secure 0 0 0) 0x7f438c09b240 con 0x7f43a00ffd00 2023-07-14T19:17:17.134 INFO:teuthology.orchestra.run.smithi040.stdout:pg_num: 64 2023-07-14T19:17:17.135 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:17.134+0000 7f43a6f2b700 1 -- 172.21.15.40:0/4043650530 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f43900a11e0 msgr2=0x7f43900a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:17:17.136 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:17.134+0000 7f43a6f2b700 1 --2- 172.21.15.40:0/4043650530 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f43900a11e0 0x7f43900a3690 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f43a00ffb60 tx=0x7f4394009260 comp rx=0 tx=0).stop 2023-07-14T19:17:17.136 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:17.134+0000 7f43a6f2b700 1 -- 172.21.15.40:0/4043650530 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f43a00ffd00 msgr2=0x7f43a019a5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:17:17.136 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:17.134+0000 7f43a6f2b700 1 --2- 172.21.15.40:0/4043650530 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f43a00ffd00 0x7f43a019a5e0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f438c000c00 tx=0x7f438c00f510 comp rx=0 tx=0).stop 2023-07-14T19:17:17.136 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:17.134+0000 7f43a6f2b700 1 -- 172.21.15.40:0/4043650530 shutdown_connections 2023-07-14T19:17:17.136 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:17.134+0000 7f43a6f2b700 1 --2- 172.21.15.40:0/4043650530 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f43900a11e0 0x7f43900a3690 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:17.136 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:17.134+0000 7f43a6f2b700 1 --2- 172.21.15.40:0/4043650530 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f43a0100690 0x7f43a019cc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:17.137 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:17.134+0000 7f43a6f2b700 1 --2- 172.21.15.40:0/4043650530 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f43a00feb10 0x7f43a0068370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:17.137 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:17.134+0000 7f43a6f2b700 1 --2- 172.21.15.40:0/4043650530 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f43a00ffd00 0x7f43a019a5e0 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:17.137 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:17.134+0000 7f43a6f2b700 1 -- 172.21.15.40:0/4043650530 >> 172.21.15.40:0/4043650530 conn(0x7f43a00fa130 msgr2=0x7f43a00fc350 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:17:17.137 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:17.134+0000 7f43a6f2b700 1 -- 172.21.15.40:0/4043650530 shutdown_connections 2023-07-14T19:17:17.137 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:17.134+0000 7f43a6f2b700 1 -- 172.21.15.40:0/4043650530 wait complete. 2023-07-14T19:17:17.147 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:17:16 smithi040 ceph-mon[96518]: from='client.? 172.21.15.40:0/491575093' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T19:17:17.147 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:17:16 smithi040 ceph-mon[96518]: from='client.? 172.21.15.40:0/1105510431' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-07-14T19:17:17.147 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:17:16 smithi040 ceph-mon[96518]: from='client.? 172.21.15.40:0/3928327744' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-07-14T19:17:17.148 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-07-14T19:17:17.222 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:17:16 smithi191 ceph-mon[98311]: from='client.? 172.21.15.40:0/491575093' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T19:17:17.222 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:17:16 smithi191 ceph-mon[98311]: from='client.? 172.21.15.40:0/1105510431' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-07-14T19:17:17.222 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:17:16 smithi191 ceph-mon[98311]: from='client.? 172.21.15.40:0/3928327744' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-07-14T19:17:17.242 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:17:16 smithi130 ceph-mon[97832]: from='client.? 172.21.15.40:0/491575093' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T19:17:17.242 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:17:16 smithi130 ceph-mon[97832]: from='client.? 172.21.15.40:0/1105510431' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-07-14T19:17:17.242 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:17:16 smithi130 ceph-mon[97832]: from='client.? 172.21.15.40:0/3928327744' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-07-14T19:17:17.276 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:17.274+0000 7f8ffdd68700 1 -- 172.21.15.40:0/118736868 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8ff8100680 msgr2=0x7f8ff8106f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:17:17.276 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:17.274+0000 7f8ffdd68700 1 --2- 172.21.15.40:0/118736868 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8ff8100680 0x7f8ff8106f40 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f8ff000b0a0 tx=0x7f8ff000b3b0 comp rx=0 tx=0).stop 2023-07-14T19:17:17.276 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:17.274+0000 7f8ffdd68700 1 -- 172.21.15.40:0/118736868 shutdown_connections 2023-07-14T19:17:17.277 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:17.274+0000 7f8ffdd68700 1 --2- 172.21.15.40:0/118736868 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8ff80feb00 0x7f8ff80fef10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:17.277 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:17.274+0000 7f8ffdd68700 1 --2- 172.21.15.40:0/118736868 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8ff8100680 0x7f8ff8106f40 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:17.277 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:17.274+0000 7f8ffdd68700 1 --2- 172.21.15.40:0/118736868 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8ff80ffcf0 0x7f8ff8100140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:17.277 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:17.274+0000 7f8ffdd68700 1 -- 172.21.15.40:0/118736868 >> 172.21.15.40:0/118736868 conn(0x7f8ff80fa100 msgr2=0x7f8ff80fc550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:17:17.277 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:17.274+0000 7f8ffdd68700 1 -- 172.21.15.40:0/118736868 shutdown_connections 2023-07-14T19:17:17.277 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:17.274+0000 7f8ffdd68700 1 -- 172.21.15.40:0/118736868 wait complete. 2023-07-14T19:17:17.277 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:17.275+0000 7f8ffdd68700 1 Processor -- start 2023-07-14T19:17:17.279 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:17.275+0000 7f8ffdd68700 1 -- start start 2023-07-14T19:17:17.280 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:17.275+0000 7f8ffdd68700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8ff80feb00 0x7f8ff8068310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:17:17.280 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:17.275+0000 7f8ffdd68700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8ff80ffcf0 0x7f8ff819a740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:17:17.280 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:17.275+0000 7f8ffdd68700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8ff8100680 0x7f8ff819cd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:17:17.280 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:17.275+0000 7f8ffdd68700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ff819d520 con 0x7f8ff80feb00 2023-07-14T19:17:17.280 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:17.275+0000 7f8ffdd68700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ff819d690 con 0x7f8ff80ffcf0 2023-07-14T19:17:17.280 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:17.275+0000 7f8ffdd68700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ff819d850 con 0x7f8ff8100680 2023-07-14T19:17:17.281 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:17.275+0000 7f8ff77fe700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8ff80feb00 0x7f8ff8068310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:17:17.281 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:17.275+0000 7f8ff77fe700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8ff80feb00 0x7f8ff8068310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:34048/0 (socket says 172.21.15.40:34048) 2023-07-14T19:17:17.281 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:17.275+0000 7f8ff77fe700 1 -- 172.21.15.40:0/1815731590 learned_addr learned my addr 172.21.15.40:0/1815731590 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-07-14T19:17:17.281 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:17.275+0000 7f8ff6ffd700 1 --2- 172.21.15.40:0/1815731590 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8ff80ffcf0 0x7f8ff819a740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:17:17.281 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:17.275+0000 7f8ff77fe700 1 -- 172.21.15.40:0/1815731590 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8ff8100680 msgr2=0x7f8ff819cd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:17:17.282 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:17.275+0000 7f8ff7fff700 1 --2- 172.21.15.40:0/1815731590 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8ff8100680 0x7f8ff819cd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:17:17.282 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:17.275+0000 7f8ff77fe700 1 --2- 172.21.15.40:0/1815731590 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8ff8100680 0x7f8ff819cd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:17.282 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:17.275+0000 7f8ff77fe700 1 -- 172.21.15.40:0/1815731590 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8ff80ffcf0 msgr2=0x7f8ff819a740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:17:17.282 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:17.275+0000 7f8ff77fe700 1 --2- 172.21.15.40:0/1815731590 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8ff80ffcf0 0x7f8ff819a740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:17.282 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:17.276+0000 7f8ff77fe700 1 -- 172.21.15.40:0/1815731590 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ff0009d00 con 0x7f8ff80feb00 2023-07-14T19:17:17.282 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:17.276+0000 7f8ff6ffd700 1 --2- 172.21.15.40:0/1815731590 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8ff80ffcf0 0x7f8ff819a740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T19:17:17.282 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:17.276+0000 7f8ff7fff700 1 --2- 172.21.15.40:0/1815731590 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8ff8100680 0x7f8ff819cd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-14T19:17:17.282 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:17.276+0000 7f8ff77fe700 1 --2- 172.21.15.40:0/1815731590 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8ff80feb00 0x7f8ff8068310 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f8fe800e970 tx=0x7f8fe800ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:17:17.283 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:17.276+0000 7f8ff4ff9700 1 -- 172.21.15.40:0/1815731590 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fe8009790 con 0x7f8ff80feb00 2023-07-14T19:17:17.283 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:17.276+0000 7f8ff4ff9700 1 -- 172.21.15.40:0/1815731590 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f8fe80044e0 con 0x7f8ff80feb00 2023-07-14T19:17:17.283 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:17.276+0000 7f8ffdd68700 1 -- 172.21.15.40:0/1815731590 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ff819db30 con 0x7f8ff80feb00 2023-07-14T19:17:17.283 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:17.276+0000 7f8ff4ff9700 1 -- 172.21.15.40:0/1815731590 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fe8003e30 con 0x7f8ff80feb00 2023-07-14T19:17:17.283 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:17.276+0000 7f8ffdd68700 1 -- 172.21.15.40:0/1815731590 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ff819e040 con 0x7f8ff80feb00 2023-07-14T19:17:17.284 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:17.277+0000 7f8ffdd68700 1 -- 172.21.15.40:0/1815731590 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8ff8105ff0 con 0x7f8ff80feb00 2023-07-14T19:17:17.284 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:17.278+0000 7f8ff4ff9700 1 -- 172.21.15.40:0/1815731590 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7f8fe8004000 con 0x7f8ff80feb00 2023-07-14T19:17:17.284 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:17.278+0000 7f8ff4ff9700 1 --2- 172.21.15.40:0/1815731590 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f8fe00a1220 0x7f8fe00a36d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:17:17.285 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:17.279+0000 7f8ff4ff9700 1 -- 172.21.15.40:0/1815731590 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 8269+0+0 (secure 0 0 0) 0x7f8fe8014070 con 0x7f8ff80feb00 2023-07-14T19:17:17.285 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:17.279+0000 7f8ff6ffd700 1 --2- 172.21.15.40:0/1815731590 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f8fe00a1220 0x7f8fe00a36d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:17:17.285 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:17.279+0000 7f8ff6ffd700 1 --2- 172.21.15.40:0/1815731590 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f8fe00a1220 0x7f8fe00a36d0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f8fec005950 tx=0x7f8fec0058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:17:17.285 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:17.282+0000 7f8ff4ff9700 1 -- 172.21.15.40:0/1815731590 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f8fe8096530 con 0x7f8ff80feb00 2023-07-14T19:17:17.528 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:17.526+0000 7f8ffdd68700 1 -- 172.21.15.40:0/1815731590 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8ff802cb90 con 0x7f8ff80feb00 2023-07-14T19:17:17.529 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:17.527+0000 7f8ff4ff9700 1 -- 172.21.15.40:0/1815731590 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) v1 ==== 94+0+10877 (secure 0 0 0) 0x7f8fe80ade60 con 0x7f8ff80feb00 2023-07-14T19:17:17.529 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-07-14T19:17:17.529 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14523,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6835/501143718","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6834","nonce":501143718},{"type":"v1","addr":"172.21.15.40:6835","nonce":501143718}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14535,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6837/2634476117","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6836","nonce":2634476117},{"type":"v1","addr":"172.21.15.40:6837","nonce":2634476117}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":26,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2023-07-14T19:15:57.650710+0000","modified":"2023-07-14T19:16:05.624568+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24539,"mds_1":24508,"mds_2":24506,"mds_3":24490,"mds_4":24482},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14586":{"gid":14586,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.40:6839/959937545","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6838","nonce":959937545},{"type":"v1","addr":"172.21.15.40:6839","nonce":959937545}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_14598":{"gid":14598,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.40:6841/1590486077","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6840","nonce":1590486077},{"type":"v1","addr":"172.21.15.40:6841","nonce":1590486077}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24433":{"gid":24433,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.191:6833/31509862","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":31509862},{"type":"v1","addr":"172.21.15.191:6833","nonce":31509862}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24452":{"gid":24452,"name":"k","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.130:6833/3677452688","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6832","nonce":3677452688},{"type":"v1","addr":"172.21.15.130:6833","nonce":3677452688}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24478":{"gid":24478,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.191:6835/3712674634","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":3712674634},{"type":"v1","addr":"172.21.15.191:6835","nonce":3712674634}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24482":{"gid":24482,"name":"e","rank":4,"incarnation":24,"state":"up:active","state_seq":8,"addr":"172.21.15.130:6835/3088490808","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6834","nonce":3088490808},{"type":"v1","addr":"172.21.15.130:6835","nonce":3088490808}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24490":{"gid":24490,"name":"c","rank":3,"incarnation":22,"state":"up:active","state_seq":7,"addr":"172.21.15.191:6837/2406892975","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6836","nonce":2406892975},{"type":"v1","addr":"172.21.15.191:6837","nonce":2406892975}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24506":{"gid":24506,"name":"h","rank":2,"incarnation":20,"state":"up:active","state_seq":6,"addr":"172.21.15.130:6837/2604019369","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6836","nonce":2604019369},{"type":"v1","addr":"172.21.15.130:6837","nonce":2604019369}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24508":{"gid":24508,"name":"i","rank":1,"incarnation":18,"state":"up:active","state_seq":5,"addr":"172.21.15.191:6839/1977507302","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":1977507302},{"type":"v1","addr":"172.21.15.191:6839","nonce":1977507302}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24539":{"gid":24539,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.130:6839/2145479535","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6838","nonce":2145479535},{"type":"v1","addr":"172.21.15.130:6839","nonce":2145479535}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-07-14T19:17:17.531 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:17.529+0000 7f8ffdd68700 1 -- 172.21.15.40:0/1815731590 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f8fe00a1220 msgr2=0x7f8fe00a36d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:17:17.531 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:17.529+0000 7f8ffdd68700 1 --2- 172.21.15.40:0/1815731590 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f8fe00a1220 0x7f8fe00a36d0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f8fec005950 tx=0x7f8fec0058e0 comp rx=0 tx=0).stop 2023-07-14T19:17:17.531 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:17.529+0000 7f8ffdd68700 1 -- 172.21.15.40:0/1815731590 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8ff80feb00 msgr2=0x7f8ff8068310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:17:17.531 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:17.529+0000 7f8ffdd68700 1 --2- 172.21.15.40:0/1815731590 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8ff80feb00 0x7f8ff8068310 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f8fe800e970 tx=0x7f8fe800ed30 comp rx=0 tx=0).stop 2023-07-14T19:17:17.532 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:17.529+0000 7f8ffdd68700 1 -- 172.21.15.40:0/1815731590 shutdown_connections 2023-07-14T19:17:17.532 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:17.529+0000 7f8ffdd68700 1 --2- 172.21.15.40:0/1815731590 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f8fe00a1220 0x7f8fe00a36d0 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:17.532 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:17.529+0000 7f8ffdd68700 1 --2- 172.21.15.40:0/1815731590 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8ff8100680 0x7f8ff819cd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:17.532 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:17.529+0000 7f8ffdd68700 1 --2- 172.21.15.40:0/1815731590 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8ff80feb00 0x7f8ff8068310 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:17.532 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:17.529+0000 7f8ffdd68700 1 --2- 172.21.15.40:0/1815731590 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8ff80ffcf0 0x7f8ff819a740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:17.532 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:17.529+0000 7f8ffdd68700 1 -- 172.21.15.40:0/1815731590 >> 172.21.15.40:0/1815731590 conn(0x7f8ff80fa100 msgr2=0x7f8ff8103130 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:17:17.533 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:17.529+0000 7f8ffdd68700 1 -- 172.21.15.40:0/1815731590 shutdown_connections 2023-07-14T19:17:17.533 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:17.529+0000 7f8ffdd68700 1 -- 172.21.15.40:0/1815731590 wait complete. 2023-07-14T19:17:17.533 INFO:teuthology.orchestra.run.smithi040.stderr:dumped fsmap epoch 26 2023-07-14T19:17:17.544 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-07-14T19:17:17.670 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:17.668+0000 7f1d0ab6e700 1 -- 172.21.15.40:0/1506092448 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1d040ffd30 msgr2=0x7f1d04100180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:17:17.670 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:17.668+0000 7f1d0ab6e700 1 --2- 172.21.15.40:0/1506092448 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1d040ffd30 0x7f1d04100180 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f1cf0009990 tx=0x7f1cf0009ca0 comp rx=0 tx=0).stop 2023-07-14T19:17:17.670 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:17.668+0000 7f1d0ab6e700 1 -- 172.21.15.40:0/1506092448 shutdown_connections 2023-07-14T19:17:17.671 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:17.668+0000 7f1d0ab6e700 1 --2- 172.21.15.40:0/1506092448 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1d041006c0 0x7f1d04106f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:17.671 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:17.668+0000 7f1d0ab6e700 1 --2- 172.21.15.40:0/1506092448 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1d040feb40 0x7f1d040fef50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:17.671 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:17.668+0000 7f1d0ab6e700 1 --2- 172.21.15.40:0/1506092448 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1d040ffd30 0x7f1d04100180 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:17.671 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:17.668+0000 7f1d0ab6e700 1 -- 172.21.15.40:0/1506092448 >> 172.21.15.40:0/1506092448 conn(0x7f1d040fa160 msgr2=0x7f1d040fc590 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:17:17.671 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:17.668+0000 7f1d0ab6e700 1 -- 172.21.15.40:0/1506092448 shutdown_connections 2023-07-14T19:17:17.671 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:17.668+0000 7f1d0ab6e700 1 -- 172.21.15.40:0/1506092448 wait complete. 2023-07-14T19:17:17.673 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:17.669+0000 7f1d0ab6e700 1 Processor -- start 2023-07-14T19:17:17.673 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:17.669+0000 7f1d0ab6e700 1 -- start start 2023-07-14T19:17:17.674 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:17.669+0000 7f1d0ab6e700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1d040feb40 0x7f1d041958e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:17:17.674 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:17.669+0000 7f1d0ab6e700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1d040ffd30 0x7f1d04195e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:17:17.674 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:17.669+0000 7f1d0ab6e700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1d041006c0 0x7f1d0419ce90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:17:17.674 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:17.669+0000 7f1d0ab6e700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d0419d460 con 0x7f1d040feb40 2023-07-14T19:17:17.674 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:17.669+0000 7f1d0ab6e700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d0419d5a0 con 0x7f1d040ffd30 2023-07-14T19:17:17.675 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:17.669+0000 7f1d0ab6e700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d0419d730 con 0x7f1d041006c0 2023-07-14T19:17:17.675 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:17.669+0000 7f1d0890a700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1d040feb40 0x7f1d041958e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:17:17.675 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:17.669+0000 7f1d0890a700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1d040feb40 0x7f1d041958e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:34062/0 (socket says 172.21.15.40:34062) 2023-07-14T19:17:17.675 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:17.669+0000 7f1d0890a700 1 -- 172.21.15.40:0/1869369709 learned_addr learned my addr 172.21.15.40:0/1869369709 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-07-14T19:17:17.675 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:17.669+0000 7f1d03fff700 1 --2- 172.21.15.40:0/1869369709 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1d040ffd30 0x7f1d04195e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:17:17.675 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:17.670+0000 7f1d0910b700 1 --2- 172.21.15.40:0/1869369709 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1d041006c0 0x7f1d0419ce90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:17:17.675 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:17.670+0000 7f1d0890a700 1 -- 172.21.15.40:0/1869369709 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1d041006c0 msgr2=0x7f1d0419ce90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:17:17.676 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:17.670+0000 7f1d0890a700 1 --2- 172.21.15.40:0/1869369709 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1d041006c0 0x7f1d0419ce90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:17.676 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:17.670+0000 7f1d0890a700 1 -- 172.21.15.40:0/1869369709 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1d040ffd30 msgr2=0x7f1d04195e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:17:17.676 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:17.670+0000 7f1d0890a700 1 --2- 172.21.15.40:0/1869369709 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1d040ffd30 0x7f1d04195e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:17.676 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:17.670+0000 7f1d0890a700 1 -- 172.21.15.40:0/1869369709 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1cf00095f0 con 0x7f1d040feb40 2023-07-14T19:17:17.676 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:17.670+0000 7f1d0890a700 1 --2- 172.21.15.40:0/1869369709 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1d040feb40 0x7f1d041958e0 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f1cf800d980 tx=0x7f1cf800dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:17:17.676 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:17.670+0000 7f1d01ffb700 1 -- 172.21.15.40:0/1869369709 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1cf8004020 con 0x7f1d040feb40 2023-07-14T19:17:17.676 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:17.670+0000 7f1d01ffb700 1 -- 172.21.15.40:0/1869369709 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f1cf8009730 con 0x7f1d040feb40 2023-07-14T19:17:17.677 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:17.670+0000 7f1d0ab6e700 1 -- 172.21.15.40:0/1869369709 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1d0419da10 con 0x7f1d040feb40 2023-07-14T19:17:17.677 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:17.670+0000 7f1d01ffb700 1 -- 172.21.15.40:0/1869369709 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1cf8003a80 con 0x7f1d040feb40 2023-07-14T19:17:17.677 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:17.670+0000 7f1d0ab6e700 1 -- 172.21.15.40:0/1869369709 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1d0419df20 con 0x7f1d040feb40 2023-07-14T19:17:17.677 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:17.671+0000 7f1d0ab6e700 1 -- 172.21.15.40:0/1869369709 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1d0410a310 con 0x7f1d040feb40 2023-07-14T19:17:17.678 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:17.672+0000 7f1d01ffb700 1 -- 172.21.15.40:0/1869369709 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7f1cf80098a0 con 0x7f1d040feb40 2023-07-14T19:17:17.678 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:17.673+0000 7f1d01ffb700 1 --2- 172.21.15.40:0/1869369709 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f1cf40a1220 0x7f1cf40a36d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:17:17.678 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:17.673+0000 7f1d01ffb700 1 -- 172.21.15.40:0/1869369709 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 8269+0+0 (secure 0 0 0) 0x7f1cf80cc7f0 con 0x7f1d040feb40 2023-07-14T19:17:17.678 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:17.673+0000 7f1d03fff700 1 --2- 172.21.15.40:0/1869369709 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f1cf40a1220 0x7f1cf40a36d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:17:17.678 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:17.673+0000 7f1d03fff700 1 --2- 172.21.15.40:0/1869369709 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f1cf40a1220 0x7f1cf40a36d0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f1cf000b2e0 tx=0x7f1cf0005960 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:17:17.679 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:17.677+0000 7f1d01ffb700 1 -- 172.21.15.40:0/1869369709 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f1cf8095d60 con 0x7f1d040feb40 2023-07-14T19:17:17.878 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:17.876+0000 7f1d0ab6e700 1 -- 172.21.15.40:0/1869369709 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1d0410a4d0 con 0x7f1d040feb40 2023-07-14T19:17:17.879 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:17.878+0000 7f1d01ffb700 1 -- 172.21.15.40:0/1869369709 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v83) v1 ==== 74+0+22647 (secure 0 0 0) 0x7f1cf8099380 con 0x7f1d040feb40 2023-07-14T19:17:17.880 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-07-14T19:17:17.880 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":83,"fsid":"adc52f04-2279-11ee-9b33-001a4aab830c","created":"2023-07-14T19:08:56.893730+0000","modified":"2023-07-14T19:16:52.763512+0000","last_up_change":"2023-07-14T19:15:16.632979+0000","last_in_change":"2023-07-14T19:14:58.961702+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-07-14T19:12:03.271333+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-07-14T19:15:56.073464+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-07-14T19:15:57.018726+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"0d79356a-95d2-49ad-857b-2092ae9f7e13","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":3599703125},{"type":"v1","addr":"172.21.15.40:6803","nonce":3599703125}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":3599703125},{"type":"v1","addr":"172.21.15.40:6805","nonce":3599703125}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":3599703125},{"type":"v1","addr":"172.21.15.40:6809","nonce":3599703125}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":3599703125},{"type":"v1","addr":"172.21.15.40:6807","nonce":3599703125}]},"public_addr":"172.21.15.40:6803/3599703125","cluster_addr":"172.21.15.40:6805/3599703125","heartbeat_back_addr":"172.21.15.40:6809/3599703125","heartbeat_front_addr":"172.21.15.40:6807/3599703125","state":["exists","up"]},{"osd":1,"uuid":"40f8a3d1-a5e4-473c-b9f2-7600ec22be0d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6800","nonce":2512760702},{"type":"v1","addr":"172.21.15.130:6801","nonce":2512760702}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6802","nonce":2512760702},{"type":"v1","addr":"172.21.15.130:6803","nonce":2512760702}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6806","nonce":2512760702},{"type":"v1","addr":"172.21.15.130:6807","nonce":2512760702}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6804","nonce":2512760702},{"type":"v1","addr":"172.21.15.130:6805","nonce":2512760702}]},"public_addr":"172.21.15.130:6801/2512760702","cluster_addr":"172.21.15.130:6803/2512760702","heartbeat_back_addr":"172.21.15.130:6807/2512760702","heartbeat_front_addr":"172.21.15.130:6805/2512760702","state":["exists","up"]},{"osd":2,"uuid":"5a157891-cb7a-4ea6-9239-f48a7c67c003","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6800","nonce":589569135},{"type":"v1","addr":"172.21.15.191:6801","nonce":589569135}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6802","nonce":589569135},{"type":"v1","addr":"172.21.15.191:6803","nonce":589569135}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6806","nonce":589569135},{"type":"v1","addr":"172.21.15.191:6807","nonce":589569135}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6804","nonce":589569135},{"type":"v1","addr":"172.21.15.191:6805","nonce":589569135}]},"public_addr":"172.21.15.191:6801/589569135","cluster_addr":"172.21.15.191:6803/589569135","heartbeat_back_addr":"172.21.15.191:6807/589569135","heartbeat_front_addr":"172.21.15.191:6805/589569135","state":["exists","up"]},{"osd":3,"uuid":"8020e7dc-ac40-487f-857a-8da40bccb459","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":996287860},{"type":"v1","addr":"172.21.15.40:6811","nonce":996287860}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6812","nonce":996287860},{"type":"v1","addr":"172.21.15.40:6813","nonce":996287860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6816","nonce":996287860},{"type":"v1","addr":"172.21.15.40:6817","nonce":996287860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":996287860},{"type":"v1","addr":"172.21.15.40:6815","nonce":996287860}]},"public_addr":"172.21.15.40:6811/996287860","cluster_addr":"172.21.15.40:6813/996287860","heartbeat_back_addr":"172.21.15.40:6817/996287860","heartbeat_front_addr":"172.21.15.40:6815/996287860","state":["exists","up"]},{"osd":4,"uuid":"c5501b35-f62d-482d-8d22-115c88f0298e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6808","nonce":3507448720},{"type":"v1","addr":"172.21.15.130:6809","nonce":3507448720}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6810","nonce":3507448720},{"type":"v1","addr":"172.21.15.130:6811","nonce":3507448720}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6814","nonce":3507448720},{"type":"v1","addr":"172.21.15.130:6815","nonce":3507448720}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6812","nonce":3507448720},{"type":"v1","addr":"172.21.15.130:6813","nonce":3507448720}]},"public_addr":"172.21.15.130:6809/3507448720","cluster_addr":"172.21.15.130:6811/3507448720","heartbeat_back_addr":"172.21.15.130:6815/3507448720","heartbeat_front_addr":"172.21.15.130:6813/3507448720","state":["exists","up"]},{"osd":5,"uuid":"7770b6d6-5874-493a-986a-84db482cee42","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6808","nonce":1781944663},{"type":"v1","addr":"172.21.15.191:6809","nonce":1781944663}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6810","nonce":1781944663},{"type":"v1","addr":"172.21.15.191:6811","nonce":1781944663}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6814","nonce":1781944663},{"type":"v1","addr":"172.21.15.191:6815","nonce":1781944663}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6812","nonce":1781944663},{"type":"v1","addr":"172.21.15.191:6813","nonce":1781944663}]},"public_addr":"172.21.15.191:6809/1781944663","cluster_addr":"172.21.15.191:6811/1781944663","heartbeat_back_addr":"172.21.15.191:6815/1781944663","heartbeat_front_addr":"172.21.15.191:6813/1781944663","state":["exists","up"]},{"osd":6,"uuid":"ef65ad00-dfee-4300-9961-dfc8573aab06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6818","nonce":693186773},{"type":"v1","addr":"172.21.15.40:6819","nonce":693186773}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6820","nonce":693186773},{"type":"v1","addr":"172.21.15.40:6821","nonce":693186773}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6824","nonce":693186773},{"type":"v1","addr":"172.21.15.40:6825","nonce":693186773}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6822","nonce":693186773},{"type":"v1","addr":"172.21.15.40:6823","nonce":693186773}]},"public_addr":"172.21.15.40:6819/693186773","cluster_addr":"172.21.15.40:6821/693186773","heartbeat_back_addr":"172.21.15.40:6825/693186773","heartbeat_front_addr":"172.21.15.40:6823/693186773","state":["exists","up"]},{"osd":7,"uuid":"1700fbfa-cbfc-4f76-97be-727a3b100740","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":50,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6816","nonce":2384019528},{"type":"v1","addr":"172.21.15.130:6817","nonce":2384019528}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6818","nonce":2384019528},{"type":"v1","addr":"172.21.15.130:6819","nonce":2384019528}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6822","nonce":2384019528},{"type":"v1","addr":"172.21.15.130:6823","nonce":2384019528}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6820","nonce":2384019528},{"type":"v1","addr":"172.21.15.130:6821","nonce":2384019528}]},"public_addr":"172.21.15.130:6817/2384019528","cluster_addr":"172.21.15.130:6819/2384019528","heartbeat_back_addr":"172.21.15.130:6823/2384019528","heartbeat_front_addr":"172.21.15.130:6821/2384019528","state":["exists","up"]},{"osd":8,"uuid":"b76d4033-c1e2-457b-8571-c8a49fe443f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":57,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6816","nonce":2386818417},{"type":"v1","addr":"172.21.15.191:6817","nonce":2386818417}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6818","nonce":2386818417},{"type":"v1","addr":"172.21.15.191:6819","nonce":2386818417}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6822","nonce":2386818417},{"type":"v1","addr":"172.21.15.191:6823","nonce":2386818417}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6820","nonce":2386818417},{"type":"v1","addr":"172.21.15.191:6821","nonce":2386818417}]},"public_addr":"172.21.15.191:6817/2386818417","cluster_addr":"172.21.15.191:6819/2386818417","heartbeat_back_addr":"172.21.15.191:6823/2386818417","heartbeat_front_addr":"172.21.15.191:6821/2386818417","state":["exists","up"]},{"osd":9,"uuid":"df31bf2f-e30e-41b6-b2b9-0e26a07a88bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":64,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6826","nonce":2228380263},{"type":"v1","addr":"172.21.15.40:6827","nonce":2228380263}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6828","nonce":2228380263},{"type":"v1","addr":"172.21.15.40:6829","nonce":2228380263}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":2228380263},{"type":"v1","addr":"172.21.15.40:6833","nonce":2228380263}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6830","nonce":2228380263},{"type":"v1","addr":"172.21.15.40:6831","nonce":2228380263}]},"public_addr":"172.21.15.40:6827/2228380263","cluster_addr":"172.21.15.40:6829/2228380263","heartbeat_back_addr":"172.21.15.40:6833/2228380263","heartbeat_front_addr":"172.21.15.40:6831/2228380263","state":["exists","up"]},{"osd":10,"uuid":"4a93d621-49ee-4ad7-9e49-750614faf912","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6824","nonce":2417067798},{"type":"v1","addr":"172.21.15.130:6825","nonce":2417067798}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6826","nonce":2417067798},{"type":"v1","addr":"172.21.15.130:6827","nonce":2417067798}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6830","nonce":2417067798},{"type":"v1","addr":"172.21.15.130:6831","nonce":2417067798}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6828","nonce":2417067798},{"type":"v1","addr":"172.21.15.130:6829","nonce":2417067798}]},"public_addr":"172.21.15.130:6825/2417067798","cluster_addr":"172.21.15.130:6827/2417067798","heartbeat_back_addr":"172.21.15.130:6831/2417067798","heartbeat_front_addr":"172.21.15.130:6829/2417067798","state":["exists","up"]},{"osd":11,"uuid":"f43a2a6f-82f3-42ac-b77c-4f1b34da6918","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":76,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6824","nonce":2886059446},{"type":"v1","addr":"172.21.15.191:6825","nonce":2886059446}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6826","nonce":2886059446},{"type":"v1","addr":"172.21.15.191:6827","nonce":2886059446}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6830","nonce":2886059446},{"type":"v1","addr":"172.21.15.191:6831","nonce":2886059446}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6828","nonce":2886059446},{"type":"v1","addr":"172.21.15.191:6829","nonce":2886059446}]},"public_addr":"172.21.15.191:6825/2886059446","cluster_addr":"172.21.15.191:6827/2886059446","heartbeat_back_addr":"172.21.15.191:6831/2886059446","heartbeat_front_addr":"172.21.15.191:6829/2886059446","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:11:18.320495+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:11:39.901199+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:12:00.869042+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:12:23.292742+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:12:44.587019+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:13:06.494893+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:13:28.378374+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:13:50.033491+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:14:10.937062+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:14:32.358498+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:14:53.666584+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:15:14.837566+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"2.b","mappings":[{"from":8,"to":11}]},{"pgid":"2.c","mappings":[{"from":2,"to":4}]},{"pgid":"2.1f","mappings":[{"from":8,"to":1}]},{"pgid":"3.e","mappings":[{"from":4,"to":7}]},{"pgid":"3.17","mappings":[{"from":4,"to":8}]},{"pgid":"3.1a","mappings":[{"from":5,"to":6}]},{"pgid":"3.20","mappings":[{"from":5,"to":9}]},{"pgid":"3.27","mappings":[{"from":3,"to":9}]},{"pgid":"3.2c","mappings":[{"from":4,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.40:6801/3698757264":"2023-07-15T19:09:50.732458+0000","172.21.15.40:6800/3698757264":"2023-07-15T19:09:50.732458+0000","172.21.15.40:6800/2569015667":"2023-07-15T19:09:24.441543+0000","172.21.15.40:0/3653961003":"2023-07-15T19:09:24.441543+0000","172.21.15.40:0/3568861680":"2023-07-15T19:09:24.441543+0000","172.21.15.40:0/3249773172":"2023-07-15T19:09:50.732458+0000","172.21.15.40:0/4084215480":"2023-07-15T19:09:24.441543+0000","172.21.15.40:0/1163941760":"2023-07-15T19:09:50.732458+0000","172.21.15.40:6801/2569015667":"2023-07-15T19:09:24.441543+0000","172.21.15.40:0/2326639960":"2023-07-15T19:09:50.732458+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-07-14T19:17:17.881 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:17.880+0000 7f1d0ab6e700 1 -- 172.21.15.40:0/1869369709 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f1cf40a1220 msgr2=0x7f1cf40a36d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:17:17.882 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:17.880+0000 7f1d0ab6e700 1 --2- 172.21.15.40:0/1869369709 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f1cf40a1220 0x7f1cf40a36d0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f1cf000b2e0 tx=0x7f1cf0005960 comp rx=0 tx=0).stop 2023-07-14T19:17:17.882 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:17.880+0000 7f1d0ab6e700 1 -- 172.21.15.40:0/1869369709 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1d040feb40 msgr2=0x7f1d041958e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:17:17.882 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:17.880+0000 7f1d0ab6e700 1 --2- 172.21.15.40:0/1869369709 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1d040feb40 0x7f1d041958e0 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f1cf800d980 tx=0x7f1cf800dc90 comp rx=0 tx=0).stop 2023-07-14T19:17:17.882 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:17.880+0000 7f1d0ab6e700 1 -- 172.21.15.40:0/1869369709 shutdown_connections 2023-07-14T19:17:17.882 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:17.880+0000 7f1d0ab6e700 1 --2- 172.21.15.40:0/1869369709 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f1cf40a1220 0x7f1cf40a36d0 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:17.882 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:17.880+0000 7f1d0ab6e700 1 --2- 172.21.15.40:0/1869369709 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1d041006c0 0x7f1d0419ce90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:17.883 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:17.880+0000 7f1d0ab6e700 1 --2- 172.21.15.40:0/1869369709 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1d040feb40 0x7f1d041958e0 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:17.883 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:17.880+0000 7f1d0ab6e700 1 --2- 172.21.15.40:0/1869369709 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1d040ffd30 0x7f1d04195e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:17.883 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:17.880+0000 7f1d0ab6e700 1 -- 172.21.15.40:0/1869369709 >> 172.21.15.40:0/1869369709 conn(0x7f1d040fa160 msgr2=0x7f1d040fa930 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:17:17.883 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:17.880+0000 7f1d0ab6e700 1 -- 172.21.15.40:0/1869369709 shutdown_connections 2023-07-14T19:17:17.883 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:17.880+0000 7f1d0ab6e700 1 -- 172.21.15.40:0/1869369709 wait complete. 2023-07-14T19:17:17.895 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-07-14T19:12:03.271333+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-07-14T19:15:56.073464+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '80', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-07-14T19:15:57.018726+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '81', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-07-14T19:17:17.896 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-07-14T19:17:18.024 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:18.022+0000 7fe75a179700 1 -- 172.21.15.40:0/1071778366 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe7540ffd00 msgr2=0x7fe754100150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:17:18.024 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:18.022+0000 7fe75a179700 1 --2- 172.21.15.40:0/1071778366 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe7540ffd00 0x7fe754100150 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7fe748009990 tx=0x7fe748009ca0 comp rx=0 tx=0).stop 2023-07-14T19:17:18.024 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:18.022+0000 7fe75a179700 1 -- 172.21.15.40:0/1071778366 shutdown_connections 2023-07-14T19:17:18.025 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:18.022+0000 7fe75a179700 1 --2- 172.21.15.40:0/1071778366 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe7540feb10 0x7fe7540fef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:18.025 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:18.022+0000 7fe75a179700 1 --2- 172.21.15.40:0/1071778366 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe754100690 0x7fe754106f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:18.025 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:18.022+0000 7fe75a179700 1 --2- 172.21.15.40:0/1071778366 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe7540ffd00 0x7fe754100150 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:18.025 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:18.022+0000 7fe75a179700 1 -- 172.21.15.40:0/1071778366 >> 172.21.15.40:0/1071778366 conn(0x7fe7540fa130 msgr2=0x7fe7540fc560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:17:18.025 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:18.022+0000 7fe75a179700 1 -- 172.21.15.40:0/1071778366 shutdown_connections 2023-07-14T19:17:18.025 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:18.022+0000 7fe75a179700 1 -- 172.21.15.40:0/1071778366 wait complete. 2023-07-14T19:17:18.027 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:18.023+0000 7fe75a179700 1 Processor -- start 2023-07-14T19:17:18.027 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:18.023+0000 7fe75a179700 1 -- start start 2023-07-14T19:17:18.027 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:18.023+0000 7fe75a179700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe7540feb10 0x7fe754068330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:17:18.028 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:18.023+0000 7fe75a179700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe7540ffd00 0x7fe75419a670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:17:18.028 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:18.023+0000 7fe75a179700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe754100690 0x7fe75419ccc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:17:18.028 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:18.023+0000 7fe75a179700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe75419d490 con 0x7fe754100690 2023-07-14T19:17:18.028 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:18.023+0000 7fe75a179700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe75419d600 con 0x7fe7540feb10 2023-07-14T19:17:18.028 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:18.023+0000 7fe75a179700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe75419d7c0 con 0x7fe7540ffd00 2023-07-14T19:17:18.028 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:18.023+0000 7fe753fff700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe754100690 0x7fe75419ccc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:17:18.028 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:18.023+0000 7fe753fff700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe754100690 0x7fe75419ccc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:34080/0 (socket says 172.21.15.40:34080) 2023-07-14T19:17:18.028 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:18.023+0000 7fe753fff700 1 -- 172.21.15.40:0/2687442009 learned_addr learned my addr 172.21.15.40:0/2687442009 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-07-14T19:17:18.029 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:18.023+0000 7fe7537fe700 1 --2- 172.21.15.40:0/2687442009 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe7540feb10 0x7fe754068330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:17:18.029 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:18.023+0000 7fe752ffd700 1 --2- 172.21.15.40:0/2687442009 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe7540ffd00 0x7fe75419a670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:17:18.029 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:18.024+0000 7fe753fff700 1 -- 172.21.15.40:0/2687442009 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe7540ffd00 msgr2=0x7fe75419a670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:17:18.029 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:18.024+0000 7fe753fff700 1 --2- 172.21.15.40:0/2687442009 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe7540ffd00 0x7fe75419a670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:18.029 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:18.024+0000 7fe753fff700 1 -- 172.21.15.40:0/2687442009 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe7540feb10 msgr2=0x7fe754068330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:17:18.029 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:18.024+0000 7fe753fff700 1 --2- 172.21.15.40:0/2687442009 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe7540feb10 0x7fe754068330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:18.029 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:18.024+0000 7fe753fff700 1 -- 172.21.15.40:0/2687442009 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7480095f0 con 0x7fe754100690 2023-07-14T19:17:18.030 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:18.024+0000 7fe753fff700 1 --2- 172.21.15.40:0/2687442009 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe754100690 0x7fe75419ccc0 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7fe74400d980 tx=0x7fe74400dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:17:18.030 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:18.024+0000 7fe750ff9700 1 -- 172.21.15.40:0/2687442009 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe744004a30 con 0x7fe754100690 2023-07-14T19:17:18.030 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:18.024+0000 7fe750ff9700 1 -- 172.21.15.40:0/2687442009 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe7440054e0 con 0x7fe754100690 2023-07-14T19:17:18.030 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:18.024+0000 7fe750ff9700 1 -- 172.21.15.40:0/2687442009 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe74400fbc0 con 0x7fe754100690 2023-07-14T19:17:18.030 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:18.024+0000 7fe75a179700 1 -- 172.21.15.40:0/2687442009 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe75419daa0 con 0x7fe754100690 2023-07-14T19:17:18.030 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:18.024+0000 7fe75a179700 1 -- 172.21.15.40:0/2687442009 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe75419df60 con 0x7fe754100690 2023-07-14T19:17:18.030 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:18.025+0000 7fe75a179700 1 -- 172.21.15.40:0/2687442009 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe75410a330 con 0x7fe754100690 2023-07-14T19:17:18.031 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:18.026+0000 7fe750ff9700 1 -- 172.21.15.40:0/2687442009 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7fe744004c20 con 0x7fe754100690 2023-07-14T19:17:18.031 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:18.027+0000 7fe750ff9700 1 --2- 172.21.15.40:0/2687442009 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7fe73c0a1260 0x7fe73c0a3710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:17:18.031 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:18.027+0000 7fe7537fe700 1 --2- 172.21.15.40:0/2687442009 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7fe73c0a1260 0x7fe73c0a3710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:17:18.032 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:18.027+0000 7fe750ff9700 1 -- 172.21.15.40:0/2687442009 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 8269+0+0 (secure 0 0 0) 0x7fe7440cc7d0 con 0x7fe754100690 2023-07-14T19:17:18.032 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:18.027+0000 7fe7537fe700 1 --2- 172.21.15.40:0/2687442009 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7fe73c0a1260 0x7fe73c0a3710 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fe7540ffb60 tx=0x7fe74c007a30 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:17:18.032 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:18.031+0000 7fe750ff9700 1 -- 172.21.15.40:0/2687442009 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fe744095da0 con 0x7fe754100690 2023-07-14T19:17:18.221 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:17:17 smithi040 ceph-mon[96518]: pgmap v296: 97 pgs: 97 active+clean; 638 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2023-07-14T19:17:18.221 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:17:17 smithi040 ceph-mon[96518]: from='client.? 172.21.15.40:0/4043650530' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-07-14T19:17:18.221 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:17:17 smithi040 ceph-mon[96518]: from='client.? 172.21.15.40:0/1815731590' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T19:17:18.221 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:17:17 smithi040 ceph-mon[96518]: from='client.? 172.21.15.40:0/1869369709' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T19:17:18.222 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:17:17 smithi191 ceph-mon[98311]: pgmap v296: 97 pgs: 97 active+clean; 638 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2023-07-14T19:17:18.222 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:17:17 smithi191 ceph-mon[98311]: from='client.? 172.21.15.40:0/4043650530' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-07-14T19:17:18.222 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:17:17 smithi191 ceph-mon[98311]: from='client.? 172.21.15.40:0/1815731590' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T19:17:18.222 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:17:17 smithi191 ceph-mon[98311]: from='client.? 172.21.15.40:0/1869369709' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T19:17:18.234 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:18.232+0000 7fe75a179700 1 -- 172.21.15.40:0/2687442009 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fe7541059a0 con 0x7fe754100690 2023-07-14T19:17:18.234 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:18.233+0000 7fe750ff9700 1 -- 172.21.15.40:0/2687442009 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v83) v1 ==== 93+0+10 (secure 0 0 0) 0x7fe7440993c0 con 0x7fe754100690 2023-07-14T19:17:18.234 INFO:teuthology.orchestra.run.smithi040.stdout:pg_num: 1 2023-07-14T19:17:18.236 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:18.235+0000 7fe75a179700 1 -- 172.21.15.40:0/2687442009 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7fe73c0a1260 msgr2=0x7fe73c0a3710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:17:18.237 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:18.235+0000 7fe75a179700 1 --2- 172.21.15.40:0/2687442009 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7fe73c0a1260 0x7fe73c0a3710 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fe7540ffb60 tx=0x7fe74c007a30 comp rx=0 tx=0).stop 2023-07-14T19:17:18.237 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:18.235+0000 7fe75a179700 1 -- 172.21.15.40:0/2687442009 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe754100690 msgr2=0x7fe75419ccc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:17:18.237 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:18.235+0000 7fe75a179700 1 --2- 172.21.15.40:0/2687442009 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe754100690 0x7fe75419ccc0 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7fe74400d980 tx=0x7fe74400dc90 comp rx=0 tx=0).stop 2023-07-14T19:17:18.237 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:18.235+0000 7fe75a179700 1 -- 172.21.15.40:0/2687442009 shutdown_connections 2023-07-14T19:17:18.237 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:18.235+0000 7fe75a179700 1 --2- 172.21.15.40:0/2687442009 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7fe73c0a1260 0x7fe73c0a3710 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:18.238 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:18.235+0000 7fe75a179700 1 --2- 172.21.15.40:0/2687442009 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe754100690 0x7fe75419ccc0 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:18.238 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:18.235+0000 7fe75a179700 1 --2- 172.21.15.40:0/2687442009 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe7540feb10 0x7fe754068330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:18.238 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:18.235+0000 7fe75a179700 1 --2- 172.21.15.40:0/2687442009 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe7540ffd00 0x7fe75419a670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:18.238 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:18.235+0000 7fe75a179700 1 -- 172.21.15.40:0/2687442009 >> 172.21.15.40:0/2687442009 conn(0x7fe7540fa130 msgr2=0x7fe7540fc310 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:17:18.238 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:18.235+0000 7fe75a179700 1 -- 172.21.15.40:0/2687442009 shutdown_connections 2023-07-14T19:17:18.238 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:18.235+0000 7fe75a179700 1 -- 172.21.15.40:0/2687442009 wait complete. 2023-07-14T19:17:18.242 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:17:17 smithi130 ceph-mon[97832]: pgmap v296: 97 pgs: 97 active+clean; 638 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2023-07-14T19:17:18.242 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:17:17 smithi130 ceph-mon[97832]: from='client.? 172.21.15.40:0/4043650530' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-07-14T19:17:18.242 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:17:17 smithi130 ceph-mon[97832]: from='client.? 172.21.15.40:0/1815731590' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T19:17:18.242 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:17:17 smithi130 ceph-mon[97832]: from='client.? 172.21.15.40:0/1869369709' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T19:17:18.250 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-07-14T19:17:18.379 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:18.377+0000 7f7682c67700 1 -- 172.21.15.40:0/510807964 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f767c0ffd30 msgr2=0x7f767c100180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:17:18.380 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:18.377+0000 7f7682c67700 1 --2- 172.21.15.40:0/510807964 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f767c0ffd30 0x7f767c100180 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f7668009990 tx=0x7f7668009ca0 comp rx=0 tx=0).stop 2023-07-14T19:17:18.380 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:18.377+0000 7f7682c67700 1 -- 172.21.15.40:0/510807964 shutdown_connections 2023-07-14T19:17:18.380 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:18.377+0000 7f7682c67700 1 --2- 172.21.15.40:0/510807964 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f767c1006c0 0x7f767c106f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:18.380 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:18.377+0000 7f7682c67700 1 --2- 172.21.15.40:0/510807964 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f767c0feb40 0x7f767c0fef50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:18.380 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:18.377+0000 7f7682c67700 1 --2- 172.21.15.40:0/510807964 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f767c0ffd30 0x7f767c100180 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:18.381 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:18.377+0000 7f7682c67700 1 -- 172.21.15.40:0/510807964 >> 172.21.15.40:0/510807964 conn(0x7f767c0fa160 msgr2=0x7f767c0fc590 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:17:18.381 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:18.377+0000 7f7682c67700 1 -- 172.21.15.40:0/510807964 shutdown_connections 2023-07-14T19:17:18.381 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:18.378+0000 7f7682c67700 1 -- 172.21.15.40:0/510807964 wait complete. 2023-07-14T19:17:18.381 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:18.378+0000 7f7682c67700 1 Processor -- start 2023-07-14T19:17:18.383 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:18.378+0000 7f7682c67700 1 -- start start 2023-07-14T19:17:18.383 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:18.378+0000 7f7682c67700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f767c0feb40 0x7f767c194740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:17:18.383 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:18.378+0000 7f7682c67700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f767c0ffd30 0x7f767c194c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:17:18.383 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:18.378+0000 7f7682c67700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f767c1006c0 0x7f767c19bcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:17:18.383 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:18.378+0000 7f7682c67700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f767c19c2c0 con 0x7f767c1006c0 2023-07-14T19:17:18.383 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:18.378+0000 7f7682c67700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f767c19c400 con 0x7f767c0feb40 2023-07-14T19:17:18.384 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:18.378+0000 7f7682c67700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f767c19c590 con 0x7f767c0ffd30 2023-07-14T19:17:18.384 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:18.378+0000 7f7681204700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f767c1006c0 0x7f767c19bcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:17:18.384 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:18.378+0000 7f7681204700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f767c1006c0 0x7f767c19bcf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:34094/0 (socket says 172.21.15.40:34094) 2023-07-14T19:17:18.384 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:18.379+0000 7f7681204700 1 -- 172.21.15.40:0/3569022721 learned_addr learned my addr 172.21.15.40:0/3569022721 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-07-14T19:17:18.384 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:18.379+0000 7f7680a03700 1 --2- 172.21.15.40:0/3569022721 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f767c0feb40 0x7f767c194740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:17:18.384 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:18.379+0000 7f767bfff700 1 --2- 172.21.15.40:0/3569022721 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f767c0ffd30 0x7f767c194c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:17:18.384 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:18.379+0000 7f7681204700 1 -- 172.21.15.40:0/3569022721 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f767c0ffd30 msgr2=0x7f767c194c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:17:18.385 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:18.379+0000 7f7681204700 1 --2- 172.21.15.40:0/3569022721 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f767c0ffd30 0x7f767c194c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:18.385 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:18.379+0000 7f7681204700 1 -- 172.21.15.40:0/3569022721 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f767c0feb40 msgr2=0x7f767c194740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:17:18.385 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:18.379+0000 7f7681204700 1 --2- 172.21.15.40:0/3569022721 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f767c0feb40 0x7f767c194740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:18.385 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:18.379+0000 7f7681204700 1 -- 172.21.15.40:0/3569022721 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76680095f0 con 0x7f767c1006c0 2023-07-14T19:17:18.385 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:18.379+0000 7f7681204700 1 --2- 172.21.15.40:0/3569022721 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f767c1006c0 0x7f767c19bcf0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f7674007d70 tx=0x7f767400a340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:17:18.386 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:18.379+0000 7f7679ffb700 1 -- 172.21.15.40:0/3569022721 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f767400ad40 con 0x7f767c1006c0 2023-07-14T19:17:18.386 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:18.379+0000 7f7682c67700 1 -- 172.21.15.40:0/3569022721 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f767c19c870 con 0x7f767c1006c0 2023-07-14T19:17:18.386 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:18.379+0000 7f7679ffb700 1 -- 172.21.15.40:0/3569022721 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f76740128e0 con 0x7f767c1006c0 2023-07-14T19:17:18.387 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:18.379+0000 7f7679ffb700 1 -- 172.21.15.40:0/3569022721 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7674011780 con 0x7f767c1006c0 2023-07-14T19:17:18.387 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:18.379+0000 7f7682c67700 1 -- 172.21.15.40:0/3569022721 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f767c068950 con 0x7f767c1006c0 2023-07-14T19:17:18.388 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:18.380+0000 7f7682c67700 1 -- 172.21.15.40:0/3569022721 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f767c10a310 con 0x7f767c1006c0 2023-07-14T19:17:18.388 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:18.381+0000 7f7679ffb700 1 -- 172.21.15.40:0/3569022721 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7f7674012a50 con 0x7f767c1006c0 2023-07-14T19:17:18.390 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:18.382+0000 7f7679ffb700 1 --2- 172.21.15.40:0/3569022721 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f766c0a33f0 0x7f766c0a58a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:17:18.390 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:18.382+0000 7f7679ffb700 1 -- 172.21.15.40:0/3569022721 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 8269+0+0 (secure 0 0 0) 0x7f76740ccd10 con 0x7f767c1006c0 2023-07-14T19:17:18.390 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:18.382+0000 7f7680a03700 1 --2- 172.21.15.40:0/3569022721 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f766c0a33f0 0x7f766c0a58a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:17:18.390 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:18.382+0000 7f7680a03700 1 --2- 172.21.15.40:0/3569022721 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f766c0a33f0 0x7f766c0a58a0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f7670005fd0 tx=0x7f7670005e50 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:17:18.391 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:18.386+0000 7f7679ffb700 1 -- 172.21.15.40:0/3569022721 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f76740963f0 con 0x7f767c1006c0 2023-07-14T19:17:18.592 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:18.590+0000 7f7682c67700 1 -- 172.21.15.40:0/3569022721 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f767c1056c0 con 0x7f767c1006c0 2023-07-14T19:17:18.592 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:18.590+0000 7f7679ffb700 1 -- 172.21.15.40:0/3569022721 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v83) v1 ==== 104+0+11 (secure 0 0 0) 0x7f7674016020 con 0x7f767c1006c0 2023-07-14T19:17:18.592 INFO:teuthology.orchestra.run.smithi040.stdout:pg_num: 32 2023-07-14T19:17:18.594 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:18.592+0000 7f7682c67700 1 -- 172.21.15.40:0/3569022721 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f766c0a33f0 msgr2=0x7f766c0a58a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:17:18.594 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:18.592+0000 7f7682c67700 1 --2- 172.21.15.40:0/3569022721 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f766c0a33f0 0x7f766c0a58a0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f7670005fd0 tx=0x7f7670005e50 comp rx=0 tx=0).stop 2023-07-14T19:17:18.595 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:18.592+0000 7f7682c67700 1 -- 172.21.15.40:0/3569022721 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f767c1006c0 msgr2=0x7f767c19bcf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:17:18.595 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:18.592+0000 7f7682c67700 1 --2- 172.21.15.40:0/3569022721 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f767c1006c0 0x7f767c19bcf0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f7674007d70 tx=0x7f767400a340 comp rx=0 tx=0).stop 2023-07-14T19:17:18.595 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:18.592+0000 7f7682c67700 1 -- 172.21.15.40:0/3569022721 shutdown_connections 2023-07-14T19:17:18.595 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:18.592+0000 7f7682c67700 1 --2- 172.21.15.40:0/3569022721 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f766c0a33f0 0x7f766c0a58a0 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:18.595 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:18.592+0000 7f7682c67700 1 --2- 172.21.15.40:0/3569022721 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f767c1006c0 0x7f767c19bcf0 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:18.596 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:18.592+0000 7f7682c67700 1 --2- 172.21.15.40:0/3569022721 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f767c0feb40 0x7f767c194740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:18.596 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:18.592+0000 7f7682c67700 1 --2- 172.21.15.40:0/3569022721 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f767c0ffd30 0x7f767c194c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:18.596 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:18.592+0000 7f7682c67700 1 -- 172.21.15.40:0/3569022721 >> 172.21.15.40:0/3569022721 conn(0x7f767c0fa160 msgr2=0x7f767c0fc2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:17:18.596 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:18.592+0000 7f7682c67700 1 -- 172.21.15.40:0/3569022721 shutdown_connections 2023-07-14T19:17:18.596 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:18.592+0000 7f7682c67700 1 -- 172.21.15.40:0/3569022721 wait complete. 2023-07-14T19:17:18.606 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-07-14T19:17:18.736 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:18.734+0000 7f601ab3f700 1 -- 172.21.15.40:0/4121111951 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6014105930 msgr2=0x7f6014107d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:17:18.736 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:18.734+0000 7f601ab3f700 1 --2- 172.21.15.40:0/4121111951 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6014105930 0x7f6014107d30 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f600c00b0a0 tx=0x7f600c00b3b0 comp rx=0 tx=0).stop 2023-07-14T19:17:18.736 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:18.734+0000 7f601ab3f700 1 -- 172.21.15.40:0/4121111951 shutdown_connections 2023-07-14T19:17:18.736 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:18.734+0000 7f601ab3f700 1 --2- 172.21.15.40:0/4121111951 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f60141006f0 0x7f6014102ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:18.736 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:18.734+0000 7f601ab3f700 1 --2- 172.21.15.40:0/4121111951 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6014105930 0x7f6014107d30 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:18.736 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:18.734+0000 7f601ab3f700 1 --2- 172.21.15.40:0/4121111951 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6014103010 0x7f60141053f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:18.737 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:18.734+0000 7f601ab3f700 1 -- 172.21.15.40:0/4121111951 >> 172.21.15.40:0/4121111951 conn(0x7f60140fa130 msgr2=0x7f60140fc560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:17:18.737 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:18.734+0000 7f601ab3f700 1 -- 172.21.15.40:0/4121111951 shutdown_connections 2023-07-14T19:17:18.737 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:18.734+0000 7f601ab3f700 1 -- 172.21.15.40:0/4121111951 wait complete. 2023-07-14T19:17:18.739 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:18.734+0000 7f601ab3f700 1 Processor -- start 2023-07-14T19:17:18.739 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:18.734+0000 7f601ab3f700 1 -- start start 2023-07-14T19:17:18.739 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:18.735+0000 7f601ab3f700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f60141006f0 0x7f6014100060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:17:18.739 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:18.735+0000 7f601ab3f700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6014103010 0x7f60140fe6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:17:18.739 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:18.735+0000 7f601ab3f700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6014105930 0x7f60140febe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:17:18.739 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:18.735+0000 7f601ab3f700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60140ff380 con 0x7f6014105930 2023-07-14T19:17:18.739 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:18.735+0000 7f601ab3f700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60140ff4f0 con 0x7f6014103010 2023-07-14T19:17:18.740 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:18.735+0000 7f601ab3f700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f601419fac0 con 0x7f60141006f0 2023-07-14T19:17:18.740 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:18.735+0000 7f60190dc700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6014105930 0x7f60140febe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:17:18.740 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:18.735+0000 7f60190dc700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6014105930 0x7f60140febe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:34108/0 (socket says 172.21.15.40:34108) 2023-07-14T19:17:18.740 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:18.735+0000 7f60188db700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f60141006f0 0x7f6014100060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:17:18.740 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:18.735+0000 7f60190dc700 1 -- 172.21.15.40:0/3423596551 learned_addr learned my addr 172.21.15.40:0/3423596551 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-07-14T19:17:18.740 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:18.735+0000 7f6013fff700 1 --2- 172.21.15.40:0/3423596551 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6014103010 0x7f60140fe6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:17:18.740 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:18.735+0000 7f60190dc700 1 -- 172.21.15.40:0/3423596551 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f60141006f0 msgr2=0x7f6014100060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:17:18.741 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:18.735+0000 7f60190dc700 1 --2- 172.21.15.40:0/3423596551 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f60141006f0 0x7f6014100060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:18.741 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:18.735+0000 7f60190dc700 1 -- 172.21.15.40:0/3423596551 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6014103010 msgr2=0x7f60140fe6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:17:18.741 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:18.735+0000 7f60190dc700 1 --2- 172.21.15.40:0/3423596551 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6014103010 0x7f60140fe6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:18.741 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:18.735+0000 7f60190dc700 1 -- 172.21.15.40:0/3423596551 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f600c009d00 con 0x7f6014105930 2023-07-14T19:17:18.741 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:18.735+0000 7f6013fff700 1 --2- 172.21.15.40:0/3423596551 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6014103010 0x7f60140fe6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-14T19:17:18.741 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:18.735+0000 7f60190dc700 1 --2- 172.21.15.40:0/3423596551 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6014105930 0x7f60140febe0 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f600c00b070 tx=0x7f600c004a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:17:18.741 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:18.735+0000 7f6011ffb700 1 -- 172.21.15.40:0/3423596551 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f600c00e040 con 0x7f6014105930 2023-07-14T19:17:18.742 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:18.735+0000 7f6011ffb700 1 -- 172.21.15.40:0/3423596551 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f600c004e80 con 0x7f6014105930 2023-07-14T19:17:18.742 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:18.735+0000 7f6011ffb700 1 -- 172.21.15.40:0/3423596551 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f600c01b6b0 con 0x7f6014105930 2023-07-14T19:17:18.742 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:18.736+0000 7f601ab3f700 1 -- 172.21.15.40:0/3423596551 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f601419fd40 con 0x7f6014105930 2023-07-14T19:17:18.742 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:18.736+0000 7f601ab3f700 1 -- 172.21.15.40:0/3423596551 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f60141a0170 con 0x7f6014105930 2023-07-14T19:17:18.742 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:18.737+0000 7f601ab3f700 1 -- 172.21.15.40:0/3423596551 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f60140fbca0 con 0x7f6014105930 2023-07-14T19:17:18.743 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:18.738+0000 7f6011ffb700 1 -- 172.21.15.40:0/3423596551 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7f600c01b810 con 0x7f6014105930 2023-07-14T19:17:18.743 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:18.738+0000 7f6011ffb700 1 --2- 172.21.15.40:0/3423596551 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f60040a12e0 0x7f60040a3790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:17:18.743 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:18.739+0000 7f60188db700 1 --2- 172.21.15.40:0/3423596551 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f60040a12e0 0x7f60040a3790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:17:18.743 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:18.739+0000 7f6011ffb700 1 -- 172.21.15.40:0/3423596551 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 8269+0+0 (secure 0 0 0) 0x7f600c0ce340 con 0x7f6014105930 2023-07-14T19:17:18.743 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:18.739+0000 7f60188db700 1 --2- 172.21.15.40:0/3423596551 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f60040a12e0 0x7f60040a3790 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f6008005950 tx=0x7f600800b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:17:18.744 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:18.742+0000 7f6011ffb700 1 -- 172.21.15.40:0/3423596551 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f600c097940 con 0x7f6014105930 2023-07-14T19:17:18.890 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:17:18 smithi040 ceph-mon[96518]: from='client.? 172.21.15.40:0/2687442009' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-07-14T19:17:18.941 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:18.940+0000 7f601ab3f700 1 -- 172.21.15.40:0/3423596551 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f601402cd70 con 0x7f6014105930 2023-07-14T19:17:18.942 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:18.940+0000 7f6011ffb700 1 -- 172.21.15.40:0/3423596551 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v83) v1 ==== 100+0+11 (secure 0 0 0) 0x7f600c09af60 con 0x7f6014105930 2023-07-14T19:17:18.942 INFO:teuthology.orchestra.run.smithi040.stdout:pg_num: 64 2023-07-14T19:17:18.944 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:18.942+0000 7f601ab3f700 1 -- 172.21.15.40:0/3423596551 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f60040a12e0 msgr2=0x7f60040a3790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:17:18.944 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:18.942+0000 7f601ab3f700 1 --2- 172.21.15.40:0/3423596551 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f60040a12e0 0x7f60040a3790 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f6008005950 tx=0x7f600800b410 comp rx=0 tx=0).stop 2023-07-14T19:17:18.944 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:18.942+0000 7f601ab3f700 1 -- 172.21.15.40:0/3423596551 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6014105930 msgr2=0x7f60140febe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:17:18.944 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:18.942+0000 7f601ab3f700 1 --2- 172.21.15.40:0/3423596551 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6014105930 0x7f60140febe0 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f600c00b070 tx=0x7f600c004a50 comp rx=0 tx=0).stop 2023-07-14T19:17:18.945 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:18.942+0000 7f601ab3f700 1 -- 172.21.15.40:0/3423596551 shutdown_connections 2023-07-14T19:17:18.945 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:18.942+0000 7f601ab3f700 1 --2- 172.21.15.40:0/3423596551 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f60040a12e0 0x7f60040a3790 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:18.945 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:18.942+0000 7f601ab3f700 1 --2- 172.21.15.40:0/3423596551 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f60141006f0 0x7f6014100060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:18.945 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:18.942+0000 7f601ab3f700 1 --2- 172.21.15.40:0/3423596551 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6014105930 0x7f60140febe0 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:18.945 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:18.942+0000 7f601ab3f700 1 --2- 172.21.15.40:0/3423596551 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6014103010 0x7f60140fe6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:18.945 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:18.942+0000 7f601ab3f700 1 -- 172.21.15.40:0/3423596551 >> 172.21.15.40:0/3423596551 conn(0x7f60140fa130 msgr2=0x7f6014101350 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:17:18.945 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:18.942+0000 7f601ab3f700 1 -- 172.21.15.40:0/3423596551 shutdown_connections 2023-07-14T19:17:18.946 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:18.942+0000 7f601ab3f700 1 -- 172.21.15.40:0/3423596551 wait complete. 2023-07-14T19:17:18.957 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-07-14T19:17:19.083 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:19.081+0000 7f13e0825700 1 -- 172.21.15.40:0/1596831658 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f13dc100680 msgr2=0x7f13dc106f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:17:19.083 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:19.081+0000 7f13e0825700 1 --2- 172.21.15.40:0/1596831658 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f13dc100680 0x7f13dc106f40 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f13d400b0a0 tx=0x7f13d400b3b0 comp rx=0 tx=0).stop 2023-07-14T19:17:19.083 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:19.081+0000 7f13e0825700 1 -- 172.21.15.40:0/1596831658 shutdown_connections 2023-07-14T19:17:19.083 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:19.081+0000 7f13e0825700 1 --2- 172.21.15.40:0/1596831658 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f13dc100680 0x7f13dc106f40 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:19.084 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:19.081+0000 7f13e0825700 1 --2- 172.21.15.40:0/1596831658 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f13dc0feb00 0x7f13dc0fef10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:19.084 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:19.081+0000 7f13e0825700 1 --2- 172.21.15.40:0/1596831658 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f13dc0ffcf0 0x7f13dc100140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:19.084 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:19.081+0000 7f13e0825700 1 -- 172.21.15.40:0/1596831658 >> 172.21.15.40:0/1596831658 conn(0x7f13dc0fa100 msgr2=0x7f13dc0fc550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:17:19.084 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:19.081+0000 7f13e0825700 1 -- 172.21.15.40:0/1596831658 shutdown_connections 2023-07-14T19:17:19.084 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:19.081+0000 7f13e0825700 1 -- 172.21.15.40:0/1596831658 wait complete. 2023-07-14T19:17:19.086 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:19.082+0000 7f13e0825700 1 Processor -- start 2023-07-14T19:17:19.086 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:19.082+0000 7f13e0825700 1 -- start start 2023-07-14T19:17:19.086 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:19.082+0000 7f13e0825700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f13dc0feb00 0x7f13dc194930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:17:19.087 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:19.082+0000 7f13e0825700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f13dc0ffcf0 0x7f13dc194e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:17:19.087 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:19.082+0000 7f13e0825700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f13dc100680 0x7f13dc19bee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:17:19.087 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:19.082+0000 7f13e0825700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13dc19c4b0 con 0x7f13dc0feb00 2023-07-14T19:17:19.087 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:19.082+0000 7f13e0825700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13dc19c5f0 con 0x7f13dc0ffcf0 2023-07-14T19:17:19.087 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:19.082+0000 7f13e0825700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13dc19c780 con 0x7f13dc100680 2023-07-14T19:17:19.087 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:19.082+0000 7f13da59c700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f13dc0feb00 0x7f13dc194930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:17:19.087 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:19.082+0000 7f13da59c700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f13dc0feb00 0x7f13dc194930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:34118/0 (socket says 172.21.15.40:34118) 2023-07-14T19:17:19.088 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:19.082+0000 7f13da59c700 1 -- 172.21.15.40:0/2362745860 learned_addr learned my addr 172.21.15.40:0/2362745860 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-07-14T19:17:19.088 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:19.082+0000 7f13dad9d700 1 --2- 172.21.15.40:0/2362745860 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f13dc100680 0x7f13dc19bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:17:19.088 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:19.082+0000 7f13da59c700 1 -- 172.21.15.40:0/2362745860 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f13dc100680 msgr2=0x7f13dc19bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:17:19.088 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:19.082+0000 7f13da59c700 1 --2- 172.21.15.40:0/2362745860 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f13dc100680 0x7f13dc19bee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:19.088 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:19.082+0000 7f13da59c700 1 -- 172.21.15.40:0/2362745860 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f13dc0ffcf0 msgr2=0x7f13dc194e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:17:19.088 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:19.082+0000 7f13d9d9b700 1 --2- 172.21.15.40:0/2362745860 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f13dc0ffcf0 0x7f13dc194e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:17:19.088 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:19.082+0000 7f13da59c700 1 --2- 172.21.15.40:0/2362745860 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f13dc0ffcf0 0x7f13dc194e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:19.089 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:19.082+0000 7f13da59c700 1 -- 172.21.15.40:0/2362745860 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13d4009d00 con 0x7f13dc0feb00 2023-07-14T19:17:19.089 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:19.082+0000 7f13dad9d700 1 --2- 172.21.15.40:0/2362745860 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f13dc100680 0x7f13dc19bee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-14T19:17:19.089 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:19.082+0000 7f13d9d9b700 1 --2- 172.21.15.40:0/2362745860 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f13dc0ffcf0 0x7f13dc194e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-14T19:17:19.089 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:19.082+0000 7f13da59c700 1 --2- 172.21.15.40:0/2362745860 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f13dc0feb00 0x7f13dc194930 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f13c400d980 tx=0x7f13c400dd40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:17:19.089 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:19.083+0000 7f13d37fe700 1 -- 172.21.15.40:0/2362745860 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13c400de60 con 0x7f13dc0feb00 2023-07-14T19:17:19.089 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:19.083+0000 7f13d37fe700 1 -- 172.21.15.40:0/2362745860 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f13c40044e0 con 0x7f13dc0feb00 2023-07-14T19:17:19.090 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:19.083+0000 7f13e0825700 1 -- 172.21.15.40:0/2362745860 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f13dc067800 con 0x7f13dc0feb00 2023-07-14T19:17:19.090 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:19.083+0000 7f13d37fe700 1 -- 172.21.15.40:0/2362745860 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13c4003be0 con 0x7f13dc0feb00 2023-07-14T19:17:19.090 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:19.083+0000 7f13e0825700 1 -- 172.21.15.40:0/2362745860 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f13dc067d60 con 0x7f13dc0feb00 2023-07-14T19:17:19.090 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:19.084+0000 7f13e0825700 1 -- 172.21.15.40:0/2362745860 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f13dc10a330 con 0x7f13dc0feb00 2023-07-14T19:17:19.091 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:19.085+0000 7f13d37fe700 1 -- 172.21.15.40:0/2362745860 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7f13c4009bb0 con 0x7f13dc0feb00 2023-07-14T19:17:19.091 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:19.086+0000 7f13d37fe700 1 --2- 172.21.15.40:0/2362745860 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f13c80a12a0 0x7f13c80a3750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:17:19.091 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:19.086+0000 7f13d9d9b700 1 --2- 172.21.15.40:0/2362745860 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f13c80a12a0 0x7f13c80a3750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:17:19.091 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:19.086+0000 7f13d37fe700 1 -- 172.21.15.40:0/2362745860 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 8269+0+0 (secure 0 0 0) 0x7f13c40cc7f0 con 0x7f13dc0feb00 2023-07-14T19:17:19.091 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:19.086+0000 7f13d9d9b700 1 --2- 172.21.15.40:0/2362745860 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f13c80a12a0 0x7f13c80a3750 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f13cc005fd0 tx=0x7f13cc005ee0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:17:19.092 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:19.090+0000 7f13d37fe700 1 -- 172.21.15.40:0/2362745860 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f13c4095dc0 con 0x7f13dc0feb00 2023-07-14T19:17:19.221 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:17:18 smithi040 ceph-mon[96518]: from='client.? 172.21.15.40:0/3569022721' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-07-14T19:17:19.221 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:17:18 smithi191 ceph-mon[98311]: from='client.? 172.21.15.40:0/2687442009' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-07-14T19:17:19.222 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:17:18 smithi191 ceph-mon[98311]: from='client.? 172.21.15.40:0/3569022721' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-07-14T19:17:19.242 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:17:18 smithi130 ceph-mon[97832]: from='client.? 172.21.15.40:0/2687442009' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-07-14T19:17:19.242 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:17:18 smithi130 ceph-mon[97832]: from='client.? 172.21.15.40:0/3569022721' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-07-14T19:17:19.338 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:19.336+0000 7f13e0825700 1 -- 172.21.15.40:0/2362745860 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f13dc0686b0 con 0x7f13dc0feb00 2023-07-14T19:17:19.339 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:19.337+0000 7f13d37fe700 1 -- 172.21.15.40:0/2362745860 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) v1 ==== 94+0+10877 (secure 0 0 0) 0x7f13c4009940 con 0x7f13dc0feb00 2023-07-14T19:17:19.339 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-07-14T19:17:19.339 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14523,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6835/501143718","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6834","nonce":501143718},{"type":"v1","addr":"172.21.15.40:6835","nonce":501143718}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14535,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6837/2634476117","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6836","nonce":2634476117},{"type":"v1","addr":"172.21.15.40:6837","nonce":2634476117}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":26,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2023-07-14T19:15:57.650710+0000","modified":"2023-07-14T19:16:05.624568+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24539,"mds_1":24508,"mds_2":24506,"mds_3":24490,"mds_4":24482},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14586":{"gid":14586,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.40:6839/959937545","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6838","nonce":959937545},{"type":"v1","addr":"172.21.15.40:6839","nonce":959937545}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_14598":{"gid":14598,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.40:6841/1590486077","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6840","nonce":1590486077},{"type":"v1","addr":"172.21.15.40:6841","nonce":1590486077}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24433":{"gid":24433,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.191:6833/31509862","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":31509862},{"type":"v1","addr":"172.21.15.191:6833","nonce":31509862}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24452":{"gid":24452,"name":"k","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.130:6833/3677452688","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6832","nonce":3677452688},{"type":"v1","addr":"172.21.15.130:6833","nonce":3677452688}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24478":{"gid":24478,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.191:6835/3712674634","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":3712674634},{"type":"v1","addr":"172.21.15.191:6835","nonce":3712674634}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24482":{"gid":24482,"name":"e","rank":4,"incarnation":24,"state":"up:active","state_seq":8,"addr":"172.21.15.130:6835/3088490808","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6834","nonce":3088490808},{"type":"v1","addr":"172.21.15.130:6835","nonce":3088490808}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24490":{"gid":24490,"name":"c","rank":3,"incarnation":22,"state":"up:active","state_seq":7,"addr":"172.21.15.191:6837/2406892975","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6836","nonce":2406892975},{"type":"v1","addr":"172.21.15.191:6837","nonce":2406892975}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24506":{"gid":24506,"name":"h","rank":2,"incarnation":20,"state":"up:active","state_seq":6,"addr":"172.21.15.130:6837/2604019369","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6836","nonce":2604019369},{"type":"v1","addr":"172.21.15.130:6837","nonce":2604019369}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24508":{"gid":24508,"name":"i","rank":1,"incarnation":18,"state":"up:active","state_seq":5,"addr":"172.21.15.191:6839/1977507302","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":1977507302},{"type":"v1","addr":"172.21.15.191:6839","nonce":1977507302}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24539":{"gid":24539,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.130:6839/2145479535","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6838","nonce":2145479535},{"type":"v1","addr":"172.21.15.130:6839","nonce":2145479535}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-07-14T19:17:19.341 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:19.339+0000 7f13e0825700 1 -- 172.21.15.40:0/2362745860 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f13c80a12a0 msgr2=0x7f13c80a3750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:17:19.341 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:19.339+0000 7f13e0825700 1 --2- 172.21.15.40:0/2362745860 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f13c80a12a0 0x7f13c80a3750 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f13cc005fd0 tx=0x7f13cc005ee0 comp rx=0 tx=0).stop 2023-07-14T19:17:19.341 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:19.339+0000 7f13e0825700 1 -- 172.21.15.40:0/2362745860 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f13dc0feb00 msgr2=0x7f13dc194930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:17:19.341 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:19.339+0000 7f13e0825700 1 --2- 172.21.15.40:0/2362745860 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f13dc0feb00 0x7f13dc194930 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f13c400d980 tx=0x7f13c400dd40 comp rx=0 tx=0).stop 2023-07-14T19:17:19.341 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:19.339+0000 7f13e0825700 1 -- 172.21.15.40:0/2362745860 shutdown_connections 2023-07-14T19:17:19.342 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:19.339+0000 7f13e0825700 1 --2- 172.21.15.40:0/2362745860 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f13c80a12a0 0x7f13c80a3750 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:19.342 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:19.339+0000 7f13e0825700 1 --2- 172.21.15.40:0/2362745860 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f13dc100680 0x7f13dc19bee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:19.342 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:19.339+0000 7f13e0825700 1 --2- 172.21.15.40:0/2362745860 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f13dc0feb00 0x7f13dc194930 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:19.342 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:19.339+0000 7f13e0825700 1 --2- 172.21.15.40:0/2362745860 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f13dc0ffcf0 0x7f13dc194e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:19.342 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:19.339+0000 7f13e0825700 1 -- 172.21.15.40:0/2362745860 >> 172.21.15.40:0/2362745860 conn(0x7f13dc0fa100 msgr2=0x7f13dc0fc2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:17:19.342 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:19.339+0000 7f13e0825700 1 -- 172.21.15.40:0/2362745860 shutdown_connections 2023-07-14T19:17:19.342 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:19.339+0000 7f13e0825700 1 -- 172.21.15.40:0/2362745860 wait complete. 2023-07-14T19:17:19.343 INFO:teuthology.orchestra.run.smithi040.stderr:dumped fsmap epoch 26 2023-07-14T19:17:19.354 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-07-14T19:17:19.481 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:19.478+0000 7fe927be2700 1 -- 172.21.15.40:0/1797368177 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe920100680 msgr2=0x7fe920106f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:17:19.482 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:19.478+0000 7fe927be2700 1 --2- 172.21.15.40:0/1797368177 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe920100680 0x7fe920106f40 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7fe91800b0a0 tx=0x7fe91800b3b0 comp rx=0 tx=0).stop 2023-07-14T19:17:19.482 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:19.478+0000 7fe927be2700 1 -- 172.21.15.40:0/1797368177 shutdown_connections 2023-07-14T19:17:19.482 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:19.478+0000 7fe927be2700 1 --2- 172.21.15.40:0/1797368177 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe920100680 0x7fe920106f40 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:19.482 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:19.478+0000 7fe927be2700 1 --2- 172.21.15.40:0/1797368177 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe9200feb00 0x7fe9200fef10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:19.482 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:19.478+0000 7fe927be2700 1 --2- 172.21.15.40:0/1797368177 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe9200ffcf0 0x7fe920100140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:19.482 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:19.478+0000 7fe927be2700 1 -- 172.21.15.40:0/1797368177 >> 172.21.15.40:0/1797368177 conn(0x7fe9200fa100 msgr2=0x7fe9200fc550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:17:19.483 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:19.478+0000 7fe927be2700 1 -- 172.21.15.40:0/1797368177 shutdown_connections 2023-07-14T19:17:19.483 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:19.478+0000 7fe927be2700 1 -- 172.21.15.40:0/1797368177 wait complete. 2023-07-14T19:17:19.483 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:19.479+0000 7fe927be2700 1 Processor -- start 2023-07-14T19:17:19.483 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:19.479+0000 7fe927be2700 1 -- start start 2023-07-14T19:17:19.483 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:19.479+0000 7fe927be2700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe9200feb00 0x7fe920068310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:17:19.483 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:19.479+0000 7fe927be2700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe9200ffcf0 0x7fe92019a740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:17:19.483 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:19.479+0000 7fe927be2700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe920100680 0x7fe92019cd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:17:19.484 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:19.479+0000 7fe927be2700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe92019d520 con 0x7fe920100680 2023-07-14T19:17:19.484 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:19.479+0000 7fe927be2700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe92019d690 con 0x7fe9200ffcf0 2023-07-14T19:17:19.484 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:19.479+0000 7fe927be2700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe92019d850 con 0x7fe9200feb00 2023-07-14T19:17:19.484 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:19.479+0000 7fe92617f700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe920100680 0x7fe92019cd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:17:19.484 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:19.479+0000 7fe92617f700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe920100680 0x7fe92019cd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:34136/0 (socket says 172.21.15.40:34136) 2023-07-14T19:17:19.484 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:19.479+0000 7fe92617f700 1 -- 172.21.15.40:0/420793535 learned_addr learned my addr 172.21.15.40:0/420793535 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-07-14T19:17:19.484 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:19.479+0000 7fe92597e700 1 --2- 172.21.15.40:0/420793535 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe9200feb00 0x7fe920068310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:17:19.485 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:19.479+0000 7fe92517d700 1 --2- 172.21.15.40:0/420793535 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe9200ffcf0 0x7fe92019a740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:17:19.485 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:19.479+0000 7fe92617f700 1 -- 172.21.15.40:0/420793535 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe9200feb00 msgr2=0x7fe920068310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:17:19.485 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:19.479+0000 7fe92617f700 1 --2- 172.21.15.40:0/420793535 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe9200feb00 0x7fe920068310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:19.485 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:19.479+0000 7fe92617f700 1 -- 172.21.15.40:0/420793535 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe9200ffcf0 msgr2=0x7fe92019a740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:17:19.485 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:19.479+0000 7fe92617f700 1 --2- 172.21.15.40:0/420793535 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe9200ffcf0 0x7fe92019a740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:19.485 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:19.479+0000 7fe92617f700 1 -- 172.21.15.40:0/420793535 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe918009d00 con 0x7fe920100680 2023-07-14T19:17:19.486 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:19.479+0000 7fe92617f700 1 --2- 172.21.15.40:0/420793535 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe920100680 0x7fe92019cd80 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7fe91800b830 tx=0x7fe918004c80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:17:19.486 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:19.480+0000 7fe916ffd700 1 -- 172.21.15.40:0/420793535 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe91800e040 con 0x7fe920100680 2023-07-14T19:17:19.486 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:19.480+0000 7fe927be2700 1 -- 172.21.15.40:0/420793535 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe92019dad0 con 0x7fe920100680 2023-07-14T19:17:19.486 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:19.480+0000 7fe916ffd700 1 -- 172.21.15.40:0/420793535 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe91801db50 con 0x7fe920100680 2023-07-14T19:17:19.486 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:19.480+0000 7fe916ffd700 1 -- 172.21.15.40:0/420793535 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe91801bbd0 con 0x7fe920100680 2023-07-14T19:17:19.487 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:19.480+0000 7fe927be2700 1 -- 172.21.15.40:0/420793535 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe92019df50 con 0x7fe920100680 2023-07-14T19:17:19.488 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:19.480+0000 7fe927be2700 1 -- 172.21.15.40:0/420793535 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe920106020 con 0x7fe920100680 2023-07-14T19:17:19.488 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:19.482+0000 7fe916ffd700 1 -- 172.21.15.40:0/420793535 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7fe91801dcc0 con 0x7fe920100680 2023-07-14T19:17:19.488 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:19.482+0000 7fe916ffd700 1 --2- 172.21.15.40:0/420793535 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7fe90c0a1220 0x7fe90c0a36d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:17:19.488 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:19.483+0000 7fe92597e700 1 --2- 172.21.15.40:0/420793535 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7fe90c0a1220 0x7fe90c0a36d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:17:19.489 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:19.483+0000 7fe916ffd700 1 -- 172.21.15.40:0/420793535 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 8269+0+0 (secure 0 0 0) 0x7fe9180cece0 con 0x7fe920100680 2023-07-14T19:17:19.489 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:19.483+0000 7fe92597e700 1 --2- 172.21.15.40:0/420793535 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7fe90c0a1220 0x7fe90c0a36d0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fe9200ffb50 tx=0x7fe91c00b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:17:19.489 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:19.486+0000 7fe916ffd700 1 -- 172.21.15.40:0/420793535 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fe918098340 con 0x7fe920100680 2023-07-14T19:17:19.690 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:19.688+0000 7fe927be2700 1 -- 172.21.15.40:0/420793535 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe92002c9d0 con 0x7fe920100680 2023-07-14T19:17:19.691 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:19.690+0000 7fe916ffd700 1 -- 172.21.15.40:0/420793535 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v83) v1 ==== 74+0+22647 (secure 0 0 0) 0x7fe91809b960 con 0x7fe920100680 2023-07-14T19:17:19.692 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-07-14T19:17:19.692 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":83,"fsid":"adc52f04-2279-11ee-9b33-001a4aab830c","created":"2023-07-14T19:08:56.893730+0000","modified":"2023-07-14T19:16:52.763512+0000","last_up_change":"2023-07-14T19:15:16.632979+0000","last_in_change":"2023-07-14T19:14:58.961702+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-07-14T19:12:03.271333+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-07-14T19:15:56.073464+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-07-14T19:15:57.018726+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"0d79356a-95d2-49ad-857b-2092ae9f7e13","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":3599703125},{"type":"v1","addr":"172.21.15.40:6803","nonce":3599703125}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":3599703125},{"type":"v1","addr":"172.21.15.40:6805","nonce":3599703125}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":3599703125},{"type":"v1","addr":"172.21.15.40:6809","nonce":3599703125}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":3599703125},{"type":"v1","addr":"172.21.15.40:6807","nonce":3599703125}]},"public_addr":"172.21.15.40:6803/3599703125","cluster_addr":"172.21.15.40:6805/3599703125","heartbeat_back_addr":"172.21.15.40:6809/3599703125","heartbeat_front_addr":"172.21.15.40:6807/3599703125","state":["exists","up"]},{"osd":1,"uuid":"40f8a3d1-a5e4-473c-b9f2-7600ec22be0d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6800","nonce":2512760702},{"type":"v1","addr":"172.21.15.130:6801","nonce":2512760702}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6802","nonce":2512760702},{"type":"v1","addr":"172.21.15.130:6803","nonce":2512760702}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6806","nonce":2512760702},{"type":"v1","addr":"172.21.15.130:6807","nonce":2512760702}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6804","nonce":2512760702},{"type":"v1","addr":"172.21.15.130:6805","nonce":2512760702}]},"public_addr":"172.21.15.130:6801/2512760702","cluster_addr":"172.21.15.130:6803/2512760702","heartbeat_back_addr":"172.21.15.130:6807/2512760702","heartbeat_front_addr":"172.21.15.130:6805/2512760702","state":["exists","up"]},{"osd":2,"uuid":"5a157891-cb7a-4ea6-9239-f48a7c67c003","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6800","nonce":589569135},{"type":"v1","addr":"172.21.15.191:6801","nonce":589569135}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6802","nonce":589569135},{"type":"v1","addr":"172.21.15.191:6803","nonce":589569135}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6806","nonce":589569135},{"type":"v1","addr":"172.21.15.191:6807","nonce":589569135}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6804","nonce":589569135},{"type":"v1","addr":"172.21.15.191:6805","nonce":589569135}]},"public_addr":"172.21.15.191:6801/589569135","cluster_addr":"172.21.15.191:6803/589569135","heartbeat_back_addr":"172.21.15.191:6807/589569135","heartbeat_front_addr":"172.21.15.191:6805/589569135","state":["exists","up"]},{"osd":3,"uuid":"8020e7dc-ac40-487f-857a-8da40bccb459","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":996287860},{"type":"v1","addr":"172.21.15.40:6811","nonce":996287860}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6812","nonce":996287860},{"type":"v1","addr":"172.21.15.40:6813","nonce":996287860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6816","nonce":996287860},{"type":"v1","addr":"172.21.15.40:6817","nonce":996287860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":996287860},{"type":"v1","addr":"172.21.15.40:6815","nonce":996287860}]},"public_addr":"172.21.15.40:6811/996287860","cluster_addr":"172.21.15.40:6813/996287860","heartbeat_back_addr":"172.21.15.40:6817/996287860","heartbeat_front_addr":"172.21.15.40:6815/996287860","state":["exists","up"]},{"osd":4,"uuid":"c5501b35-f62d-482d-8d22-115c88f0298e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6808","nonce":3507448720},{"type":"v1","addr":"172.21.15.130:6809","nonce":3507448720}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6810","nonce":3507448720},{"type":"v1","addr":"172.21.15.130:6811","nonce":3507448720}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6814","nonce":3507448720},{"type":"v1","addr":"172.21.15.130:6815","nonce":3507448720}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6812","nonce":3507448720},{"type":"v1","addr":"172.21.15.130:6813","nonce":3507448720}]},"public_addr":"172.21.15.130:6809/3507448720","cluster_addr":"172.21.15.130:6811/3507448720","heartbeat_back_addr":"172.21.15.130:6815/3507448720","heartbeat_front_addr":"172.21.15.130:6813/3507448720","state":["exists","up"]},{"osd":5,"uuid":"7770b6d6-5874-493a-986a-84db482cee42","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6808","nonce":1781944663},{"type":"v1","addr":"172.21.15.191:6809","nonce":1781944663}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6810","nonce":1781944663},{"type":"v1","addr":"172.21.15.191:6811","nonce":1781944663}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6814","nonce":1781944663},{"type":"v1","addr":"172.21.15.191:6815","nonce":1781944663}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6812","nonce":1781944663},{"type":"v1","addr":"172.21.15.191:6813","nonce":1781944663}]},"public_addr":"172.21.15.191:6809/1781944663","cluster_addr":"172.21.15.191:6811/1781944663","heartbeat_back_addr":"172.21.15.191:6815/1781944663","heartbeat_front_addr":"172.21.15.191:6813/1781944663","state":["exists","up"]},{"osd":6,"uuid":"ef65ad00-dfee-4300-9961-dfc8573aab06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6818","nonce":693186773},{"type":"v1","addr":"172.21.15.40:6819","nonce":693186773}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6820","nonce":693186773},{"type":"v1","addr":"172.21.15.40:6821","nonce":693186773}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6824","nonce":693186773},{"type":"v1","addr":"172.21.15.40:6825","nonce":693186773}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6822","nonce":693186773},{"type":"v1","addr":"172.21.15.40:6823","nonce":693186773}]},"public_addr":"172.21.15.40:6819/693186773","cluster_addr":"172.21.15.40:6821/693186773","heartbeat_back_addr":"172.21.15.40:6825/693186773","heartbeat_front_addr":"172.21.15.40:6823/693186773","state":["exists","up"]},{"osd":7,"uuid":"1700fbfa-cbfc-4f76-97be-727a3b100740","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":50,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6816","nonce":2384019528},{"type":"v1","addr":"172.21.15.130:6817","nonce":2384019528}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6818","nonce":2384019528},{"type":"v1","addr":"172.21.15.130:6819","nonce":2384019528}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6822","nonce":2384019528},{"type":"v1","addr":"172.21.15.130:6823","nonce":2384019528}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6820","nonce":2384019528},{"type":"v1","addr":"172.21.15.130:6821","nonce":2384019528}]},"public_addr":"172.21.15.130:6817/2384019528","cluster_addr":"172.21.15.130:6819/2384019528","heartbeat_back_addr":"172.21.15.130:6823/2384019528","heartbeat_front_addr":"172.21.15.130:6821/2384019528","state":["exists","up"]},{"osd":8,"uuid":"b76d4033-c1e2-457b-8571-c8a49fe443f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":57,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6816","nonce":2386818417},{"type":"v1","addr":"172.21.15.191:6817","nonce":2386818417}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6818","nonce":2386818417},{"type":"v1","addr":"172.21.15.191:6819","nonce":2386818417}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6822","nonce":2386818417},{"type":"v1","addr":"172.21.15.191:6823","nonce":2386818417}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6820","nonce":2386818417},{"type":"v1","addr":"172.21.15.191:6821","nonce":2386818417}]},"public_addr":"172.21.15.191:6817/2386818417","cluster_addr":"172.21.15.191:6819/2386818417","heartbeat_back_addr":"172.21.15.191:6823/2386818417","heartbeat_front_addr":"172.21.15.191:6821/2386818417","state":["exists","up"]},{"osd":9,"uuid":"df31bf2f-e30e-41b6-b2b9-0e26a07a88bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":64,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6826","nonce":2228380263},{"type":"v1","addr":"172.21.15.40:6827","nonce":2228380263}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6828","nonce":2228380263},{"type":"v1","addr":"172.21.15.40:6829","nonce":2228380263}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":2228380263},{"type":"v1","addr":"172.21.15.40:6833","nonce":2228380263}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6830","nonce":2228380263},{"type":"v1","addr":"172.21.15.40:6831","nonce":2228380263}]},"public_addr":"172.21.15.40:6827/2228380263","cluster_addr":"172.21.15.40:6829/2228380263","heartbeat_back_addr":"172.21.15.40:6833/2228380263","heartbeat_front_addr":"172.21.15.40:6831/2228380263","state":["exists","up"]},{"osd":10,"uuid":"4a93d621-49ee-4ad7-9e49-750614faf912","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6824","nonce":2417067798},{"type":"v1","addr":"172.21.15.130:6825","nonce":2417067798}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6826","nonce":2417067798},{"type":"v1","addr":"172.21.15.130:6827","nonce":2417067798}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6830","nonce":2417067798},{"type":"v1","addr":"172.21.15.130:6831","nonce":2417067798}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6828","nonce":2417067798},{"type":"v1","addr":"172.21.15.130:6829","nonce":2417067798}]},"public_addr":"172.21.15.130:6825/2417067798","cluster_addr":"172.21.15.130:6827/2417067798","heartbeat_back_addr":"172.21.15.130:6831/2417067798","heartbeat_front_addr":"172.21.15.130:6829/2417067798","state":["exists","up"]},{"osd":11,"uuid":"f43a2a6f-82f3-42ac-b77c-4f1b34da6918","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":76,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6824","nonce":2886059446},{"type":"v1","addr":"172.21.15.191:6825","nonce":2886059446}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6826","nonce":2886059446},{"type":"v1","addr":"172.21.15.191:6827","nonce":2886059446}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6830","nonce":2886059446},{"type":"v1","addr":"172.21.15.191:6831","nonce":2886059446}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6828","nonce":2886059446},{"type":"v1","addr":"172.21.15.191:6829","nonce":2886059446}]},"public_addr":"172.21.15.191:6825/2886059446","cluster_addr":"172.21.15.191:6827/2886059446","heartbeat_back_addr":"172.21.15.191:6831/2886059446","heartbeat_front_addr":"172.21.15.191:6829/2886059446","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:11:18.320495+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:11:39.901199+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:12:00.869042+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:12:23.292742+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:12:44.587019+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:13:06.494893+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:13:28.378374+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:13:50.033491+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:14:10.937062+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:14:32.358498+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:14:53.666584+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:15:14.837566+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"2.b","mappings":[{"from":8,"to":11}]},{"pgid":"2.c","mappings":[{"from":2,"to":4}]},{"pgid":"2.1f","mappings":[{"from":8,"to":1}]},{"pgid":"3.e","mappings":[{"from":4,"to":7}]},{"pgid":"3.17","mappings":[{"from":4,"to":8}]},{"pgid":"3.1a","mappings":[{"from":5,"to":6}]},{"pgid":"3.20","mappings":[{"from":5,"to":9}]},{"pgid":"3.27","mappings":[{"from":3,"to":9}]},{"pgid":"3.2c","mappings":[{"from":4,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.40:6801/3698757264":"2023-07-15T19:09:50.732458+0000","172.21.15.40:6800/3698757264":"2023-07-15T19:09:50.732458+0000","172.21.15.40:6800/2569015667":"2023-07-15T19:09:24.441543+0000","172.21.15.40:0/3653961003":"2023-07-15T19:09:24.441543+0000","172.21.15.40:0/3568861680":"2023-07-15T19:09:24.441543+0000","172.21.15.40:0/3249773172":"2023-07-15T19:09:50.732458+0000","172.21.15.40:0/4084215480":"2023-07-15T19:09:24.441543+0000","172.21.15.40:0/1163941760":"2023-07-15T19:09:50.732458+0000","172.21.15.40:6801/2569015667":"2023-07-15T19:09:24.441543+0000","172.21.15.40:0/2326639960":"2023-07-15T19:09:50.732458+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-07-14T19:17:19.693 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:19.692+0000 7fe927be2700 1 -- 172.21.15.40:0/420793535 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7fe90c0a1220 msgr2=0x7fe90c0a36d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:17:19.694 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:19.692+0000 7fe927be2700 1 --2- 172.21.15.40:0/420793535 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7fe90c0a1220 0x7fe90c0a36d0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fe9200ffb50 tx=0x7fe91c00b410 comp rx=0 tx=0).stop 2023-07-14T19:17:19.694 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:19.692+0000 7fe927be2700 1 -- 172.21.15.40:0/420793535 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe920100680 msgr2=0x7fe92019cd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:17:19.694 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:19.692+0000 7fe927be2700 1 --2- 172.21.15.40:0/420793535 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe920100680 0x7fe92019cd80 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7fe91800b830 tx=0x7fe918004c80 comp rx=0 tx=0).stop 2023-07-14T19:17:19.694 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:19.692+0000 7fe927be2700 1 -- 172.21.15.40:0/420793535 shutdown_connections 2023-07-14T19:17:19.694 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:19.692+0000 7fe927be2700 1 --2- 172.21.15.40:0/420793535 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7fe90c0a1220 0x7fe90c0a36d0 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:19.694 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:19.692+0000 7fe927be2700 1 --2- 172.21.15.40:0/420793535 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe9200feb00 0x7fe920068310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:19.695 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:19.692+0000 7fe927be2700 1 --2- 172.21.15.40:0/420793535 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe920100680 0x7fe92019cd80 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:19.695 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:19.692+0000 7fe927be2700 1 --2- 172.21.15.40:0/420793535 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe9200ffcf0 0x7fe92019a740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:19.695 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:19.692+0000 7fe927be2700 1 -- 172.21.15.40:0/420793535 >> 172.21.15.40:0/420793535 conn(0x7fe9200fa100 msgr2=0x7fe920103160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:17:19.695 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:19.692+0000 7fe927be2700 1 -- 172.21.15.40:0/420793535 shutdown_connections 2023-07-14T19:17:19.695 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:19.692+0000 7fe927be2700 1 -- 172.21.15.40:0/420793535 wait complete. 2023-07-14T19:17:19.707 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 26, 'flags': 50, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2023-07-14T19:15:57.650710+0000', 'modified': '2023-07-14T19:16:05.624568+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 24539, 'mds_1': 24508, 'mds_2': 24506, 'mds_3': 24490, 'mds_4': 24482}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14586': {'gid': 14586, 'name': 'g', 'rank': 4, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.40:6839/959937545', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.40:6838', 'nonce': 959937545}, {'type': 'v1', 'addr': '172.21.15.40:6839', 'nonce': 959937545}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}, 'gid_14598': {'gid': 14598, 'name': 'a', 'rank': 3, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.40:6841/1590486077', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.40:6840', 'nonce': 1590486077}, {'type': 'v1', 'addr': '172.21.15.40:6841', 'nonce': 1590486077}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}, 'gid_24433': {'gid': 24433, 'name': 'l', 'rank': 2, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.191:6833/31509862', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.191:6832', 'nonce': 31509862}, {'type': 'v1', 'addr': '172.21.15.191:6833', 'nonce': 31509862}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}, 'gid_24452': {'gid': 24452, 'name': 'k', 'rank': 1, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.130:6833/3677452688', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.130:6832', 'nonce': 3677452688}, {'type': 'v1', 'addr': '172.21.15.130:6833', 'nonce': 3677452688}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}, 'gid_24478': {'gid': 24478, 'name': 'f', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.191:6835/3712674634', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.191:6834', 'nonce': 3712674634}, {'type': 'v1', 'addr': '172.21.15.191:6835', 'nonce': 3712674634}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}, 'gid_24482': {'gid': 24482, 'name': 'e', 'rank': 4, 'incarnation': 24, 'state': 'up:active', 'state_seq': 8, 'addr': '172.21.15.130:6835/3088490808', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.130:6834', 'nonce': 3088490808}, {'type': 'v1', 'addr': '172.21.15.130:6835', 'nonce': 3088490808}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}, 'gid_24490': {'gid': 24490, 'name': 'c', 'rank': 3, 'incarnation': 22, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.191:6837/2406892975', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.191:6836', 'nonce': 2406892975}, {'type': 'v1', 'addr': '172.21.15.191:6837', 'nonce': 2406892975}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}, 'gid_24506': {'gid': 24506, 'name': 'h', 'rank': 2, 'incarnation': 20, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.130:6837/2604019369', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.130:6836', 'nonce': 2604019369}, {'type': 'v1', 'addr': '172.21.15.130:6837', 'nonce': 2604019369}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}, 'gid_24508': {'gid': 24508, 'name': 'i', 'rank': 1, 'incarnation': 18, 'state': 'up:active', 'state_seq': 5, 'addr': '172.21.15.191:6839/1977507302', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.191:6838', 'nonce': 1977507302}, {'type': 'v1', 'addr': '172.21.15.191:6839', 'nonce': 1977507302}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}, 'gid_24539': {'gid': 24539, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.130:6839/2145479535', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.130:6838', 'nonce': 2145479535}, {'type': 'v1', 'addr': '172.21.15.130:6839', 'nonce': 2145479535}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2023-07-14T19:17:19.707 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 5/5 2023-07-14T19:17:19.707 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.e status 2023-07-14T19:17:19.834 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:19.832+0000 7f2d15b7f700 1 -- 172.21.15.40:0/489485322 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2d100fdee0 msgr2=0x7f2d100fe2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:17:19.834 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:19.832+0000 7f2d15b7f700 1 --2- 172.21.15.40:0/489485322 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2d100fdee0 0x7f2d100fe2f0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f2d00009990 tx=0x7f2d00009ca0 comp rx=0 tx=0).stop 2023-07-14T19:17:19.834 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:19.832+0000 7f2d15b7f700 1 -- 172.21.15.40:0/489485322 shutdown_connections 2023-07-14T19:17:19.834 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:19.832+0000 7f2d15b7f700 1 --2- 172.21.15.40:0/489485322 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2d101018e0 0x7f2d101081a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:19.835 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:19.832+0000 7f2d15b7f700 1 --2- 172.21.15.40:0/489485322 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2d100fdee0 0x7f2d100fe2f0 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:19.835 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:19.832+0000 7f2d15b7f700 1 --2- 172.21.15.40:0/489485322 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2d10100f50 0x7f2d101013a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:19.835 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:19.832+0000 7f2d15b7f700 1 -- 172.21.15.40:0/489485322 >> 172.21.15.40:0/489485322 conn(0x7f2d100f98a0 msgr2=0x7f2d100fbcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:17:19.835 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:19.832+0000 7f2d15b7f700 1 -- 172.21.15.40:0/489485322 shutdown_connections 2023-07-14T19:17:19.835 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:19.832+0000 7f2d15b7f700 1 -- 172.21.15.40:0/489485322 wait complete. 2023-07-14T19:17:19.835 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:19.832+0000 7f2d15b7f700 1 Processor -- start 2023-07-14T19:17:19.837 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:19.833+0000 7f2d15b7f700 1 -- start start 2023-07-14T19:17:19.837 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:19.833+0000 7f2d15b7f700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2d100fdee0 0x7f2d101008c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:17:19.837 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:19.833+0000 7f2d15b7f700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2d10100f50 0x7f2d100fef00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:17:19.838 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:19.833+0000 7f2d15b7f700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2d101018e0 0x7f2d100ff440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:17:19.838 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:19.833+0000 7f2d15b7f700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d100ffbe0 con 0x7f2d100fdee0 2023-07-14T19:17:19.838 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:19.833+0000 7f2d15b7f700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d100ffd50 con 0x7f2d101018e0 2023-07-14T19:17:19.838 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:19.833+0000 7f2d15b7f700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d1019f960 con 0x7f2d10100f50 2023-07-14T19:17:19.838 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:19.833+0000 7f2d0f7fe700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2d100fdee0 0x7f2d101008c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:17:19.838 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:19.833+0000 7f2d0f7fe700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2d100fdee0 0x7f2d101008c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:34146/0 (socket says 172.21.15.40:34146) 2023-07-14T19:17:19.839 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:19.833+0000 7f2d0f7fe700 1 -- 172.21.15.40:0/1468759132 learned_addr learned my addr 172.21.15.40:0/1468759132 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-07-14T19:17:19.839 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:19.833+0000 7f2d0effd700 1 --2- 172.21.15.40:0/1468759132 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2d10100f50 0x7f2d100fef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:17:19.839 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:19.833+0000 7f2d0f7fe700 1 -- 172.21.15.40:0/1468759132 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2d10100f50 msgr2=0x7f2d100fef00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:17:19.839 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:19.833+0000 7f2d0f7fe700 1 --2- 172.21.15.40:0/1468759132 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2d10100f50 0x7f2d100fef00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:19.839 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:19.833+0000 7f2d0f7fe700 1 -- 172.21.15.40:0/1468759132 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2d101018e0 msgr2=0x7f2d100ff440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:17:19.839 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:19.833+0000 7f2d0f7fe700 1 --2- 172.21.15.40:0/1468759132 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2d101018e0 0x7f2d100ff440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:19.839 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:19.833+0000 7f2d0f7fe700 1 -- 172.21.15.40:0/1468759132 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d000095f0 con 0x7f2d100fdee0 2023-07-14T19:17:19.840 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:19.833+0000 7f2d0f7fe700 1 --2- 172.21.15.40:0/1468759132 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2d100fdee0 0x7f2d101008c0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f2d000095c0 tx=0x7f2d0000f690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:17:19.840 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:19.833+0000 7f2d0cff9700 1 -- 172.21.15.40:0/1468759132 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d0001d070 con 0x7f2d100fdee0 2023-07-14T19:17:19.840 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:19.834+0000 7f2d15b7f700 1 -- 172.21.15.40:0/1468759132 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2d1019fbe0 con 0x7f2d100fdee0 2023-07-14T19:17:19.840 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:19.834+0000 7f2d0cff9700 1 -- 172.21.15.40:0/1468759132 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f2d0000fb80 con 0x7f2d100fdee0 2023-07-14T19:17:19.840 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:19.834+0000 7f2d0cff9700 1 -- 172.21.15.40:0/1468759132 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d00017650 con 0x7f2d100fdee0 2023-07-14T19:17:19.840 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:19.834+0000 7f2d15b7f700 1 -- 172.21.15.40:0/1468759132 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2d101a0010 con 0x7f2d100fdee0 2023-07-14T19:17:19.840 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:19.836+0000 7f2d0cff9700 1 -- 172.21.15.40:0/1468759132 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7f2d00022470 con 0x7f2d100fdee0 2023-07-14T19:17:19.846 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:19.836+0000 7f2d0cff9700 1 --2- 172.21.15.40:0/1468759132 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f2cfc0a11e0 0x7f2cfc0a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:17:19.846 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:19.837+0000 7f2d0effd700 1 --2- 172.21.15.40:0/1468759132 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f2cfc0a11e0 0x7f2cfc0a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:17:19.846 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:19.837+0000 7f2d0cff9700 1 -- 172.21.15.40:0/1468759132 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 8269+0+0 (secure 0 0 0) 0x7f2d000ce000 con 0x7f2d100fdee0 2023-07-14T19:17:19.847 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:19.837+0000 7f2d0effd700 1 --2- 172.21.15.40:0/1468759132 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f2cfc0a11e0 0x7f2cfc0a3690 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f2cf8005950 tx=0x7f2cf80058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:17:19.847 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:19.837+0000 7f2d21d17b80 1 Processor -- start 2023-07-14T19:17:19.847 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:19.837+0000 7f2d21d17b80 1 -- start start 2023-07-14T19:17:19.847 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:19.838+0000 7f2d21d17b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55eca27fef10 0x55eca27ff320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:17:19.847 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:19.838+0000 7f2d21d17b80 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55eca27ff860 0x55eca27ffe80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:17:19.847 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:19.838+0000 7f2d21d17b80 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55eca2800460 0x55eca28008d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:17:19.847 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:19.838+0000 7f2d21d17b80 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x55eca2703c60 con 0x55eca27ff860 2023-07-14T19:17:19.848 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:19.838+0000 7f2d21d17b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55eca27569f0 con 0x55eca27fef10 2023-07-14T19:17:19.848 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:19.838+0000 7f2d21d17b80 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x55eca274fe80 con 0x55eca2800460 2023-07-14T19:17:19.848 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:19.838+0000 7f2d0effd700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55eca27ff860 0x55eca27ffe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:17:19.848 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:19.838+0000 7f2d0effd700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55eca27ff860 0x55eca27ffe80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:34156/0 (socket says 172.21.15.40:34156) 2023-07-14T19:17:19.848 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:19.838+0000 7f2d0effd700 1 -- 172.21.15.40:0/3510121755 learned_addr learned my addr 172.21.15.40:0/3510121755 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-07-14T19:17:19.848 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:19.838+0000 7f2d0ffff700 1 --2- 172.21.15.40:0/3510121755 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55eca2800460 0x55eca28008d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:17:19.848 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:19.838+0000 7f2d0f7fe700 1 --2- 172.21.15.40:0/3510121755 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55eca27fef10 0x55eca27ff320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:17:19.849 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:19.838+0000 7f2d0effd700 1 -- 172.21.15.40:0/3510121755 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55eca2800460 msgr2=0x55eca28008d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T19:17:19.849 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:19.838+0000 7f2d0effd700 1 --2- 172.21.15.40:0/3510121755 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55eca2800460 0x55eca28008d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:19.849 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:19.838+0000 7f2d0effd700 1 -- 172.21.15.40:0/3510121755 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55eca27fef10 msgr2=0x55eca27ff320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T19:17:19.849 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:19.838+0000 7f2d0effd700 1 --2- 172.21.15.40:0/3510121755 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55eca27fef10 0x55eca27ff320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:19.849 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:19.838+0000 7f2d0effd700 1 -- 172.21.15.40:0/3510121755 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55eca257dd90 con 0x55eca27ff860 2023-07-14T19:17:19.849 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:19.838+0000 7f2d0f7fe700 1 --2- 172.21.15.40:0/3510121755 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55eca27fef10 0x55eca27ff320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T19:17:19.850 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:19.838+0000 7f2d0effd700 1 --2- 172.21.15.40:0/3510121755 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55eca27ff860 0x55eca27ffe80 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f2cf800fd70 tx=0x7f2cf8012000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=564941d16ddc9a2a server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:17:19.850 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:19.838+0000 7f2d05ffb700 1 -- 172.21.15.40:0/3510121755 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cf8011bb0 con 0x55eca27ff860 2023-07-14T19:17:19.851 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:19.839+0000 7f2d05ffb700 1 -- 172.21.15.40:0/3510121755 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f2cf8011d10 con 0x55eca27ff860 2023-07-14T19:17:19.851 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:19.839+0000 7f2d05ffb700 1 -- 172.21.15.40:0/3510121755 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cf801d6e0 con 0x55eca27ff860 2023-07-14T19:17:19.851 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:19.839+0000 7f2d21d17b80 1 -- 172.21.15.40:0/3510121755 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55eca27ff860 msgr2=0x55eca27ffe80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:17:19.851 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:19.839+0000 7f2d21d17b80 1 --2- 172.21.15.40:0/3510121755 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55eca27ff860 0x55eca27ffe80 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f2cf800fd70 tx=0x7f2cf8012000 comp rx=0 tx=0).stop 2023-07-14T19:17:19.851 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:19.839+0000 7f2d21d17b80 1 -- 172.21.15.40:0/3510121755 shutdown_connections 2023-07-14T19:17:19.851 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:19.839+0000 7f2d21d17b80 1 --2- 172.21.15.40:0/3510121755 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55eca2800460 0x55eca28008d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:19.851 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:19.839+0000 7f2d21d17b80 1 --2- 172.21.15.40:0/3510121755 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55eca27fef10 0x55eca27ff320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:19.852 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:19.839+0000 7f2d21d17b80 1 --2- 172.21.15.40:0/3510121755 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55eca27ff860 0x55eca27ffe80 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:19.852 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:19.839+0000 7f2d21d17b80 1 -- 172.21.15.40:0/3510121755 >> 172.21.15.40:0/3510121755 conn(0x55eca27f4630 msgr2=0x55eca27f9980 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:17:19.852 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:19.839+0000 7f2d21d17b80 1 -- 172.21.15.40:0/3510121755 shutdown_connections 2023-07-14T19:17:19.852 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:19.839+0000 7f2d21d17b80 1 -- 172.21.15.40:0/3510121755 wait complete. 2023-07-14T19:17:19.852 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:19.840+0000 7f2d21d17b80 1 -- start start 2023-07-14T19:17:19.852 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:19.840+0000 7f2d21d17b80 1 Processor -- start 2023-07-14T19:17:19.852 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:19.840+0000 7f2d21d17b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55eca27fef10 0x55eca28088d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:17:19.853 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:19.840+0000 7f2d21d17b80 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55eca27ff860 0x55eca2808e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:17:19.853 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:19.840+0000 7f2d21d17b80 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55eca2800460 0x55eca280fe50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:17:19.853 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:19.840+0000 7f2d21d17b80 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x55eca26efa70 con 0x55eca2800460 2023-07-14T19:17:19.853 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:19.840+0000 7f2d21d17b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55eca26ef770 con 0x55eca27fef10 2023-07-14T19:17:19.853 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:19.840+0000 7f2d21d17b80 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x55eca25d38d0 con 0x55eca27ff860 2023-07-14T19:17:19.853 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:19.840+0000 7f2d21d17b80 10 client.0 _ll_register_callbacks cb 0x55eca26ed710 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-14T19:17:19.853 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:19.840+0000 7f2d0ffff700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55eca2800460 0x55eca280fe50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:17:19.854 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:19.840+0000 7f2d0ffff700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55eca2800460 0x55eca280fe50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:34164/0 (socket says 172.21.15.40:34164) 2023-07-14T19:17:19.854 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:19.840+0000 7f2d0ffff700 1 -- 172.21.15.40:0/636708827 learned_addr learned my addr 172.21.15.40:0/636708827 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-07-14T19:17:19.854 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:19.840+0000 7f2d0f7fe700 1 --2- 172.21.15.40:0/636708827 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55eca27fef10 0x55eca28088d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:17:19.854 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:19.840+0000 7f2d0effd700 1 --2- 172.21.15.40:0/636708827 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55eca27ff860 0x55eca2808e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:17:19.854 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:19.841+0000 7f2d0ffff700 1 -- 172.21.15.40:0/636708827 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55eca27ff860 msgr2=0x55eca2808e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T19:17:19.854 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:19.841+0000 7f2d0ffff700 1 --2- 172.21.15.40:0/636708827 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55eca27ff860 0x55eca2808e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:19.854 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:19.841+0000 7f2d0ffff700 1 -- 172.21.15.40:0/636708827 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55eca27fef10 msgr2=0x55eca28088d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T19:17:19.854 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:19.841+0000 7f2d0ffff700 1 --2- 172.21.15.40:0/636708827 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55eca27fef10 0x55eca28088d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:19.855 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:19.841+0000 7f2d0ffff700 1 -- 172.21.15.40:0/636708827 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2cf800fa20 con 0x55eca2800460 2023-07-14T19:17:19.855 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:19.841+0000 7f2d0f7fe700 1 --2- 172.21.15.40:0/636708827 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55eca27fef10 0x55eca28088d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T19:17:19.855 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:19.841+0000 7f2d0effd700 1 --2- 172.21.15.40:0/636708827 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55eca27ff860 0x55eca2808e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T19:17:19.855 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:19.841+0000 7f2d0ffff700 1 --2- 172.21.15.40:0/636708827 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55eca2800460 0x55eca280fe50 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f2d0800fc10 tx=0x7f2d08008ea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f9faefaf2b33e9c9 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:17:19.855 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:19.841+0000 7f2ceeffd700 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2023-07-14T19:17:19.855 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:19.841+0000 7f2ceeffd700 1 -- 172.21.15.40:0/636708827 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d0800d8b0 con 0x55eca2800460 2023-07-14T19:17:19.855 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:19.841+0000 7f2ceeffd700 1 -- 172.21.15.40:0/636708827 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(83..83 src has 1..83) v4 ==== 8269+0+0 (secure 0 0 0) 0x7f2d0800da10 con 0x55eca2800460 2023-07-14T19:17:19.856 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:19.841+0000 7f2d21d17b80 1 -- 172.21.15.40:0/636708827 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55eca26a94a0 con 0x55eca2800460 2023-07-14T19:17:19.856 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:19.841+0000 7f2ceeffd700 1 -- 172.21.15.40:0/636708827 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2d0800dbd0 con 0x55eca2800460 2023-07-14T19:17:19.856 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:19.841+0000 7f2d21d17b80 10 client.15204 fetch_fsmap learned FSMap version 26 2023-07-14T19:17:19.856 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:19.841+0000 7f2d21d17b80 1 -- 172.21.15.40:0/636708827 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) v3 -- 0x55eca2566c10 con 0x55eca2800460 2023-07-14T19:17:19.856 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:19.841+0000 7f2ceeffd700 1 -- 172.21.15.40:0/636708827 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f2d08025020 con 0x55eca2800460 2023-07-14T19:17:19.856 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:19.841+0000 7f2ceeffd700 1 -- 172.21.15.40:0/636708827 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 26) v1 ==== 7462+0+0 (secure 0 0 0) 0x7f2d0801f400 con 0x55eca2800460 2023-07-14T19:17:19.856 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:19.841+0000 7f2d21d17b80 10 client.15204 fetch_fsmap finished waiting for FSMap version 26 2023-07-14T19:17:19.856 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:19.842+0000 7f2d21d17b80 10 client.15204 resolve_mds: resolved name 'e' to daemon mds.e 2023-07-14T19:17:19.857 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:19.842+0000 7f2d21d17b80 20 client.15204 populate_metadata read hostname 'smithi040' 2023-07-14T19:17:19.857 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:19.842+0000 7f2d21d17b80 1 --2- 172.21.15.40:0/636708827 >> [v2:172.21.15.130:6834/3088490808,v1:172.21.15.130:6835/3088490808] conn(0x55eca2813c50 0x55eca2813000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:17:19.857 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:19.842+0000 7f2d21d17b80 4 client.15204 mds_command: new command op to 24482 tid=0[{"prefix": "get_command_descriptions"}] 2023-07-14T19:17:19.857 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:19.842+0000 7f2d21d17b80 1 -- 172.21.15.40:0/636708827 --> [v2:172.21.15.130:6834/3088490808,v1:172.21.15.130:6835/3088490808] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55eca27cdd50 con 0x55eca2813c50 2023-07-14T19:17:19.857 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:19.842+0000 7f2d0f7fe700 1 --2- 172.21.15.40:0/636708827 >> [v2:172.21.15.130:6834/3088490808,v1:172.21.15.130:6835/3088490808] conn(0x55eca2813c50 0x55eca2813000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:17:19.857 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:19.842+0000 7f2d0f7fe700 1 --2- 172.21.15.40:0/636708827 >> [v2:172.21.15.130:6834/3088490808,v1:172.21.15.130:6835/3088490808] conn(0x55eca2813c50 0x55eca2813000 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=7649608d3da012fb server_cookie=c882850d3fd37aeb in_seq=0 out_seq=0 2023-07-14T19:17:19.857 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:19.842+0000 7f2ceeffd700 10 client.15204 ms_handle_connect on v2:172.21.15.130:6834/3088490808 2023-07-14T19:17:19.858 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:19.843+0000 7f2ceeffd700 1 -- 172.21.15.40:0/636708827 <== mds.4 v2:172.21.15.130:6834/3088490808 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18017 (crc 0 0 0) 0x55eca27cdd50 con 0x55eca2813c50 2023-07-14T19:17:19.858 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:19.843+0000 7f2ceeffd700 10 client.15204 handle_command_reply: tid=0 2023-07-14T19:17:19.858 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:19.843+0000 7f2d21d17b80 1 client.15204 shutdown 2023-07-14T19:17:19.858 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:19.843+0000 7f2d21d17b80 1 -- 172.21.15.40:0/636708827 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55eca2800460 msgr2=0x55eca280fe50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:17:19.858 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:19.843+0000 7f2d21d17b80 1 --2- 172.21.15.40:0/636708827 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55eca2800460 0x55eca280fe50 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f2d0800fc10 tx=0x7f2d08008ea0 comp rx=0 tx=0).stop 2023-07-14T19:17:19.858 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:19.843+0000 7f2d21d17b80 1 -- 172.21.15.40:0/636708827 shutdown_connections 2023-07-14T19:17:19.858 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:19.843+0000 7f2d21d17b80 1 --2- 172.21.15.40:0/636708827 >> [v2:172.21.15.130:6834/3088490808,v1:172.21.15.130:6835/3088490808] conn(0x55eca2813c50 0x55eca2813000 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:19.858 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:19.843+0000 7f2d21d17b80 1 --2- 172.21.15.40:0/636708827 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55eca2800460 0x55eca280fe50 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:19.859 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:19.843+0000 7f2d21d17b80 1 --2- 172.21.15.40:0/636708827 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55eca27fef10 0x55eca28088d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:19.859 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:19.843+0000 7f2ceeffd700 0 client.15204 ms_handle_reset on v2:172.21.15.130:6834/3088490808 2023-07-14T19:17:19.859 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:19.843+0000 7f2d21d17b80 1 --2- 172.21.15.40:0/636708827 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55eca27ff860 0x55eca2808e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:19.859 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:19.843+0000 7f2d21d17b80 1 -- 172.21.15.40:0/636708827 >> 172.21.15.40:0/636708827 conn(0x55eca27f4630 msgr2=0x55eca27f8600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:17:19.859 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:19.843+0000 7f2d21d17b80 1 -- 172.21.15.40:0/636708827 shutdown_connections 2023-07-14T19:17:19.859 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:19.843+0000 7f2d21d17b80 1 -- 172.21.15.40:0/636708827 wait complete. 2023-07-14T19:17:19.859 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:19.843+0000 7f2d21d17b80 20 client.15204 trim_cache size 0 max 16384 2023-07-14T19:17:19.863 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:19.861+0000 7f2d21d17b80 1 Processor -- start 2023-07-14T19:17:19.864 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:19.861+0000 7f2d21d17b80 1 -- start start 2023-07-14T19:17:19.864 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:19.861+0000 7f2d21d17b80 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55eca27fef10 0x55eca27f5b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:17:19.864 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:19.861+0000 7f2d21d17b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55eca27ff860 0x55eca281c9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:17:19.864 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:19.861+0000 7f2d21d17b80 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55eca2800460 0x55eca27f90a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:17:19.865 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:19.861+0000 7f2d21d17b80 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x55eca2800d50 con 0x55eca27fef10 2023-07-14T19:17:19.865 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:19.861+0000 7f2d21d17b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55eca2807d80 con 0x55eca27ff860 2023-07-14T19:17:19.865 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:19.861+0000 7f2d21d17b80 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x55eca27fb470 con 0x55eca2800460 2023-07-14T19:17:19.865 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:19.861+0000 7f2d0f7fe700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55eca27fef10 0x55eca27f5b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:17:19.865 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:19.861+0000 7f2d0f7fe700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55eca27fef10 0x55eca27f5b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:34168/0 (socket says 172.21.15.40:34168) 2023-07-14T19:17:19.865 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:19.861+0000 7f2d0f7fe700 1 -- 172.21.15.40:0/2368547142 learned_addr learned my addr 172.21.15.40:0/2368547142 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-07-14T19:17:19.865 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:19.862+0000 7f2d0f7fe700 1 -- 172.21.15.40:0/2368547142 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55eca2800460 msgr2=0x55eca27f90a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T19:17:19.866 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:19.862+0000 7f2d0f7fe700 1 --2- 172.21.15.40:0/2368547142 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55eca2800460 0x55eca27f90a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:19.866 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:19.862+0000 7f2d0f7fe700 1 -- 172.21.15.40:0/2368547142 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55eca27ff860 msgr2=0x55eca281c9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T19:17:19.866 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:19.862+0000 7f2d0f7fe700 1 --2- 172.21.15.40:0/2368547142 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55eca27ff860 0x55eca281c9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:19.866 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:19.862+0000 7f2d0f7fe700 1 -- 172.21.15.40:0/2368547142 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55eca25a4200 con 0x55eca27fef10 2023-07-14T19:17:19.866 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:19.862+0000 7f2d0f7fe700 1 --2- 172.21.15.40:0/2368547142 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55eca27fef10 0x55eca27f5b00 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f2d0009b020 tx=0x7f2d00009310 comp rx=0 tx=0).ready entity=mon.0 client_cookie=24fd44384b22a09 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:17:19.866 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:19.862+0000 7f2d067fc700 1 -- 172.21.15.40:0/2368547142 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d0009b320 con 0x55eca27fef10 2023-07-14T19:17:19.866 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:19.862+0000 7f2d067fc700 1 -- 172.21.15.40:0/2368547142 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f2d0009f350 con 0x55eca27fef10 2023-07-14T19:17:19.866 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:19.862+0000 7f2d067fc700 1 -- 172.21.15.40:0/2368547142 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d0009b580 con 0x55eca27fef10 2023-07-14T19:17:19.870 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:19.862+0000 7f2d21d17b80 1 -- 172.21.15.40:0/2368547142 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55eca27fef10 msgr2=0x55eca27f5b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:17:19.871 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:19.862+0000 7f2d21d17b80 1 --2- 172.21.15.40:0/2368547142 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55eca27fef10 0x55eca27f5b00 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f2d0009b020 tx=0x7f2d00009310 comp rx=0 tx=0).stop 2023-07-14T19:17:19.871 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:19.862+0000 7f2d21d17b80 1 -- 172.21.15.40:0/2368547142 shutdown_connections 2023-07-14T19:17:19.872 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:19.862+0000 7f2d21d17b80 1 --2- 172.21.15.40:0/2368547142 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55eca2800460 0x55eca27f90a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:19.872 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:19.862+0000 7f2d21d17b80 1 --2- 172.21.15.40:0/2368547142 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55eca27fef10 0x55eca27f5b00 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:19.872 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:19.862+0000 7f2d21d17b80 1 --2- 172.21.15.40:0/2368547142 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55eca27ff860 0x55eca281c9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:19.872 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:19.862+0000 7f2d21d17b80 1 -- 172.21.15.40:0/2368547142 >> 172.21.15.40:0/2368547142 conn(0x55eca27f4630 msgr2=0x55eca27f9950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:17:19.873 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:19.863+0000 7f2d21d17b80 1 -- 172.21.15.40:0/2368547142 shutdown_connections 2023-07-14T19:17:19.873 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:19.863+0000 7f2d21d17b80 1 -- 172.21.15.40:0/2368547142 wait complete. 2023-07-14T19:17:19.873 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:19.863+0000 7f2d21d17b80 1 -- start start 2023-07-14T19:17:19.873 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:19.863+0000 7f2d21d17b80 1 Processor -- start 2023-07-14T19:17:19.873 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:19.863+0000 7f2d21d17b80 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55eca27fef10 0x55eca27f9800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:17:19.873 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:19.863+0000 7f2d21d17b80 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55eca27ff860 0x55eca281c9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:17:19.873 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:19.863+0000 7f2d21d17b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55eca2800460 0x55eca28678f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:17:19.874 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:19.863+0000 7f2d21d17b80 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x55eca2867e30 con 0x55eca27ff860 2023-07-14T19:17:19.874 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:19.863+0000 7f2d21d17b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55eca2867f70 con 0x55eca2800460 2023-07-14T19:17:19.874 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:19.863+0000 7f2d21d17b80 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x55eca2868100 con 0x55eca27fef10 2023-07-14T19:17:19.874 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:19.863+0000 7f2d21d17b80 10 client.0 _ll_register_callbacks cb 0x7f2d0800f500 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-14T19:17:19.874 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:19.863+0000 7f2d0effd700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55eca27ff860 0x55eca281c9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:17:19.874 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:19.863+0000 7f2d0effd700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55eca27ff860 0x55eca281c9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:34176/0 (socket says 172.21.15.40:34176) 2023-07-14T19:17:19.874 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:19.863+0000 7f2d0effd700 1 -- 172.21.15.40:0/2213959620 learned_addr learned my addr 172.21.15.40:0/2213959620 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-07-14T19:17:19.875 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:19.864+0000 7f2d0f7fe700 1 --2- 172.21.15.40:0/2213959620 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55eca27fef10 0x55eca27f9800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:17:19.875 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:19.864+0000 7f2d0ffff700 1 --2- 172.21.15.40:0/2213959620 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55eca2800460 0x55eca28678f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:17:19.875 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:19.864+0000 7f2d0effd700 1 -- 172.21.15.40:0/2213959620 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55eca27fef10 msgr2=0x55eca27f9800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T19:17:19.875 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:19.864+0000 7f2d0effd700 1 --2- 172.21.15.40:0/2213959620 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55eca27fef10 0x55eca27f9800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:19.875 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:19.864+0000 7f2d0effd700 1 -- 172.21.15.40:0/2213959620 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55eca2800460 msgr2=0x55eca28678f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T19:17:19.875 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:19.864+0000 7f2d0effd700 1 --2- 172.21.15.40:0/2213959620 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55eca2800460 0x55eca28678f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:19.875 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:19.864+0000 7f2d0effd700 1 -- 172.21.15.40:0/2213959620 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2d101a0010 con 0x55eca27ff860 2023-07-14T19:17:19.875 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:19.864+0000 7f2d0f7fe700 1 --2- 172.21.15.40:0/2213959620 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55eca27fef10 0x55eca27f9800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T19:17:19.876 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:19.864+0000 7f2d0ffff700 1 --2- 172.21.15.40:0/2213959620 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55eca2800460 0x55eca28678f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T19:17:19.876 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:19.864+0000 7f2d0effd700 1 --2- 172.21.15.40:0/2213959620 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55eca27ff860 0x55eca281c9e0 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f2cf800fd10 tx=0x7f2cf800ffd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8b232d88b87a5d38 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:17:19.876 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:19.864+0000 7f2ceeffd700 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2023-07-14T19:17:19.876 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:19.864+0000 7f2ceeffd700 1 -- 172.21.15.40:0/2213959620 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cf802e040 con 0x55eca27ff860 2023-07-14T19:17:19.876 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:19.864+0000 7f2d21d17b80 1 -- 172.21.15.40:0/2213959620 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f2cf8011d10 con 0x55eca27ff860 2023-07-14T19:17:19.876 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:19.864+0000 7f2ceeffd700 1 -- 172.21.15.40:0/2213959620 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(83..83 src has 1..83) v4 ==== 8269+0+0 (secure 0 0 0) 0x7f2cf8004710 con 0x55eca27ff860 2023-07-14T19:17:19.876 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:19.864+0000 7f2ceeffd700 1 -- 172.21.15.40:0/2213959620 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2cf80048d0 con 0x55eca27ff860 2023-07-14T19:17:19.877 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:19.864+0000 7f2d21d17b80 10 client.15210 fetch_fsmap learned FSMap version 26 2023-07-14T19:17:19.877 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:19.864+0000 7f2d21d17b80 1 -- 172.21.15.40:0/2213959620 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) v3 -- 0x55eca257c260 con 0x55eca27ff860 2023-07-14T19:17:19.877 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:19.864+0000 7f2ceeffd700 1 -- 172.21.15.40:0/2213959620 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f2cf8042080 con 0x55eca27ff860 2023-07-14T19:17:19.877 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:19.865+0000 7f2ceeffd700 1 -- 172.21.15.40:0/2213959620 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 26) v1 ==== 7462+0+0 (secure 0 0 0) 0x7f2cf803c030 con 0x55eca27ff860 2023-07-14T19:17:19.877 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:19.865+0000 7f2d21d17b80 10 client.15210 fetch_fsmap finished waiting for FSMap version 26 2023-07-14T19:17:19.877 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:19.865+0000 7f2d21d17b80 10 client.15210 resolve_mds: resolved name 'e' to daemon mds.e 2023-07-14T19:17:19.877 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:19.865+0000 7f2d21d17b80 20 client.15210 populate_metadata read hostname 'smithi040' 2023-07-14T19:17:19.877 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:19.865+0000 7f2d21d17b80 1 --2- 172.21.15.40:0/2213959620 >> [v2:172.21.15.130:6834/3088490808,v1:172.21.15.130:6835/3088490808] conn(0x55eca286ce40 0x55eca2869810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:17:19.878 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:19.865+0000 7f2d21d17b80 4 client.15210 mds_command: new command op to 24482 tid=0[{"prefix": "status"}] 2023-07-14T19:17:19.878 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:19.865+0000 7f2d21d17b80 1 -- 172.21.15.40:0/2213959620 --> [v2:172.21.15.130:6834/3088490808,v1:172.21.15.130:6835/3088490808] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55eca2748e60 con 0x55eca286ce40 2023-07-14T19:17:19.878 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:19.865+0000 7f2d0f7fe700 1 --2- 172.21.15.40:0/2213959620 >> [v2:172.21.15.130:6834/3088490808,v1:172.21.15.130:6835/3088490808] conn(0x55eca286ce40 0x55eca2869810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:17:19.878 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:19.865+0000 7f2ceeffd700 10 client.15210 ms_handle_connect on v2:172.21.15.130:6834/3088490808 2023-07-14T19:17:19.878 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:19.865+0000 7f2d0f7fe700 1 --2- 172.21.15.40:0/2213959620 >> [v2:172.21.15.130:6834/3088490808,v1:172.21.15.130:6835/3088490808] conn(0x55eca286ce40 0x55eca2869810 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=af85b526a85bd1dc server_cookie=1f6a816d1504d408 in_seq=0 out_seq=0 2023-07-14T19:17:19.878 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:19.866+0000 7f2ceeffd700 1 -- 172.21.15.40:0/2213959620 <== mds.4 v2:172.21.15.130:6834/3088490808 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+332 (crc 0 0 0) 0x55eca2748e60 con 0x55eca286ce40 2023-07-14T19:17:19.878 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:19.866+0000 7f2ceeffd700 10 client.15210 handle_command_reply: tid=0 2023-07-14T19:17:19.879 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:19.866+0000 7f2d21d17b80 1 client.15210 shutdown 2023-07-14T19:17:19.879 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:19.866+0000 7f2d21d17b80 1 -- 172.21.15.40:0/2213959620 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55eca27ff860 msgr2=0x55eca281c9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:17:19.879 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:19.866+0000 7f2d21d17b80 1 --2- 172.21.15.40:0/2213959620 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55eca27ff860 0x55eca281c9e0 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f2cf800fd10 tx=0x7f2cf800ffd0 comp rx=0 tx=0).stop 2023-07-14T19:17:19.879 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:19.866+0000 7f2d21d17b80 1 -- 172.21.15.40:0/2213959620 shutdown_connections 2023-07-14T19:17:19.879 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:19.866+0000 7f2d21d17b80 1 --2- 172.21.15.40:0/2213959620 >> [v2:172.21.15.130:6834/3088490808,v1:172.21.15.130:6835/3088490808] conn(0x55eca286ce40 0x55eca2869810 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:19.879 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:19.866+0000 7f2d21d17b80 1 --2- 172.21.15.40:0/2213959620 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55eca27fef10 0x55eca27f9800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:19.879 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:19.866+0000 7f2d21d17b80 1 --2- 172.21.15.40:0/2213959620 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55eca2800460 0x55eca28678f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:19.879 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:19.866+0000 7f2ceeffd700 0 client.15210 ms_handle_reset on v2:172.21.15.130:6834/3088490808 2023-07-14T19:17:19.880 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:19.866+0000 7f2d21d17b80 1 --2- 172.21.15.40:0/2213959620 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55eca27ff860 0x55eca281c9e0 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:19.880 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:19.866+0000 7f2d21d17b80 1 -- 172.21.15.40:0/2213959620 >> 172.21.15.40:0/2213959620 conn(0x55eca27f4630 msgr2=0x55eca28067d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:17:19.880 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:19.866+0000 7f2d21d17b80 1 -- 172.21.15.40:0/2213959620 shutdown_connections 2023-07-14T19:17:19.880 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:19.866+0000 7f2d21d17b80 1 -- 172.21.15.40:0/2213959620 wait complete. 2023-07-14T19:17:19.880 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:19.866+0000 7f2d21d17b80 20 client.15210 trim_cache size 0 max 16384 2023-07-14T19:17:19.881 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2023-07-14T19:17:19.881 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_fsid": "adc52f04-2279-11ee-9b33-001a4aab830c", 2023-07-14T19:17:19.881 INFO:teuthology.orchestra.run.smithi040.stdout: "whoami": 4, 2023-07-14T19:17:19.881 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 24482, 2023-07-14T19:17:19.881 INFO:teuthology.orchestra.run.smithi040.stdout: "want_state": "up:active", 2023-07-14T19:17:19.881 INFO:teuthology.orchestra.run.smithi040.stdout: "state": "up:active", 2023-07-14T19:17:19.881 INFO:teuthology.orchestra.run.smithi040.stdout: "fs_name": "cephfs", 2023-07-14T19:17:19.882 INFO:teuthology.orchestra.run.smithi040.stdout: "rank_uptime": 75.257967402999995, 2023-07-14T19:17:19.882 INFO:teuthology.orchestra.run.smithi040.stdout: "mdsmap_epoch": 26, 2023-07-14T19:17:19.882 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap_epoch": 83, 2023-07-14T19:17:19.882 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap_epoch_barrier": 81, 2023-07-14T19:17:19.882 INFO:teuthology.orchestra.run.smithi040.stdout: "uptime": 102.85906861300001 2023-07-14T19:17:19.882 INFO:teuthology.orchestra.run.smithi040.stdout:} 2023-07-14T19:17:19.883 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:19.867+0000 7f2d15b7f700 1 -- 172.21.15.40:0/1468759132 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f2cfc0a11e0 msgr2=0x7f2cfc0a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:17:19.883 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:19.867+0000 7f2d15b7f700 1 --2- 172.21.15.40:0/1468759132 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f2cfc0a11e0 0x7f2cfc0a3690 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f2cf8005950 tx=0x7f2cf80058e0 comp rx=0 tx=0).stop 2023-07-14T19:17:19.884 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:19.867+0000 7f2d15b7f700 1 -- 172.21.15.40:0/1468759132 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2d100fdee0 msgr2=0x7f2d101008c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:17:19.884 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:19.867+0000 7f2d15b7f700 1 --2- 172.21.15.40:0/1468759132 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2d100fdee0 0x7f2d101008c0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f2d000095c0 tx=0x7f2d0000f690 comp rx=0 tx=0).stop 2023-07-14T19:17:19.884 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:19.867+0000 7f2d15b7f700 1 -- 172.21.15.40:0/1468759132 shutdown_connections 2023-07-14T19:17:19.884 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:19.867+0000 7f2d15b7f700 1 --2- 172.21.15.40:0/1468759132 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f2cfc0a11e0 0x7f2cfc0a3690 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:19.884 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:19.867+0000 7f2d15b7f700 1 --2- 172.21.15.40:0/1468759132 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2d101018e0 0x7f2d100ff440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:19.884 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:19.867+0000 7f2d15b7f700 1 --2- 172.21.15.40:0/1468759132 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2d100fdee0 0x7f2d101008c0 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:19.884 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:19.867+0000 7f2d15b7f700 1 --2- 172.21.15.40:0/1468759132 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f2d10100f50 0x7f2d100fef00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:19.885 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:19.867+0000 7f2d15b7f700 1 -- 172.21.15.40:0/1468759132 >> 172.21.15.40:0/1468759132 conn(0x7f2d100f98a0 msgr2=0x7f2d100fbb30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:17:19.885 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:19.867+0000 7f2d15b7f700 1 -- 172.21.15.40:0/1468759132 shutdown_connections 2023-07-14T19:17:19.885 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:19.867+0000 7f2d15b7f700 1 -- 172.21.15.40:0/1468759132 wait complete. 2023-07-14T19:17:19.885 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2023-07-14T19:17:19.970 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:17:19 smithi040 ceph-mon[96518]: pgmap v297: 97 pgs: 97 active+clean; 638 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2023-07-14T19:17:19.971 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:17:19 smithi040 ceph-mon[96518]: from='client.? 172.21.15.40:0/3423596551' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-07-14T19:17:19.971 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:17:19 smithi040 ceph-mon[96518]: from='client.? 172.21.15.40:0/2362745860' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T19:17:19.971 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:17:19 smithi040 ceph-mon[96518]: from='client.? 172.21.15.40:0/420793535' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T19:17:20.013 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.011+0000 7f9a65c94700 1 -- 172.21.15.40:0/3843418926 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9a60101200 msgr2=0x7f9a60101610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:17:20.013 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.011+0000 7f9a65c94700 1 --2- 172.21.15.40:0/3843418926 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9a60101200 0x7f9a60101610 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f9a54009990 tx=0x7f9a54009ca0 comp rx=0 tx=0).stop 2023-07-14T19:17:20.013 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.011+0000 7f9a65c94700 1 -- 172.21.15.40:0/3843418926 shutdown_connections 2023-07-14T19:17:20.013 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.011+0000 7f9a65c94700 1 --2- 172.21.15.40:0/3843418926 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9a600fe620 0x7f9a601081e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:20.013 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.011+0000 7f9a65c94700 1 --2- 172.21.15.40:0/3843418926 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9a60101200 0x7f9a60101610 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:20.013 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.011+0000 7f9a65c94700 1 --2- 172.21.15.40:0/3843418926 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9a600fdc70 0x7f9a600fe0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:20.014 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.011+0000 7f9a65c94700 1 -- 172.21.15.40:0/3843418926 >> 172.21.15.40:0/3843418926 conn(0x7f9a600f98a0 msgr2=0x7f9a600fbcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:17:20.014 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.011+0000 7f9a65c94700 1 -- 172.21.15.40:0/3843418926 shutdown_connections 2023-07-14T19:17:20.014 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.011+0000 7f9a65c94700 1 -- 172.21.15.40:0/3843418926 wait complete. 2023-07-14T19:17:20.014 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.011+0000 7f9a65c94700 1 Processor -- start 2023-07-14T19:17:20.016 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.012+0000 7f9a65c94700 1 -- start start 2023-07-14T19:17:20.016 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.012+0000 7f9a65c94700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9a600fdc70 0x7f9a60100900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:17:20.016 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.012+0000 7f9a65c94700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9a600fe620 0x7f9a600fef40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:17:20.016 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.012+0000 7f9a65c94700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9a60101200 0x7f9a600ff480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:17:20.016 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.012+0000 7f9a65c94700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a600ffc20 con 0x7f9a600fdc70 2023-07-14T19:17:20.016 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.012+0000 7f9a65c94700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a600ffd90 con 0x7f9a60101200 2023-07-14T19:17:20.016 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.012+0000 7f9a65c94700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a6019f9a0 con 0x7f9a600fe620 2023-07-14T19:17:20.017 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.012+0000 7f9a5f7fe700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9a600fdc70 0x7f9a60100900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:17:20.017 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.012+0000 7f9a5f7fe700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9a600fdc70 0x7f9a60100900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:34188/0 (socket says 172.21.15.40:34188) 2023-07-14T19:17:20.017 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.012+0000 7f9a5f7fe700 1 -- 172.21.15.40:0/1763155831 learned_addr learned my addr 172.21.15.40:0/1763155831 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-07-14T19:17:20.017 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.012+0000 7f9a5effd700 1 --2- 172.21.15.40:0/1763155831 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9a600fe620 0x7f9a600fef40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:17:20.017 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.012+0000 7f9a5ffff700 1 --2- 172.21.15.40:0/1763155831 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9a60101200 0x7f9a600ff480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:17:20.017 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.012+0000 7f9a5f7fe700 1 -- 172.21.15.40:0/1763155831 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9a600fe620 msgr2=0x7f9a600fef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:17:20.017 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.012+0000 7f9a5f7fe700 1 --2- 172.21.15.40:0/1763155831 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9a600fe620 0x7f9a600fef40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:20.018 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.012+0000 7f9a5f7fe700 1 -- 172.21.15.40:0/1763155831 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9a60101200 msgr2=0x7f9a600ff480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:17:20.018 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.012+0000 7f9a5f7fe700 1 --2- 172.21.15.40:0/1763155831 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9a60101200 0x7f9a600ff480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:20.018 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.012+0000 7f9a5f7fe700 1 -- 172.21.15.40:0/1763155831 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a540095f0 con 0x7f9a600fdc70 2023-07-14T19:17:20.018 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.012+0000 7f9a5f7fe700 1 --2- 172.21.15.40:0/1763155831 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9a600fdc70 0x7f9a60100900 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f9a540095c0 tx=0x7f9a5400f690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:17:20.018 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.012+0000 7f9a5cff9700 1 -- 172.21.15.40:0/1763155831 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a5401d070 con 0x7f9a600fdc70 2023-07-14T19:17:20.018 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.012+0000 7f9a5cff9700 1 -- 172.21.15.40:0/1763155831 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f9a5400fb80 con 0x7f9a600fdc70 2023-07-14T19:17:20.018 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.012+0000 7f9a65c94700 1 -- 172.21.15.40:0/1763155831 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9a6019fc20 con 0x7f9a600fdc70 2023-07-14T19:17:20.018 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.013+0000 7f9a5cff9700 1 -- 172.21.15.40:0/1763155831 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a54017650 con 0x7f9a600fdc70 2023-07-14T19:17:20.019 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.013+0000 7f9a65c94700 1 -- 172.21.15.40:0/1763155831 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9a601a0040 con 0x7f9a600fdc70 2023-07-14T19:17:20.022 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.015+0000 7f9a5cff9700 1 -- 172.21.15.40:0/1763155831 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7f9a54022470 con 0x7f9a600fdc70 2023-07-14T19:17:20.023 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.015+0000 7f9a5cff9700 1 --2- 172.21.15.40:0/1763155831 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f9a480a11e0 0x7f9a480a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:17:20.023 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.015+0000 7f9a5effd700 1 --2- 172.21.15.40:0/1763155831 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f9a480a11e0 0x7f9a480a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:17:20.023 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.015+0000 7f9a5cff9700 1 -- 172.21.15.40:0/1763155831 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 8269+0+0 (secure 0 0 0) 0x7f9a540ce020 con 0x7f9a600fdc70 2023-07-14T19:17:20.023 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.016+0000 7f9a5effd700 1 --2- 172.21.15.40:0/1763155831 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f9a480a11e0 0x7f9a480a3690 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f9a50005fd0 tx=0x7f9a50005dd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:17:20.023 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.016+0000 7f9a71e2cb80 1 Processor -- start 2023-07-14T19:17:20.023 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.016+0000 7f9a71e2cb80 1 -- start start 2023-07-14T19:17:20.023 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.016+0000 7f9a71e2cb80 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5645dc7b5f10 0x5645dc7b6320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:17:20.024 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.017+0000 7f9a71e2cb80 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5645dc7b6860 0x5645dc7b6e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:17:20.024 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.017+0000 7f9a71e2cb80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5645dc7b7460 0x5645dc7b78d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:17:20.024 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.017+0000 7f9a71e2cb80 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x5645dc6bac60 con 0x5645dc7b5f10 2023-07-14T19:17:20.024 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.017+0000 7f9a71e2cb80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5645dc70d9f0 con 0x5645dc7b7460 2023-07-14T19:17:20.024 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.017+0000 7f9a71e2cb80 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x5645dc706e80 con 0x5645dc7b6860 2023-07-14T19:17:20.024 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.017+0000 7f9a5f7fe700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5645dc7b5f10 0x5645dc7b6320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:17:20.024 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.017+0000 7f9a5f7fe700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5645dc7b5f10 0x5645dc7b6320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:34196/0 (socket says 172.21.15.40:34196) 2023-07-14T19:17:20.025 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.017+0000 7f9a5f7fe700 1 -- 172.21.15.40:0/3831746529 learned_addr learned my addr 172.21.15.40:0/3831746529 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-07-14T19:17:20.025 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.017+0000 7f9a5ffff700 1 --2- 172.21.15.40:0/3831746529 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5645dc7b7460 0x5645dc7b78d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:17:20.025 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.017+0000 7f9a5f7fe700 1 -- 172.21.15.40:0/3831746529 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5645dc7b6860 msgr2=0x5645dc7b6e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T19:17:20.025 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.017+0000 7f9a5effd700 1 --2- 172.21.15.40:0/3831746529 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5645dc7b6860 0x5645dc7b6e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:17:20.025 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.017+0000 7f9a5f7fe700 1 --2- 172.21.15.40:0/3831746529 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5645dc7b6860 0x5645dc7b6e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:20.025 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.017+0000 7f9a5f7fe700 1 -- 172.21.15.40:0/3831746529 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5645dc7b7460 msgr2=0x5645dc7b78d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T19:17:20.025 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.017+0000 7f9a5f7fe700 1 --2- 172.21.15.40:0/3831746529 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5645dc7b7460 0x5645dc7b78d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:20.025 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.017+0000 7f9a5f7fe700 1 -- 172.21.15.40:0/3831746529 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5645dc534d90 con 0x5645dc7b5f10 2023-07-14T19:17:20.026 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.017+0000 7f9a5ffff700 1 --2- 172.21.15.40:0/3831746529 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5645dc7b7460 0x5645dc7b78d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T19:17:20.026 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.017+0000 7f9a5f7fe700 1 --2- 172.21.15.40:0/3831746529 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5645dc7b5f10 0x5645dc7b6320 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f9a54097840 tx=0x7f9a54009ca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=940b372f887da5a1 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:17:20.026 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.017+0000 7f9a45ffb700 1 -- 172.21.15.40:0/3831746529 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a5409b340 con 0x5645dc7b5f10 2023-07-14T19:17:20.026 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.017+0000 7f9a45ffb700 1 -- 172.21.15.40:0/3831746529 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f9a5409f370 con 0x5645dc7b5f10 2023-07-14T19:17:20.026 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.017+0000 7f9a45ffb700 1 -- 172.21.15.40:0/3831746529 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a5409b5a0 con 0x5645dc7b5f10 2023-07-14T19:17:20.026 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.018+0000 7f9a71e2cb80 1 -- 172.21.15.40:0/3831746529 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5645dc7b5f10 msgr2=0x5645dc7b6320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:17:20.026 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.018+0000 7f9a71e2cb80 1 --2- 172.21.15.40:0/3831746529 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5645dc7b5f10 0x5645dc7b6320 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f9a54097840 tx=0x7f9a54009ca0 comp rx=0 tx=0).stop 2023-07-14T19:17:20.026 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.018+0000 7f9a71e2cb80 1 -- 172.21.15.40:0/3831746529 shutdown_connections 2023-07-14T19:17:20.027 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.018+0000 7f9a71e2cb80 1 --2- 172.21.15.40:0/3831746529 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5645dc7b7460 0x5645dc7b78d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:20.027 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.018+0000 7f9a71e2cb80 1 --2- 172.21.15.40:0/3831746529 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5645dc7b5f10 0x5645dc7b6320 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:20.027 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.018+0000 7f9a71e2cb80 1 --2- 172.21.15.40:0/3831746529 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5645dc7b6860 0x5645dc7b6e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:20.027 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.018+0000 7f9a71e2cb80 1 -- 172.21.15.40:0/3831746529 >> 172.21.15.40:0/3831746529 conn(0x5645dc7ab630 msgr2=0x5645dc7b0980 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:17:20.027 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.018+0000 7f9a71e2cb80 1 -- 172.21.15.40:0/3831746529 shutdown_connections 2023-07-14T19:17:20.027 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.018+0000 7f9a71e2cb80 1 -- 172.21.15.40:0/3831746529 wait complete. 2023-07-14T19:17:20.028 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.018+0000 7f9a71e2cb80 1 -- start start 2023-07-14T19:17:20.028 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.019+0000 7f9a71e2cb80 1 Processor -- start 2023-07-14T19:17:20.028 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.019+0000 7f9a71e2cb80 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5645dc7b5f10 0x5645dc7bf560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:17:20.028 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.019+0000 7f9a71e2cb80 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5645dc7b6860 0x5645dc7bfaa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:17:20.028 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.019+0000 7f9a71e2cb80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5645dc7b7460 0x5645dc7c6ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:17:20.028 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.019+0000 7f9a71e2cb80 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x5645dc6a6a70 con 0x5645dc7b6860 2023-07-14T19:17:20.029 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.019+0000 7f9a71e2cb80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5645dc6a6770 con 0x5645dc7b7460 2023-07-14T19:17:20.029 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.019+0000 7f9a71e2cb80 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x5645dc58a8d0 con 0x5645dc7b5f10 2023-07-14T19:17:20.029 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.019+0000 7f9a71e2cb80 10 client.0 _ll_register_callbacks cb 0x5645dc6a4710 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-14T19:17:20.029 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.019+0000 7f9a5effd700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5645dc7b6860 0x5645dc7bfaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:17:20.029 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.019+0000 7f9a5effd700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5645dc7b6860 0x5645dc7bfaa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:34204/0 (socket says 172.21.15.40:34204) 2023-07-14T19:17:20.029 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.019+0000 7f9a5effd700 1 -- 172.21.15.40:0/1223856639 learned_addr learned my addr 172.21.15.40:0/1223856639 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-07-14T19:17:20.029 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.019+0000 7f9a5f7fe700 1 --2- 172.21.15.40:0/1223856639 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5645dc7b5f10 0x5645dc7bf560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:17:20.030 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.019+0000 7f9a5effd700 1 -- 172.21.15.40:0/1223856639 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5645dc7b5f10 msgr2=0x5645dc7bf560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T19:17:20.030 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.019+0000 7f9a5effd700 1 --2- 172.21.15.40:0/1223856639 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5645dc7b5f10 0x5645dc7bf560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:20.030 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.019+0000 7f9a5effd700 1 -- 172.21.15.40:0/1223856639 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5645dc7b7460 msgr2=0x5645dc7c6ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T19:17:20.030 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.019+0000 7f9a5effd700 1 --2- 172.21.15.40:0/1223856639 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5645dc7b7460 0x5645dc7c6ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:20.030 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.019+0000 7f9a5effd700 1 -- 172.21.15.40:0/1223856639 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9a601a0040 con 0x5645dc7b6860 2023-07-14T19:17:20.030 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.019+0000 7f9a5effd700 1 --2- 172.21.15.40:0/1223856639 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5645dc7b6860 0x5645dc7bfaa0 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f9a5000a7f0 tx=0x7f9a50004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2e211114414cde65 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:17:20.030 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.019+0000 7f9a3effd700 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2023-07-14T19:17:20.030 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.020+0000 7f9a3effd700 1 -- 172.21.15.40:0/1223856639 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a50004220 con 0x5645dc7b6860 2023-07-14T19:17:20.031 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.020+0000 7f9a3effd700 1 -- 172.21.15.40:0/1223856639 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(83..83 src has 1..83) v4 ==== 8269+0+0 (secure 0 0 0) 0x7f9a50004380 con 0x5645dc7b6860 2023-07-14T19:17:20.031 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.020+0000 7f9a71e2cb80 1 -- 172.21.15.40:0/1223856639 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5645dc6604a0 con 0x5645dc7b6860 2023-07-14T19:17:20.031 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.020+0000 7f9a3effd700 1 -- 172.21.15.40:0/1223856639 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9a50004540 con 0x5645dc7b6860 2023-07-14T19:17:20.031 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.020+0000 7f9a71e2cb80 10 client.15222 fetch_fsmap learned FSMap version 26 2023-07-14T19:17:20.031 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.020+0000 7f9a71e2cb80 1 -- 172.21.15.40:0/1223856639 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) v3 -- 0x5645dc51dc10 con 0x5645dc7b6860 2023-07-14T19:17:20.031 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.020+0000 7f9a3effd700 1 -- 172.21.15.40:0/1223856639 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x5645dc6604a0 con 0x5645dc7b6860 2023-07-14T19:17:20.031 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.020+0000 7f9a3effd700 1 -- 172.21.15.40:0/1223856639 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 26) v1 ==== 7462+0+0 (secure 0 0 0) 0x7f9a50023030 con 0x5645dc7b6860 2023-07-14T19:17:20.031 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.020+0000 7f9a71e2cb80 10 client.15222 fetch_fsmap finished waiting for FSMap version 26 2023-07-14T19:17:20.032 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.020+0000 7f9a71e2cb80 10 client.15222 resolve_mds: resolved name 'c' to daemon mds.c 2023-07-14T19:17:20.032 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.020+0000 7f9a71e2cb80 20 client.15222 populate_metadata read hostname 'smithi040' 2023-07-14T19:17:20.032 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.020+0000 7f9a71e2cb80 1 --2- 172.21.15.40:0/1223856639 >> [v2:172.21.15.191:6836/2406892975,v1:172.21.15.191:6837/2406892975] conn(0x5645dc7ca900 0x5645dc7c9c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:17:20.032 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.020+0000 7f9a71e2cb80 4 client.15222 mds_command: new command op to 24490 tid=0[{"prefix": "get_command_descriptions"}] 2023-07-14T19:17:20.032 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.020+0000 7f9a71e2cb80 1 -- 172.21.15.40:0/1223856639 --> [v2:172.21.15.191:6836/2406892975,v1:172.21.15.191:6837/2406892975] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5645dc784d50 con 0x5645dc7ca900 2023-07-14T19:17:20.032 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.020+0000 7f9a5f7fe700 1 --2- 172.21.15.40:0/1223856639 >> [v2:172.21.15.191:6836/2406892975,v1:172.21.15.191:6837/2406892975] conn(0x5645dc7ca900 0x5645dc7c9c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:17:20.032 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.021+0000 7f9a3effd700 10 client.15222 ms_handle_connect on v2:172.21.15.191:6836/2406892975 2023-07-14T19:17:20.033 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.021+0000 7f9a5f7fe700 1 --2- 172.21.15.40:0/1223856639 >> [v2:172.21.15.191:6836/2406892975,v1:172.21.15.191:6837/2406892975] conn(0x5645dc7ca900 0x5645dc7c9c90 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=b560ef770891e4a3 server_cookie=298b81166562c3a in_seq=0 out_seq=0 2023-07-14T19:17:20.034 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.021+0000 7f9a3effd700 1 -- 172.21.15.40:0/1223856639 <== mds.3 v2:172.21.15.191:6836/2406892975 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18017 (crc 0 0 0) 0x5645dc784d50 con 0x5645dc7ca900 2023-07-14T19:17:20.034 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.021+0000 7f9a3effd700 10 client.15222 handle_command_reply: tid=0 2023-07-14T19:17:20.034 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.022+0000 7f9a71e2cb80 1 client.15222 shutdown 2023-07-14T19:17:20.034 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.022+0000 7f9a71e2cb80 1 -- 172.21.15.40:0/1223856639 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5645dc7b6860 msgr2=0x5645dc7bfaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:17:20.034 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.022+0000 7f9a71e2cb80 1 --2- 172.21.15.40:0/1223856639 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5645dc7b6860 0x5645dc7bfaa0 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f9a5000a7f0 tx=0x7f9a50004750 comp rx=0 tx=0).stop 2023-07-14T19:17:20.034 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.022+0000 7f9a71e2cb80 1 -- 172.21.15.40:0/1223856639 shutdown_connections 2023-07-14T19:17:20.035 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.022+0000 7f9a71e2cb80 1 --2- 172.21.15.40:0/1223856639 >> [v2:172.21.15.191:6836/2406892975,v1:172.21.15.191:6837/2406892975] conn(0x5645dc7ca900 0x5645dc7c9c90 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:20.035 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.022+0000 7f9a71e2cb80 1 --2- 172.21.15.40:0/1223856639 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5645dc7b5f10 0x5645dc7bf560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:20.035 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.022+0000 7f9a3effd700 0 client.15222 ms_handle_reset on v2:172.21.15.191:6836/2406892975 2023-07-14T19:17:20.035 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.022+0000 7f9a71e2cb80 1 --2- 172.21.15.40:0/1223856639 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5645dc7b7460 0x5645dc7c6ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:20.035 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.022+0000 7f9a71e2cb80 1 --2- 172.21.15.40:0/1223856639 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5645dc7b6860 0x5645dc7bfaa0 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:20.035 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.022+0000 7f9a71e2cb80 1 -- 172.21.15.40:0/1223856639 >> 172.21.15.40:0/1223856639 conn(0x5645dc7ab630 msgr2=0x5645dc7af600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:17:20.035 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.022+0000 7f9a71e2cb80 1 -- 172.21.15.40:0/1223856639 shutdown_connections 2023-07-14T19:17:20.036 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.022+0000 7f9a71e2cb80 1 -- 172.21.15.40:0/1223856639 wait complete. 2023-07-14T19:17:20.036 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.022+0000 7f9a71e2cb80 20 client.15222 trim_cache size 0 max 16384 2023-07-14T19:17:20.042 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.040+0000 7f9a71e2cb80 1 Processor -- start 2023-07-14T19:17:20.042 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.040+0000 7f9a71e2cb80 1 -- start start 2023-07-14T19:17:20.042 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.040+0000 7f9a71e2cb80 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5645dc7b5f10 0x5645dc7b00a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:17:20.043 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.040+0000 7f9a71e2cb80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5645dc7b6860 0x5645dc7c6aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:17:20.043 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.040+0000 7f9a71e2cb80 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5645dc7b7460 0x5645dc7bd730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:17:20.043 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.040+0000 7f9a71e2cb80 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x5645dc6604d0 con 0x5645dc7b7460 2023-07-14T19:17:20.043 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.040+0000 7f9a71e2cb80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5645dc7c0780 con 0x5645dc7b6860 2023-07-14T19:17:20.043 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.040+0000 7f9a71e2cb80 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x5645dc596de0 con 0x5645dc7b5f10 2023-07-14T19:17:20.043 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.040+0000 7f9a5ffff700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5645dc7b7460 0x5645dc7bd730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:17:20.043 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.040+0000 7f9a5ffff700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5645dc7b7460 0x5645dc7bd730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:34214/0 (socket says 172.21.15.40:34214) 2023-07-14T19:17:20.044 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.040+0000 7f9a5ffff700 1 -- 172.21.15.40:0/2340394601 learned_addr learned my addr 172.21.15.40:0/2340394601 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-07-14T19:17:20.044 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.040+0000 7f9a5effd700 1 --2- 172.21.15.40:0/2340394601 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5645dc7b6860 0x5645dc7c6aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:17:20.044 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.040+0000 7f9a5f7fe700 1 --2- 172.21.15.40:0/2340394601 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5645dc7b5f10 0x5645dc7b00a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:17:20.044 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.040+0000 7f9a5ffff700 1 -- 172.21.15.40:0/2340394601 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5645dc7b5f10 msgr2=0x5645dc7b00a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T19:17:20.044 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.040+0000 7f9a5ffff700 1 --2- 172.21.15.40:0/2340394601 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5645dc7b5f10 0x5645dc7b00a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:20.044 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.040+0000 7f9a5ffff700 1 -- 172.21.15.40:0/2340394601 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5645dc7b6860 msgr2=0x5645dc7c6aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T19:17:20.044 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.040+0000 7f9a5ffff700 1 --2- 172.21.15.40:0/2340394601 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5645dc7b6860 0x5645dc7c6aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:20.044 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.040+0000 7f9a5ffff700 1 -- 172.21.15.40:0/2340394601 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5645dc55b200 con 0x5645dc7b7460 2023-07-14T19:17:20.045 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.040+0000 7f9a5f7fe700 1 --2- 172.21.15.40:0/2340394601 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5645dc7b5f10 0x5645dc7b00a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-14T19:17:20.045 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.040+0000 7f9a5effd700 1 --2- 172.21.15.40:0/2340394601 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5645dc7b6860 0x5645dc7c6aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T19:17:20.045 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.040+0000 7f9a5ffff700 1 --2- 172.21.15.40:0/2340394601 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5645dc7b7460 0x5645dc7bd730 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f9a580099d0 tx=0x7f9a58009ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=31c21bd13658b3c0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:17:20.045 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.041+0000 7f9a467fc700 1 -- 172.21.15.40:0/2340394601 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a58007c00 con 0x5645dc7b7460 2023-07-14T19:17:20.045 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.041+0000 7f9a467fc700 1 -- 172.21.15.40:0/2340394601 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f9a58009310 con 0x5645dc7b7460 2023-07-14T19:17:20.045 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.041+0000 7f9a467fc700 1 -- 172.21.15.40:0/2340394601 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a580145e0 con 0x5645dc7b7460 2023-07-14T19:17:20.048 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.041+0000 7f9a71e2cb80 1 -- 172.21.15.40:0/2340394601 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5645dc7b7460 msgr2=0x5645dc7bd730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:17:20.049 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.041+0000 7f9a71e2cb80 1 --2- 172.21.15.40:0/2340394601 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5645dc7b7460 0x5645dc7bd730 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f9a580099d0 tx=0x7f9a58009ce0 comp rx=0 tx=0).stop 2023-07-14T19:17:20.049 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.041+0000 7f9a71e2cb80 1 -- 172.21.15.40:0/2340394601 shutdown_connections 2023-07-14T19:17:20.049 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.041+0000 7f9a71e2cb80 1 --2- 172.21.15.40:0/2340394601 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5645dc7b5f10 0x5645dc7b00a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:20.049 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.041+0000 7f9a71e2cb80 1 --2- 172.21.15.40:0/2340394601 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5645dc7b7460 0x5645dc7bd730 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:20.049 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.041+0000 7f9a71e2cb80 1 --2- 172.21.15.40:0/2340394601 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5645dc7b6860 0x5645dc7c6aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:20.049 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.041+0000 7f9a71e2cb80 1 -- 172.21.15.40:0/2340394601 >> 172.21.15.40:0/2340394601 conn(0x5645dc7ab630 msgr2=0x5645dc7acb00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:17:20.049 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.041+0000 7f9a71e2cb80 1 -- 172.21.15.40:0/2340394601 shutdown_connections 2023-07-14T19:17:20.050 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.041+0000 7f9a71e2cb80 1 -- 172.21.15.40:0/2340394601 wait complete. 2023-07-14T19:17:20.050 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.041+0000 7f9a71e2cb80 1 -- start start 2023-07-14T19:17:20.050 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.042+0000 7f9a71e2cb80 1 Processor -- start 2023-07-14T19:17:20.050 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.042+0000 7f9a71e2cb80 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5645dc7b5f10 0x5645dc7c1d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:17:20.050 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.042+0000 7f9a71e2cb80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5645dc7b6860 0x5645dc7c22f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:17:20.050 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.042+0000 7f9a71e2cb80 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5645dc7b7460 0x5645dc7c5040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:17:20.050 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.042+0000 7f9a71e2cb80 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x5645dc7c5580 con 0x5645dc7b7460 2023-07-14T19:17:20.050 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.042+0000 7f9a71e2cb80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5645dc7c56f0 con 0x5645dc7b6860 2023-07-14T19:17:20.051 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.042+0000 7f9a71e2cb80 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x5645dc7c58b0 con 0x5645dc7b5f10 2023-07-14T19:17:20.051 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.042+0000 7f9a71e2cb80 10 client.0 _ll_register_callbacks cb 0x7f9a5000ec30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-14T19:17:20.051 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.042+0000 7f9a5ffff700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5645dc7b7460 0x5645dc7c5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:17:20.051 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.042+0000 7f9a5ffff700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5645dc7b7460 0x5645dc7c5040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:34220/0 (socket says 172.21.15.40:34220) 2023-07-14T19:17:20.051 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.042+0000 7f9a5ffff700 1 -- 172.21.15.40:0/80782968 learned_addr learned my addr 172.21.15.40:0/80782968 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-07-14T19:17:20.051 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.042+0000 7f9a5effd700 1 --2- 172.21.15.40:0/80782968 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5645dc7b6860 0x5645dc7c22f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:17:20.051 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.042+0000 7f9a5f7fe700 1 --2- 172.21.15.40:0/80782968 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5645dc7b5f10 0x5645dc7c1d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:17:20.052 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.042+0000 7f9a5ffff700 1 -- 172.21.15.40:0/80782968 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5645dc7b5f10 msgr2=0x5645dc7c1d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T19:17:20.052 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.042+0000 7f9a5ffff700 1 --2- 172.21.15.40:0/80782968 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5645dc7b5f10 0x5645dc7c1d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:20.052 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.042+0000 7f9a5ffff700 1 -- 172.21.15.40:0/80782968 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5645dc7b6860 msgr2=0x5645dc7c22f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T19:17:20.052 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.042+0000 7f9a5ffff700 1 --2- 172.21.15.40:0/80782968 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5645dc7b6860 0x5645dc7c22f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:20.052 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.042+0000 7f9a5ffff700 1 -- 172.21.15.40:0/80782968 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9a58011040 con 0x5645dc7b7460 2023-07-14T19:17:20.052 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.042+0000 7f9a5effd700 1 --2- 172.21.15.40:0/80782968 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5645dc7b6860 0x5645dc7c22f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T19:17:20.052 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.043+0000 7f9a5ffff700 1 --2- 172.21.15.40:0/80782968 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5645dc7b7460 0x5645dc7c5040 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f9a58000f80 tx=0x7f9a58007f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=823370317549d242 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:17:20.052 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.043+0000 7f9a3effd700 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2023-07-14T19:17:20.053 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.043+0000 7f9a3effd700 1 -- 172.21.15.40:0/80782968 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a58004da0 con 0x5645dc7b7460 2023-07-14T19:17:20.053 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.043+0000 7f9a3effd700 1 -- 172.21.15.40:0/80782968 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(83..83 src has 1..83) v4 ==== 8269+0+0 (secure 0 0 0) 0x7f9a580155a0 con 0x5645dc7b7460 2023-07-14T19:17:20.053 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.043+0000 7f9a71e2cb80 1 -- 172.21.15.40:0/80782968 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f9a5409f370 con 0x5645dc7b7460 2023-07-14T19:17:20.053 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.043+0000 7f9a3effd700 1 -- 172.21.15.40:0/80782968 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9a58015780 con 0x5645dc7b7460 2023-07-14T19:17:20.053 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.043+0000 7f9a71e2cb80 10 client.15228 fetch_fsmap learned FSMap version 26 2023-07-14T19:17:20.053 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.043+0000 7f9a71e2cb80 1 -- 172.21.15.40:0/80782968 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) v3 -- 0x5645dc533260 con 0x5645dc7b7460 2023-07-14T19:17:20.053 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.043+0000 7f9a3effd700 1 -- 172.21.15.40:0/80782968 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f9a5409f370 con 0x5645dc7b7460 2023-07-14T19:17:20.053 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.043+0000 7f9a3effd700 1 -- 172.21.15.40:0/80782968 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 26) v1 ==== 7462+0+0 (secure 0 0 0) 0x7f9a58030050 con 0x5645dc7b7460 2023-07-14T19:17:20.054 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.043+0000 7f9a71e2cb80 10 client.15228 fetch_fsmap finished waiting for FSMap version 26 2023-07-14T19:17:20.054 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.043+0000 7f9a71e2cb80 10 client.15228 resolve_mds: resolved name 'c' to daemon mds.c 2023-07-14T19:17:20.054 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.043+0000 7f9a71e2cb80 20 client.15228 populate_metadata read hostname 'smithi040' 2023-07-14T19:17:20.054 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.043+0000 7f9a71e2cb80 1 --2- 172.21.15.40:0/80782968 >> [v2:172.21.15.191:6836/2406892975,v1:172.21.15.191:6837/2406892975] conn(0x5645dc823ac0 0x5645dc8205b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:17:20.054 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.043+0000 7f9a71e2cb80 4 client.15228 mds_command: new command op to 24490 tid=0[{"prefix": "status"}] 2023-07-14T19:17:20.054 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.043+0000 7f9a71e2cb80 1 -- 172.21.15.40:0/80782968 --> [v2:172.21.15.191:6836/2406892975,v1:172.21.15.191:6837/2406892975] -- command(tid 0: {"prefix": "status"}) v1 -- 0x5645dc6ffe60 con 0x5645dc823ac0 2023-07-14T19:17:20.054 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.044+0000 7f9a5f7fe700 1 --2- 172.21.15.40:0/80782968 >> [v2:172.21.15.191:6836/2406892975,v1:172.21.15.191:6837/2406892975] conn(0x5645dc823ac0 0x5645dc8205b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:17:20.055 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.044+0000 7f9a3effd700 10 client.15228 ms_handle_connect on v2:172.21.15.191:6836/2406892975 2023-07-14T19:17:20.055 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.044+0000 7f9a5f7fe700 1 --2- 172.21.15.40:0/80782968 >> [v2:172.21.15.191:6836/2406892975,v1:172.21.15.191:6837/2406892975] conn(0x5645dc823ac0 0x5645dc8205b0 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=fa0b9c5cb8629c81 server_cookie=35988cba0e1c960a in_seq=0 out_seq=0 2023-07-14T19:17:20.055 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.044+0000 7f9a3effd700 1 -- 172.21.15.40:0/80782968 <== mds.3 v2:172.21.15.191:6836/2406892975 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+332 (crc 0 0 0) 0x5645dc6ffe60 con 0x5645dc823ac0 2023-07-14T19:17:20.055 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.044+0000 7f9a3effd700 10 client.15228 handle_command_reply: tid=0 2023-07-14T19:17:20.055 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.044+0000 7f9a71e2cb80 1 client.15228 shutdown 2023-07-14T19:17:20.055 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.044+0000 7f9a71e2cb80 1 -- 172.21.15.40:0/80782968 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5645dc7b7460 msgr2=0x5645dc7c5040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:17:20.055 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.044+0000 7f9a71e2cb80 1 --2- 172.21.15.40:0/80782968 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5645dc7b7460 0x5645dc7c5040 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f9a58000f80 tx=0x7f9a58007f40 comp rx=0 tx=0).stop 2023-07-14T19:17:20.055 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.044+0000 7f9a71e2cb80 1 -- 172.21.15.40:0/80782968 shutdown_connections 2023-07-14T19:17:20.056 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.044+0000 7f9a71e2cb80 1 --2- 172.21.15.40:0/80782968 >> [v2:172.21.15.191:6836/2406892975,v1:172.21.15.191:6837/2406892975] conn(0x5645dc823ac0 0x5645dc8205b0 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:20.056 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.044+0000 7f9a71e2cb80 1 --2- 172.21.15.40:0/80782968 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5645dc7b5f10 0x5645dc7c1d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:20.056 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.044+0000 7f9a71e2cb80 1 --2- 172.21.15.40:0/80782968 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5645dc7b7460 0x5645dc7c5040 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:20.056 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.044+0000 7f9a71e2cb80 1 --2- 172.21.15.40:0/80782968 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5645dc7b6860 0x5645dc7c22f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:20.056 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.044+0000 7f9a3effd700 0 client.15228 ms_handle_reset on v2:172.21.15.191:6836/2406892975 2023-07-14T19:17:20.056 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.044+0000 7f9a71e2cb80 1 -- 172.21.15.40:0/80782968 >> 172.21.15.40:0/80782968 conn(0x5645dc7ab630 msgr2=0x5645dc7b00a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:17:20.056 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.045+0000 7f9a71e2cb80 1 -- 172.21.15.40:0/80782968 shutdown_connections 2023-07-14T19:17:20.056 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.045+0000 7f9a71e2cb80 1 -- 172.21.15.40:0/80782968 wait complete. 2023-07-14T19:17:20.057 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.045+0000 7f9a71e2cb80 20 client.15228 trim_cache size 0 max 16384 2023-07-14T19:17:20.057 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2023-07-14T19:17:20.057 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_fsid": "adc52f04-2279-11ee-9b33-001a4aab830c", 2023-07-14T19:17:20.057 INFO:teuthology.orchestra.run.smithi040.stdout: "whoami": 3, 2023-07-14T19:17:20.057 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 24490, 2023-07-14T19:17:20.058 INFO:teuthology.orchestra.run.smithi040.stdout: "want_state": "up:active", 2023-07-14T19:17:20.058 INFO:teuthology.orchestra.run.smithi040.stdout: "state": "up:active", 2023-07-14T19:17:20.058 INFO:teuthology.orchestra.run.smithi040.stdout: "fs_name": "cephfs", 2023-07-14T19:17:20.058 INFO:teuthology.orchestra.run.smithi040.stdout: "rank_uptime": 76.442346623999995, 2023-07-14T19:17:20.058 INFO:teuthology.orchestra.run.smithi040.stdout: "mdsmap_epoch": 26, 2023-07-14T19:17:20.058 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap_epoch": 83, 2023-07-14T19:17:20.058 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap_epoch_barrier": 81, 2023-07-14T19:17:20.058 INFO:teuthology.orchestra.run.smithi040.stdout: "uptime": 98.772247773999993 2023-07-14T19:17:20.059 INFO:teuthology.orchestra.run.smithi040.stdout:} 2023-07-14T19:17:20.059 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.045+0000 7f9a65c94700 1 -- 172.21.15.40:0/1763155831 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f9a480a11e0 msgr2=0x7f9a480a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:17:20.059 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.045+0000 7f9a65c94700 1 --2- 172.21.15.40:0/1763155831 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f9a480a11e0 0x7f9a480a3690 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f9a50005fd0 tx=0x7f9a50005dd0 comp rx=0 tx=0).stop 2023-07-14T19:17:20.060 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.045+0000 7f9a65c94700 1 -- 172.21.15.40:0/1763155831 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9a600fdc70 msgr2=0x7f9a60100900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:17:20.060 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.045+0000 7f9a65c94700 1 --2- 172.21.15.40:0/1763155831 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9a600fdc70 0x7f9a60100900 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f9a540095c0 tx=0x7f9a5400f690 comp rx=0 tx=0).stop 2023-07-14T19:17:20.060 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.045+0000 7f9a65c94700 1 -- 172.21.15.40:0/1763155831 shutdown_connections 2023-07-14T19:17:20.060 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.045+0000 7f9a65c94700 1 --2- 172.21.15.40:0/1763155831 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f9a480a11e0 0x7f9a480a3690 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:20.060 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.045+0000 7f9a65c94700 1 --2- 172.21.15.40:0/1763155831 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9a60101200 0x7f9a600ff480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:20.060 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.045+0000 7f9a65c94700 1 --2- 172.21.15.40:0/1763155831 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9a600fdc70 0x7f9a60100900 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:20.060 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.045+0000 7f9a65c94700 1 --2- 172.21.15.40:0/1763155831 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9a600fe620 0x7f9a600fef40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:20.061 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.045+0000 7f9a65c94700 1 -- 172.21.15.40:0/1763155831 >> 172.21.15.40:0/1763155831 conn(0x7f9a600f98a0 msgr2=0x7f9a600fbb70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:17:20.061 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.045+0000 7f9a65c94700 1 -- 172.21.15.40:0/1763155831 shutdown_connections 2023-07-14T19:17:20.061 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.045+0000 7f9a65c94700 1 -- 172.21.15.40:0/1763155831 wait complete. 2023-07-14T19:17:20.062 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.h status 2023-07-14T19:17:20.192 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.188+0000 7ff38a649700 1 -- 172.21.15.40:0/1791087738 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff384103010 msgr2=0x7ff3841053f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:17:20.192 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.188+0000 7ff38a649700 1 --2- 172.21.15.40:0/1791087738 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff384103010 0x7ff3841053f0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7ff378009990 tx=0x7ff378009ca0 comp rx=0 tx=0).stop 2023-07-14T19:17:20.192 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.188+0000 7ff38a649700 1 -- 172.21.15.40:0/1791087738 shutdown_connections 2023-07-14T19:17:20.193 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.188+0000 7ff38a649700 1 --2- 172.21.15.40:0/1791087738 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff384105930 0x7ff384107d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:20.193 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.188+0000 7ff38a649700 1 --2- 172.21.15.40:0/1791087738 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff3841006f0 0x7ff384102ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:20.193 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.188+0000 7ff38a649700 1 --2- 172.21.15.40:0/1791087738 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff384103010 0x7ff3841053f0 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:20.193 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.188+0000 7ff38a649700 1 -- 172.21.15.40:0/1791087738 >> 172.21.15.40:0/1791087738 conn(0x7ff3840fa130 msgr2=0x7ff3840fc560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:17:20.193 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.188+0000 7ff38a649700 1 -- 172.21.15.40:0/1791087738 shutdown_connections 2023-07-14T19:17:20.194 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.188+0000 7ff38a649700 1 -- 172.21.15.40:0/1791087738 wait complete. 2023-07-14T19:17:20.194 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.188+0000 7ff38a649700 1 Processor -- start 2023-07-14T19:17:20.194 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.188+0000 7ff38a649700 1 -- start start 2023-07-14T19:17:20.194 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.189+0000 7ff38a649700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff3841006f0 0x7ff3841958e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:17:20.194 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.189+0000 7ff38a649700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff384103010 0x7ff384195e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:17:20.195 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.189+0000 7ff38a649700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff384105930 0x7ff38419ce90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:17:20.195 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.189+0000 7ff38a649700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff38419d460 con 0x7ff384103010 2023-07-14T19:17:20.195 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.189+0000 7ff38a649700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff38419d5a0 con 0x7ff3841006f0 2023-07-14T19:17:20.195 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.189+0000 7ff38a649700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff38419d730 con 0x7ff384105930 2023-07-14T19:17:20.195 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.189+0000 7ff3837fe700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff384103010 0x7ff384195e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:17:20.196 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.189+0000 7ff3837fe700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff384103010 0x7ff384195e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:34232/0 (socket says 172.21.15.40:34232) 2023-07-14T19:17:20.196 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.189+0000 7ff3837fe700 1 -- 172.21.15.40:0/1738170631 learned_addr learned my addr 172.21.15.40:0/1738170631 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-07-14T19:17:20.196 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.189+0000 7ff383fff700 1 --2- 172.21.15.40:0/1738170631 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff3841006f0 0x7ff3841958e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:17:20.196 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.189+0000 7ff388be6700 1 --2- 172.21.15.40:0/1738170631 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff384105930 0x7ff38419ce90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:17:20.196 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.189+0000 7ff3837fe700 1 -- 172.21.15.40:0/1738170631 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff384105930 msgr2=0x7ff38419ce90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:17:20.196 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.189+0000 7ff3837fe700 1 --2- 172.21.15.40:0/1738170631 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff384105930 0x7ff38419ce90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:20.196 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.189+0000 7ff3837fe700 1 -- 172.21.15.40:0/1738170631 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff3841006f0 msgr2=0x7ff3841958e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:17:20.197 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.189+0000 7ff3837fe700 1 --2- 172.21.15.40:0/1738170631 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff3841006f0 0x7ff3841958e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:20.197 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.189+0000 7ff3837fe700 1 -- 172.21.15.40:0/1738170631 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3780095f0 con 0x7ff384103010 2023-07-14T19:17:20.197 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.189+0000 7ff388be6700 1 --2- 172.21.15.40:0/1738170631 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff384105930 0x7ff38419ce90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-14T19:17:20.197 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.189+0000 7ff3837fe700 1 --2- 172.21.15.40:0/1738170631 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff384103010 0x7ff384195e20 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7ff3780098e0 tx=0x7ff37800f740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:17:20.197 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.189+0000 7ff3817fa700 1 -- 172.21.15.40:0/1738170631 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff37801d070 con 0x7ff384103010 2023-07-14T19:17:20.197 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.189+0000 7ff38a649700 1 -- 172.21.15.40:0/1738170631 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff38419d9b0 con 0x7ff384103010 2023-07-14T19:17:20.197 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.189+0000 7ff3817fa700 1 -- 172.21.15.40:0/1738170631 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff37800fb70 con 0x7ff384103010 2023-07-14T19:17:20.197 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.189+0000 7ff3817fa700 1 -- 172.21.15.40:0/1738170631 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff378017650 con 0x7ff384103010 2023-07-14T19:17:20.198 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.189+0000 7ff38a649700 1 -- 172.21.15.40:0/1738170631 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff38419de30 con 0x7ff384103010 2023-07-14T19:17:20.202 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.192+0000 7ff3817fa700 1 -- 172.21.15.40:0/1738170631 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7ff378022470 con 0x7ff384103010 2023-07-14T19:17:20.202 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.192+0000 7ff3817fa700 1 --2- 172.21.15.40:0/1738170631 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7ff36c0a11e0 0x7ff36c0a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:17:20.203 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.192+0000 7ff3817fa700 1 -- 172.21.15.40:0/1738170631 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 8269+0+0 (secure 0 0 0) 0x7ff3780cdf40 con 0x7ff384103010 2023-07-14T19:17:20.203 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.192+0000 7ff383fff700 1 --2- 172.21.15.40:0/1738170631 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7ff36c0a11e0 0x7ff36c0a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:17:20.203 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.193+0000 7ff383fff700 1 --2- 172.21.15.40:0/1738170631 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7ff36c0a11e0 0x7ff36c0a3690 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7ff374005950 tx=0x7ff3740058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:17:20.203 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.193+0000 7ff3967e1b80 1 Processor -- start 2023-07-14T19:17:20.203 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.193+0000 7ff3967e1b80 1 -- start start 2023-07-14T19:17:20.203 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.193+0000 7ff3967e1b80 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d6a774d320 0x55d6a774d730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:17:20.203 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.193+0000 7ff3967e1b80 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55d6a774dc70 0x55d6a774e290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:17:20.204 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.193+0000 7ff3967e1b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d6a774e870 0x55d6a774ece0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:17:20.204 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.193+0000 7ff3967e1b80 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d6a764dc60 con 0x55d6a774d320 2023-07-14T19:17:20.204 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.193+0000 7ff3967e1b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d6a76a09f0 con 0x55d6a774e870 2023-07-14T19:17:20.204 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.193+0000 7ff3967e1b80 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d6a7699e80 con 0x55d6a774dc70 2023-07-14T19:17:20.204 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.193+0000 7ff383fff700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d6a774d320 0x55d6a774d730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:17:20.204 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.193+0000 7ff383fff700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d6a774d320 0x55d6a774d730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:34238/0 (socket says 172.21.15.40:34238) 2023-07-14T19:17:20.204 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.193+0000 7ff383fff700 1 -- 172.21.15.40:0/1090804015 learned_addr learned my addr 172.21.15.40:0/1090804015 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-07-14T19:17:20.205 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.193+0000 7ff388be6700 1 --2- 172.21.15.40:0/1090804015 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d6a774e870 0x55d6a774ece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:17:20.205 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.193+0000 7ff3837fe700 1 --2- 172.21.15.40:0/1090804015 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55d6a774dc70 0x55d6a774e290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:17:20.205 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.193+0000 7ff383fff700 1 -- 172.21.15.40:0/1090804015 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55d6a774dc70 msgr2=0x55d6a774e290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T19:17:20.205 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.193+0000 7ff383fff700 1 --2- 172.21.15.40:0/1090804015 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55d6a774dc70 0x55d6a774e290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:20.205 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.193+0000 7ff383fff700 1 -- 172.21.15.40:0/1090804015 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d6a774e870 msgr2=0x55d6a774ece0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T19:17:20.205 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.193+0000 7ff383fff700 1 --2- 172.21.15.40:0/1090804015 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d6a774e870 0x55d6a774ece0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:20.205 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.193+0000 7ff383fff700 1 -- 172.21.15.40:0/1090804015 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d6a74c7d90 con 0x55d6a774d320 2023-07-14T19:17:20.205 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.194+0000 7ff383fff700 1 --2- 172.21.15.40:0/1090804015 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d6a774d320 0x55d6a774d730 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7ff37400fd70 tx=0x7ff374012000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bb08d42b1522641e server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:17:20.206 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.194+0000 7ff36a7fc700 1 -- 172.21.15.40:0/1090804015 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff374011c60 con 0x55d6a774d320 2023-07-14T19:17:20.206 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.194+0000 7ff36a7fc700 1 -- 172.21.15.40:0/1090804015 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff374011dc0 con 0x55d6a774d320 2023-07-14T19:17:20.206 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.194+0000 7ff36a7fc700 1 -- 172.21.15.40:0/1090804015 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff37401d7e0 con 0x55d6a774d320 2023-07-14T19:17:20.206 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.194+0000 7ff3967e1b80 1 -- 172.21.15.40:0/1090804015 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d6a774d320 msgr2=0x55d6a774d730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:17:20.206 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.194+0000 7ff3967e1b80 1 --2- 172.21.15.40:0/1090804015 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d6a774d320 0x55d6a774d730 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7ff37400fd70 tx=0x7ff374012000 comp rx=0 tx=0).stop 2023-07-14T19:17:20.206 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.194+0000 7ff3967e1b80 1 -- 172.21.15.40:0/1090804015 shutdown_connections 2023-07-14T19:17:20.206 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.194+0000 7ff3967e1b80 1 --2- 172.21.15.40:0/1090804015 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d6a774e870 0x55d6a774ece0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:20.206 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.194+0000 7ff3967e1b80 1 --2- 172.21.15.40:0/1090804015 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d6a774d320 0x55d6a774d730 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:20.207 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.194+0000 7ff3967e1b80 1 --2- 172.21.15.40:0/1090804015 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55d6a774dc70 0x55d6a774e290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:20.207 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.194+0000 7ff3967e1b80 1 -- 172.21.15.40:0/1090804015 >> 172.21.15.40:0/1090804015 conn(0x55d6a773e630 msgr2=0x55d6a7743980 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:17:20.207 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.195+0000 7ff3967e1b80 1 -- 172.21.15.40:0/1090804015 shutdown_connections 2023-07-14T19:17:20.207 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.195+0000 7ff3967e1b80 1 -- 172.21.15.40:0/1090804015 wait complete. 2023-07-14T19:17:20.207 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.195+0000 7ff3967e1b80 1 -- start start 2023-07-14T19:17:20.207 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.195+0000 7ff3967e1b80 1 Processor -- start 2023-07-14T19:17:20.207 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.195+0000 7ff3967e1b80 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55d6a774d320 0x55d6a7756d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:17:20.208 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.195+0000 7ff3967e1b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d6a774dc70 0x55d6a77572b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:17:20.208 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.195+0000 7ff3967e1b80 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d6a774e870 0x55d6a775e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:17:20.208 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.195+0000 7ff3967e1b80 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d6a7639a70 con 0x55d6a774e870 2023-07-14T19:17:20.208 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.195+0000 7ff3967e1b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d6a7639770 con 0x55d6a774dc70 2023-07-14T19:17:20.208 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.195+0000 7ff3967e1b80 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d6a751d8d0 con 0x55d6a774d320 2023-07-14T19:17:20.208 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.195+0000 7ff3967e1b80 10 client.0 _ll_register_callbacks cb 0x55d6a7637710 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-14T19:17:20.208 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.195+0000 7ff388be6700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d6a774e870 0x55d6a775e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:17:20.209 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.195+0000 7ff388be6700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d6a774e870 0x55d6a775e2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:34248/0 (socket says 172.21.15.40:34248) 2023-07-14T19:17:20.209 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.196+0000 7ff388be6700 1 -- 172.21.15.40:0/1871813374 learned_addr learned my addr 172.21.15.40:0/1871813374 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-07-14T19:17:20.209 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.196+0000 7ff3837fe700 1 --2- 172.21.15.40:0/1871813374 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d6a774dc70 0x55d6a77572b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:17:20.209 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.196+0000 7ff388be6700 1 -- 172.21.15.40:0/1871813374 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55d6a774d320 msgr2=0x55d6a7756d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T19:17:20.209 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.196+0000 7ff388be6700 1 --2- 172.21.15.40:0/1871813374 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55d6a774d320 0x55d6a7756d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:20.209 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.196+0000 7ff388be6700 1 -- 172.21.15.40:0/1871813374 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d6a774dc70 msgr2=0x55d6a77572b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T19:17:20.209 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.196+0000 7ff388be6700 1 --2- 172.21.15.40:0/1871813374 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d6a774dc70 0x55d6a77572b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:20.210 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.196+0000 7ff388be6700 1 -- 172.21.15.40:0/1871813374 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff37400fa20 con 0x55d6a774e870 2023-07-14T19:17:20.210 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.196+0000 7ff388be6700 1 --2- 172.21.15.40:0/1871813374 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d6a774e870 0x55d6a775e2f0 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7ff37c00fd30 tx=0x7ff37c00d430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cc460d976537d651 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:17:20.210 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.196+0000 7ff3637fe700 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2023-07-14T19:17:20.210 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.196+0000 7ff3637fe700 1 -- 172.21.15.40:0/1871813374 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff37c00da40 con 0x55d6a774e870 2023-07-14T19:17:20.210 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.196+0000 7ff3637fe700 1 -- 172.21.15.40:0/1871813374 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(83..83 src has 1..83) v4 ==== 8269+0+0 (secure 0 0 0) 0x7ff37c00dba0 con 0x55d6a774e870 2023-07-14T19:17:20.210 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.196+0000 7ff3967e1b80 1 -- 172.21.15.40:0/1871813374 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55d6a75f34a0 con 0x55d6a774e870 2023-07-14T19:17:20.210 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.196+0000 7ff3637fe700 1 -- 172.21.15.40:0/1871813374 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff37c00dd60 con 0x55d6a774e870 2023-07-14T19:17:20.210 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.196+0000 7ff3967e1b80 10 client.15240 fetch_fsmap learned FSMap version 26 2023-07-14T19:17:20.211 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.196+0000 7ff3967e1b80 1 -- 172.21.15.40:0/1871813374 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) v3 -- 0x55d6a74b0c10 con 0x55d6a774e870 2023-07-14T19:17:20.211 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.196+0000 7ff3637fe700 1 -- 172.21.15.40:0/1871813374 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55d6a75f34a0 con 0x55d6a774e870 2023-07-14T19:17:20.211 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.197+0000 7ff3637fe700 1 -- 172.21.15.40:0/1871813374 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 26) v1 ==== 7462+0+0 (secure 0 0 0) 0x7ff37c012070 con 0x55d6a774e870 2023-07-14T19:17:20.212 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.197+0000 7ff3967e1b80 10 client.15240 fetch_fsmap finished waiting for FSMap version 26 2023-07-14T19:17:20.212 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.197+0000 7ff3967e1b80 10 client.15240 resolve_mds: resolved name 'h' to daemon mds.h 2023-07-14T19:17:20.212 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.197+0000 7ff3967e1b80 20 client.15240 populate_metadata read hostname 'smithi040' 2023-07-14T19:17:20.212 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.197+0000 7ff3967e1b80 1 --2- 172.21.15.40:0/1871813374 >> [v2:172.21.15.130:6836/2604019369,v1:172.21.15.130:6837/2604019369] conn(0x55d6a77620a0 0x55d6a7761450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:17:20.213 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.197+0000 7ff3967e1b80 4 client.15240 mds_command: new command op to 24506 tid=0[{"prefix": "get_command_descriptions"}] 2023-07-14T19:17:20.213 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.197+0000 7ff3967e1b80 1 -- 172.21.15.40:0/1871813374 --> [v2:172.21.15.130:6836/2604019369,v1:172.21.15.130:6837/2604019369] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55d6a7717d50 con 0x55d6a77620a0 2023-07-14T19:17:20.213 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.197+0000 7ff383fff700 1 --2- 172.21.15.40:0/1871813374 >> [v2:172.21.15.130:6836/2604019369,v1:172.21.15.130:6837/2604019369] conn(0x55d6a77620a0 0x55d6a7761450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:17:20.214 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.197+0000 7ff3637fe700 10 client.15240 ms_handle_connect on v2:172.21.15.130:6836/2604019369 2023-07-14T19:17:20.214 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.197+0000 7ff383fff700 1 --2- 172.21.15.40:0/1871813374 >> [v2:172.21.15.130:6836/2604019369,v1:172.21.15.130:6837/2604019369] conn(0x55d6a77620a0 0x55d6a7761450 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=45348985e390a06 server_cookie=9c655bae9339798e in_seq=0 out_seq=0 2023-07-14T19:17:20.214 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.198+0000 7ff3637fe700 1 -- 172.21.15.40:0/1871813374 <== mds.2 v2:172.21.15.130:6836/2604019369 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18017 (crc 0 0 0) 0x55d6a7717d50 con 0x55d6a77620a0 2023-07-14T19:17:20.214 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.198+0000 7ff3637fe700 10 client.15240 handle_command_reply: tid=0 2023-07-14T19:17:20.215 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.198+0000 7ff3967e1b80 1 client.15240 shutdown 2023-07-14T19:17:20.215 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.198+0000 7ff3967e1b80 1 -- 172.21.15.40:0/1871813374 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d6a774e870 msgr2=0x55d6a775e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:17:20.215 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.198+0000 7ff3967e1b80 1 --2- 172.21.15.40:0/1871813374 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d6a774e870 0x55d6a775e2f0 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7ff37c00fd30 tx=0x7ff37c00d430 comp rx=0 tx=0).stop 2023-07-14T19:17:20.216 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.198+0000 7ff3967e1b80 1 -- 172.21.15.40:0/1871813374 shutdown_connections 2023-07-14T19:17:20.216 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.198+0000 7ff3967e1b80 1 --2- 172.21.15.40:0/1871813374 >> [v2:172.21.15.130:6836/2604019369,v1:172.21.15.130:6837/2604019369] conn(0x55d6a77620a0 0x55d6a7761450 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:20.216 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.198+0000 7ff3967e1b80 1 --2- 172.21.15.40:0/1871813374 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55d6a774d320 0x55d6a7756d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:20.217 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.198+0000 7ff3967e1b80 1 --2- 172.21.15.40:0/1871813374 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d6a774e870 0x55d6a775e2f0 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:20.217 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.198+0000 7ff3637fe700 0 client.15240 ms_handle_reset on v2:172.21.15.130:6836/2604019369 2023-07-14T19:17:20.217 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.198+0000 7ff3967e1b80 1 --2- 172.21.15.40:0/1871813374 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d6a774dc70 0x55d6a77572b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:20.218 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.198+0000 7ff3967e1b80 1 -- 172.21.15.40:0/1871813374 >> 172.21.15.40:0/1871813374 conn(0x55d6a773e630 msgr2=0x55d6a7742600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:17:20.218 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.198+0000 7ff3967e1b80 1 -- 172.21.15.40:0/1871813374 shutdown_connections 2023-07-14T19:17:20.218 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.198+0000 7ff3967e1b80 1 -- 172.21.15.40:0/1871813374 wait complete. 2023-07-14T19:17:20.218 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.198+0000 7ff3967e1b80 20 client.15240 trim_cache size 0 max 16384 2023-07-14T19:17:20.224 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.216+0000 7ff3967e1b80 1 Processor -- start 2023-07-14T19:17:20.224 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.216+0000 7ff3967e1b80 1 -- start start 2023-07-14T19:17:20.224 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.216+0000 7ff3967e1b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d6a774d320 0x55d6a77430a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:17:20.225 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.216+0000 7ff3967e1b80 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d6a774dc70 0x55d6a776ae30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:17:20.225 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.216+0000 7ff3967e1b80 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55d6a774e870 0x55d6a7755a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:17:20.225 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.216+0000 7ff3967e1b80 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d6a7756940 con 0x55d6a774dc70 2023-07-14T19:17:20.225 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.216+0000 7ff3967e1b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d6a7643470 con 0x55d6a774d320 2023-07-14T19:17:20.225 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.216+0000 7ff3967e1b80 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d6a7668790 con 0x55d6a774e870 2023-07-14T19:17:20.225 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.216+0000 7ff3837fe700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d6a774dc70 0x55d6a776ae30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:17:20.225 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.216+0000 7ff3837fe700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d6a774dc70 0x55d6a776ae30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:34254/0 (socket says 172.21.15.40:34254) 2023-07-14T19:17:20.226 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.216+0000 7ff3837fe700 1 -- 172.21.15.40:0/608653281 learned_addr learned my addr 172.21.15.40:0/608653281 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-07-14T19:17:20.226 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.216+0000 7ff383fff700 1 --2- 172.21.15.40:0/608653281 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d6a774d320 0x55d6a77430a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:17:20.226 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.216+0000 7ff3837fe700 1 -- 172.21.15.40:0/608653281 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55d6a774e870 msgr2=0x55d6a7755a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T19:17:20.226 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.216+0000 7ff388be6700 1 --2- 172.21.15.40:0/608653281 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55d6a774e870 0x55d6a7755a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:17:20.226 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.216+0000 7ff3837fe700 1 --2- 172.21.15.40:0/608653281 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55d6a774e870 0x55d6a7755a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:20.226 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.216+0000 7ff3837fe700 1 -- 172.21.15.40:0/608653281 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d6a774d320 msgr2=0x55d6a77430a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T19:17:20.226 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.216+0000 7ff3837fe700 1 --2- 172.21.15.40:0/608653281 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d6a774d320 0x55d6a77430a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:20.226 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.216+0000 7ff3837fe700 1 -- 172.21.15.40:0/608653281 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d6a74ee200 con 0x55d6a774dc70 2023-07-14T19:17:20.227 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.216+0000 7ff383fff700 1 --2- 172.21.15.40:0/608653281 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d6a774d320 0x55d6a77430a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-14T19:17:20.227 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.216+0000 7ff388be6700 1 --2- 172.21.15.40:0/608653281 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55d6a774e870 0x55d6a7755a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-14T19:17:20.227 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.217+0000 7ff3837fe700 1 --2- 172.21.15.40:0/608653281 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d6a774dc70 0x55d6a776ae30 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7ff378005cb0 tx=0x7ff378005040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e785acdbd41c0991 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:17:20.227 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.217+0000 7ff36affd700 1 -- 172.21.15.40:0/608653281 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff37809b260 con 0x55d6a774dc70 2023-07-14T19:17:20.227 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.217+0000 7ff36affd700 1 -- 172.21.15.40:0/608653281 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff37809f290 con 0x55d6a774dc70 2023-07-14T19:17:20.227 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.217+0000 7ff36affd700 1 -- 172.21.15.40:0/608653281 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff37809a9b0 con 0x55d6a774dc70 2023-07-14T19:17:20.227 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.217+0000 7ff3967e1b80 1 -- 172.21.15.40:0/608653281 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d6a774dc70 msgr2=0x55d6a776ae30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:17:20.227 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.217+0000 7ff3967e1b80 1 --2- 172.21.15.40:0/608653281 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d6a774dc70 0x55d6a776ae30 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7ff378005cb0 tx=0x7ff378005040 comp rx=0 tx=0).stop 2023-07-14T19:17:20.228 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.217+0000 7ff3967e1b80 1 -- 172.21.15.40:0/608653281 shutdown_connections 2023-07-14T19:17:20.228 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.217+0000 7ff3967e1b80 1 --2- 172.21.15.40:0/608653281 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55d6a774e870 0x55d6a7755a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:20.228 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.217+0000 7ff3967e1b80 1 --2- 172.21.15.40:0/608653281 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d6a774d320 0x55d6a77430a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:20.228 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.217+0000 7ff3967e1b80 1 --2- 172.21.15.40:0/608653281 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d6a774dc70 0x55d6a776ae30 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:20.228 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.217+0000 7ff3967e1b80 1 -- 172.21.15.40:0/608653281 >> 172.21.15.40:0/608653281 conn(0x55d6a773e630 msgr2=0x55d6a773fb00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:17:20.228 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.217+0000 7ff3967e1b80 1 -- 172.21.15.40:0/608653281 shutdown_connections 2023-07-14T19:17:20.228 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.217+0000 7ff3967e1b80 1 -- 172.21.15.40:0/608653281 wait complete. 2023-07-14T19:17:20.229 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.218+0000 7ff3967e1b80 1 -- start start 2023-07-14T19:17:20.229 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.218+0000 7ff3967e1b80 1 Processor -- start 2023-07-14T19:17:20.229 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.218+0000 7ff3967e1b80 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d6a774d320 0x55d6a77660a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:17:20.229 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.218+0000 7ff3967e1b80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d6a774dc70 0x55d6a77665e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:17:20.229 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.218+0000 7ff3967e1b80 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55d6a774e870 0x55d6a77b5d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:17:20.229 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.218+0000 7ff3967e1b80 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d6a776afb0 con 0x55d6a774d320 2023-07-14T19:17:20.229 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.218+0000 7ff3967e1b80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d6a776b120 con 0x55d6a774dc70 2023-07-14T19:17:20.230 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.218+0000 7ff3967e1b80 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d6a77b6280 con 0x55d6a774e870 2023-07-14T19:17:20.230 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.218+0000 7ff3967e1b80 10 client.0 _ll_register_callbacks cb 0x7ff37c00f570 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-14T19:17:20.230 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.218+0000 7ff383fff700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d6a774d320 0x55d6a77660a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:17:20.230 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.218+0000 7ff383fff700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d6a774d320 0x55d6a77660a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:34258/0 (socket says 172.21.15.40:34258) 2023-07-14T19:17:20.230 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.218+0000 7ff383fff700 1 -- 172.21.15.40:0/1443538319 learned_addr learned my addr 172.21.15.40:0/1443538319 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-07-14T19:17:20.230 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.218+0000 7ff3837fe700 1 --2- 172.21.15.40:0/1443538319 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d6a774dc70 0x55d6a77665e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:17:20.230 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.218+0000 7ff388be6700 1 --2- 172.21.15.40:0/1443538319 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55d6a774e870 0x55d6a77b5d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:17:20.230 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.218+0000 7ff383fff700 1 -- 172.21.15.40:0/1443538319 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55d6a774e870 msgr2=0x55d6a77b5d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T19:17:20.231 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.218+0000 7ff383fff700 1 --2- 172.21.15.40:0/1443538319 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55d6a774e870 0x55d6a77b5d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:20.231 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.219+0000 7ff383fff700 1 -- 172.21.15.40:0/1443538319 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d6a774dc70 msgr2=0x55d6a77665e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T19:17:20.231 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.219+0000 7ff383fff700 1 --2- 172.21.15.40:0/1443538319 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d6a774dc70 0x55d6a77665e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:20.231 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.219+0000 7ff383fff700 1 -- 172.21.15.40:0/1443538319 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff378095e70 con 0x55d6a774d320 2023-07-14T19:17:20.231 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.219+0000 7ff388be6700 1 --2- 172.21.15.40:0/1443538319 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55d6a774e870 0x55d6a77b5d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-14T19:17:20.231 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.219+0000 7ff383fff700 1 --2- 172.21.15.40:0/1443538319 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d6a774d320 0x55d6a77660a0 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7ff37400b4d0 tx=0x7ff37400fee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=af68074befd0c115 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:17:20.231 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.219+0000 7ff3637fe700 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2023-07-14T19:17:20.231 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.219+0000 7ff3637fe700 1 -- 172.21.15.40:0/1443538319 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff37402e040 con 0x55d6a774d320 2023-07-14T19:17:20.232 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.219+0000 7ff3637fe700 1 -- 172.21.15.40:0/1443538319 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(83..83 src has 1..83) v4 ==== 8269+0+0 (secure 0 0 0) 0x7ff374005340 con 0x55d6a774d320 2023-07-14T19:17:20.232 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.219+0000 7ff3967e1b80 1 -- 172.21.15.40:0/1443538319 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7ff374011dc0 con 0x55d6a774d320 2023-07-14T19:17:20.232 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.219+0000 7ff3637fe700 1 -- 172.21.15.40:0/1443538319 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff374032670 con 0x55d6a774d320 2023-07-14T19:17:20.232 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.219+0000 7ff3967e1b80 10 client.15246 fetch_fsmap learned FSMap version 26 2023-07-14T19:17:20.232 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.219+0000 7ff3967e1b80 1 -- 172.21.15.40:0/1443538319 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) v3 -- 0x55d6a74c6260 con 0x55d6a774d320 2023-07-14T19:17:20.232 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.219+0000 7ff3637fe700 1 -- 172.21.15.40:0/1443538319 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff374043080 con 0x55d6a774d320 2023-07-14T19:17:20.232 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.219+0000 7ff3637fe700 1 -- 172.21.15.40:0/1443538319 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 26) v1 ==== 7462+0+0 (secure 0 0 0) 0x7ff37403d030 con 0x55d6a774d320 2023-07-14T19:17:20.233 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.219+0000 7ff3967e1b80 10 client.15246 fetch_fsmap finished waiting for FSMap version 26 2023-07-14T19:17:20.233 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.219+0000 7ff3967e1b80 10 client.15246 resolve_mds: resolved name 'h' to daemon mds.h 2023-07-14T19:17:20.233 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.219+0000 7ff3967e1b80 20 client.15246 populate_metadata read hostname 'smithi040' 2023-07-14T19:17:20.233 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.219+0000 7ff3967e1b80 1 --2- 172.21.15.40:0/1443538319 >> [v2:172.21.15.130:6836/2604019369,v1:172.21.15.130:6837/2604019369] conn(0x55d6a77baeb0 0x55d6a77b77e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:17:20.233 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.219+0000 7ff3967e1b80 4 client.15246 mds_command: new command op to 24506 tid=0[{"prefix": "status"}] 2023-07-14T19:17:20.233 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.219+0000 7ff3967e1b80 1 -- 172.21.15.40:0/1443538319 --> [v2:172.21.15.130:6836/2604019369,v1:172.21.15.130:6837/2604019369] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55d6a7692e60 con 0x55d6a77baeb0 2023-07-14T19:17:20.234 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.220+0000 7ff3837fe700 1 --2- 172.21.15.40:0/1443538319 >> [v2:172.21.15.130:6836/2604019369,v1:172.21.15.130:6837/2604019369] conn(0x55d6a77baeb0 0x55d6a77b77e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:17:20.234 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.220+0000 7ff3837fe700 1 --2- 172.21.15.40:0/1443538319 >> [v2:172.21.15.130:6836/2604019369,v1:172.21.15.130:6837/2604019369] conn(0x55d6a77baeb0 0x55d6a77b77e0 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=f43e321ceda3200d server_cookie=5e4c992cf487b555 in_seq=0 out_seq=0 2023-07-14T19:17:20.234 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.220+0000 7ff3637fe700 10 client.15246 ms_handle_connect on v2:172.21.15.130:6836/2604019369 2023-07-14T19:17:20.234 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.220+0000 7ff3637fe700 1 -- 172.21.15.40:0/1443538319 <== mds.2 v2:172.21.15.130:6836/2604019369 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+332 (crc 0 0 0) 0x55d6a7692e60 con 0x55d6a77baeb0 2023-07-14T19:17:20.234 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.220+0000 7ff3637fe700 10 client.15246 handle_command_reply: tid=0 2023-07-14T19:17:20.234 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.220+0000 7ff3967e1b80 1 client.15246 shutdown 2023-07-14T19:17:20.235 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.220+0000 7ff3967e1b80 1 -- 172.21.15.40:0/1443538319 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d6a774d320 msgr2=0x55d6a77660a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:17:20.235 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.220+0000 7ff3967e1b80 1 --2- 172.21.15.40:0/1443538319 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d6a774d320 0x55d6a77660a0 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7ff37400b4d0 tx=0x7ff37400fee0 comp rx=0 tx=0).stop 2023-07-14T19:17:20.235 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.220+0000 7ff3967e1b80 1 -- 172.21.15.40:0/1443538319 shutdown_connections 2023-07-14T19:17:20.235 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.220+0000 7ff3967e1b80 1 --2- 172.21.15.40:0/1443538319 >> [v2:172.21.15.130:6836/2604019369,v1:172.21.15.130:6837/2604019369] conn(0x55d6a77baeb0 0x55d6a77b77e0 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:20.236 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.220+0000 7ff3967e1b80 1 --2- 172.21.15.40:0/1443538319 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x55d6a774e870 0x55d6a77b5d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:20.236 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.220+0000 7ff3967e1b80 1 --2- 172.21.15.40:0/1443538319 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d6a774d320 0x55d6a77660a0 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:20.236 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.220+0000 7ff3967e1b80 1 --2- 172.21.15.40:0/1443538319 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d6a774dc70 0x55d6a77665e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:20.236 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.220+0000 7ff3637fe700 0 client.15246 ms_handle_reset on v2:172.21.15.130:6836/2604019369 2023-07-14T19:17:20.237 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.220+0000 7ff3967e1b80 1 -- 172.21.15.40:0/1443538319 >> 172.21.15.40:0/1443538319 conn(0x55d6a773e630 msgr2=0x55d6a77430a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:17:20.237 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.221+0000 7ff3967e1b80 1 -- 172.21.15.40:0/1443538319 shutdown_connections 2023-07-14T19:17:20.237 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.221+0000 7ff3967e1b80 1 -- 172.21.15.40:0/1443538319 wait complete. 2023-07-14T19:17:20.237 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.221+0000 7ff3967e1b80 20 client.15246 trim_cache size 0 max 16384 2023-07-14T19:17:20.238 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2023-07-14T19:17:20.238 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_fsid": "adc52f04-2279-11ee-9b33-001a4aab830c", 2023-07-14T19:17:20.238 INFO:teuthology.orchestra.run.smithi040.stdout: "whoami": 2, 2023-07-14T19:17:20.238 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 24506, 2023-07-14T19:17:20.239 INFO:teuthology.orchestra.run.smithi040.stdout: "want_state": "up:active", 2023-07-14T19:17:20.239 INFO:teuthology.orchestra.run.smithi040.stdout: "state": "up:active", 2023-07-14T19:17:20.239 INFO:teuthology.orchestra.run.smithi040.stdout: "fs_name": "cephfs", 2023-07-14T19:17:20.239 INFO:teuthology.orchestra.run.smithi040.stdout: "rank_uptime": 77.638225438999996, 2023-07-14T19:17:20.239 INFO:teuthology.orchestra.run.smithi040.stdout: "mdsmap_epoch": 26, 2023-07-14T19:17:20.240 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap_epoch": 83, 2023-07-14T19:17:20.240 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap_epoch_barrier": 81, 2023-07-14T19:17:20.240 INFO:teuthology.orchestra.run.smithi040.stdout: "uptime": 96.698818403000004 2023-07-14T19:17:20.240 INFO:teuthology.orchestra.run.smithi040.stdout:} 2023-07-14T19:17:20.242 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.221+0000 7ff38a649700 1 -- 172.21.15.40:0/1738170631 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7ff36c0a11e0 msgr2=0x7ff36c0a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:17:20.242 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.221+0000 7ff38a649700 1 --2- 172.21.15.40:0/1738170631 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7ff36c0a11e0 0x7ff36c0a3690 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7ff374005950 tx=0x7ff3740058e0 comp rx=0 tx=0).stop 2023-07-14T19:17:20.242 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.221+0000 7ff38a649700 1 -- 172.21.15.40:0/1738170631 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff384103010 msgr2=0x7ff384195e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:17:20.243 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.221+0000 7ff38a649700 1 --2- 172.21.15.40:0/1738170631 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff384103010 0x7ff384195e20 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7ff3780098e0 tx=0x7ff37800f740 comp rx=0 tx=0).stop 2023-07-14T19:17:20.243 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.221+0000 7ff38a649700 1 -- 172.21.15.40:0/1738170631 shutdown_connections 2023-07-14T19:17:20.243 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.221+0000 7ff38a649700 1 --2- 172.21.15.40:0/1738170631 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7ff36c0a11e0 0x7ff36c0a3690 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:20.243 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.221+0000 7ff38a649700 1 --2- 172.21.15.40:0/1738170631 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff384105930 0x7ff38419ce90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:20.244 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.221+0000 7ff38a649700 1 --2- 172.21.15.40:0/1738170631 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff3841006f0 0x7ff3841958e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:20.244 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.221+0000 7ff38a649700 1 --2- 172.21.15.40:0/1738170631 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff384103010 0x7ff384195e20 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:20.244 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.221+0000 7ff38a649700 1 -- 172.21.15.40:0/1738170631 >> 172.21.15.40:0/1738170631 conn(0x7ff3840fa130 msgr2=0x7ff384101350 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:17:20.244 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.221+0000 7ff38a649700 1 -- 172.21.15.40:0/1738170631 shutdown_connections 2023-07-14T19:17:20.244 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.221+0000 7ff38a649700 1 -- 172.21.15.40:0/1738170631 wait complete. 2023-07-14T19:17:20.245 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.i status 2023-07-14T19:17:20.246 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:17:19 smithi130 ceph-mon[97832]: pgmap v297: 97 pgs: 97 active+clean; 638 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2023-07-14T19:17:20.246 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:17:19 smithi130 ceph-mon[97832]: from='client.? 172.21.15.40:0/3423596551' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-07-14T19:17:20.247 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:17:19 smithi130 ceph-mon[97832]: from='client.? 172.21.15.40:0/2362745860' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T19:17:20.247 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:17:19 smithi130 ceph-mon[97832]: from='client.? 172.21.15.40:0/420793535' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T19:17:20.247 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:17:19 smithi191 ceph-mon[98311]: pgmap v297: 97 pgs: 97 active+clean; 638 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2023-07-14T19:17:20.247 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:17:19 smithi191 ceph-mon[98311]: from='client.? 172.21.15.40:0/3423596551' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-07-14T19:17:20.248 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:17:19 smithi191 ceph-mon[98311]: from='client.? 172.21.15.40:0/2362745860' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T19:17:20.248 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:17:19 smithi191 ceph-mon[98311]: from='client.? 172.21.15.40:0/420793535' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T19:17:20.374 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.372+0000 7f609a622700 1 -- 172.21.15.40:0/3729898871 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f60941006f0 msgr2=0x7f6094102ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:17:20.375 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.372+0000 7f609a622700 1 --2- 172.21.15.40:0/3729898871 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f60941006f0 0x7f6094102ad0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f6084009990 tx=0x7f6084009ca0 comp rx=0 tx=0).stop 2023-07-14T19:17:20.375 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.372+0000 7f609a622700 1 -- 172.21.15.40:0/3729898871 shutdown_connections 2023-07-14T19:17:20.375 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.372+0000 7f609a622700 1 --2- 172.21.15.40:0/3729898871 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6094105930 0x7f6094107d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:20.375 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.372+0000 7f609a622700 1 --2- 172.21.15.40:0/3729898871 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f60941006f0 0x7f6094102ad0 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:20.375 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.372+0000 7f609a622700 1 --2- 172.21.15.40:0/3729898871 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6094103010 0x7f60941053f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:20.375 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.372+0000 7f609a622700 1 -- 172.21.15.40:0/3729898871 >> 172.21.15.40:0/3729898871 conn(0x7f60940fa130 msgr2=0x7f60940fc560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:17:20.375 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.372+0000 7f609a622700 1 -- 172.21.15.40:0/3729898871 shutdown_connections 2023-07-14T19:17:20.376 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.372+0000 7f609a622700 1 -- 172.21.15.40:0/3729898871 wait complete. 2023-07-14T19:17:20.376 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.373+0000 7f609a622700 1 Processor -- start 2023-07-14T19:17:20.376 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.373+0000 7f609a622700 1 -- start start 2023-07-14T19:17:20.378 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.373+0000 7f609a622700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f60941006f0 0x7f6094195920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:17:20.378 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.373+0000 7f609a622700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6094103010 0x7f6094195e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:17:20.378 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.373+0000 7f609a622700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6094105930 0x7f609419ced0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:17:20.378 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.373+0000 7f609a622700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f609419d4a0 con 0x7f60941006f0 2023-07-14T19:17:20.378 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.373+0000 7f609a622700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f609419d5e0 con 0x7f6094105930 2023-07-14T19:17:20.379 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.373+0000 7f609a622700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f609419d770 con 0x7f6094103010 2023-07-14T19:17:20.379 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.373+0000 7f6093fff700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f60941006f0 0x7f6094195920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:17:20.379 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.373+0000 7f6093fff700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f60941006f0 0x7f6094195920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:34272/0 (socket says 172.21.15.40:34272) 2023-07-14T19:17:20.380 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.373+0000 7f6093fff700 1 -- 172.21.15.40:0/187175802 learned_addr learned my addr 172.21.15.40:0/187175802 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-07-14T19:17:20.380 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.374+0000 7f6098bbf700 1 --2- 172.21.15.40:0/187175802 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6094105930 0x7f609419ced0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:17:20.380 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.374+0000 7f60937fe700 1 --2- 172.21.15.40:0/187175802 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6094103010 0x7f6094195e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:17:20.380 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.374+0000 7f6093fff700 1 -- 172.21.15.40:0/187175802 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6094103010 msgr2=0x7f6094195e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:17:20.381 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.374+0000 7f6093fff700 1 --2- 172.21.15.40:0/187175802 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6094103010 0x7f6094195e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:20.381 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.374+0000 7f6093fff700 1 -- 172.21.15.40:0/187175802 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6094105930 msgr2=0x7f609419ced0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:17:20.381 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.374+0000 7f6093fff700 1 --2- 172.21.15.40:0/187175802 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6094105930 0x7f609419ced0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:20.381 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.374+0000 7f6093fff700 1 -- 172.21.15.40:0/187175802 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60840095f0 con 0x7f60941006f0 2023-07-14T19:17:20.382 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.374+0000 7f6093fff700 1 --2- 172.21.15.40:0/187175802 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f60941006f0 0x7f6094195920 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f60840059d0 tx=0x7f60840047a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:17:20.382 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.374+0000 7f60917fa700 1 -- 172.21.15.40:0/187175802 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f608401d070 con 0x7f60941006f0 2023-07-14T19:17:20.382 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.374+0000 7f60917fa700 1 -- 172.21.15.40:0/187175802 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f608400fc40 con 0x7f60941006f0 2023-07-14T19:17:20.383 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.374+0000 7f609a622700 1 -- 172.21.15.40:0/187175802 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f609419d9f0 con 0x7f60941006f0 2023-07-14T19:17:20.383 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.374+0000 7f60917fa700 1 -- 172.21.15.40:0/187175802 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6084017650 con 0x7f60941006f0 2023-07-14T19:17:20.383 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.374+0000 7f609a622700 1 -- 172.21.15.40:0/187175802 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f609419de10 con 0x7f60941006f0 2023-07-14T19:17:20.383 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.376+0000 7f60917fa700 1 -- 172.21.15.40:0/187175802 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7f6084022470 con 0x7f60941006f0 2023-07-14T19:17:20.393 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.377+0000 7f60917fa700 1 --2- 172.21.15.40:0/187175802 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f607c0a11e0 0x7f607c0a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:17:20.393 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.377+0000 7f60937fe700 1 --2- 172.21.15.40:0/187175802 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f607c0a11e0 0x7f607c0a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:17:20.393 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.377+0000 7f60917fa700 1 -- 172.21.15.40:0/187175802 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 8269+0+0 (secure 0 0 0) 0x7f60840ce0f0 con 0x7f60941006f0 2023-07-14T19:17:20.393 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.378+0000 7f60937fe700 1 --2- 172.21.15.40:0/187175802 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f607c0a11e0 0x7f607c0a3690 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f6088005fd0 tx=0x7f6088005f60 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:17:20.393 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.378+0000 7f60a67bab80 1 Processor -- start 2023-07-14T19:17:20.394 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.378+0000 7f60a67bab80 1 -- start start 2023-07-14T19:17:20.394 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.378+0000 7f60a67bab80 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5601e4724f10 0x5601e4725320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:17:20.394 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.378+0000 7f60a67bab80 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5601e4725860 0x5601e4725e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:17:20.394 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.378+0000 7f60a67bab80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5601e4726460 0x5601e47268d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:17:20.394 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.378+0000 7f60a67bab80 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x5601e4629c60 con 0x5601e4724f10 2023-07-14T19:17:20.394 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.378+0000 7f60a67bab80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5601e467c9f0 con 0x5601e4726460 2023-07-14T19:17:20.394 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.378+0000 7f60a67bab80 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x5601e4675e80 con 0x5601e4725860 2023-07-14T19:17:20.394 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.378+0000 7f6093fff700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5601e4724f10 0x5601e4725320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:17:20.395 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.378+0000 7f6093fff700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5601e4724f10 0x5601e4725320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:34280/0 (socket says 172.21.15.40:34280) 2023-07-14T19:17:20.395 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.378+0000 7f6093fff700 1 -- 172.21.15.40:0/496825202 learned_addr learned my addr 172.21.15.40:0/496825202 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-07-14T19:17:20.395 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.379+0000 7f6093fff700 1 -- 172.21.15.40:0/496825202 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5601e4725860 msgr2=0x5601e4725e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T19:17:20.395 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.379+0000 7f6093fff700 1 --2- 172.21.15.40:0/496825202 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5601e4725860 0x5601e4725e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:20.395 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.379+0000 7f6093fff700 1 -- 172.21.15.40:0/496825202 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5601e4726460 msgr2=0x5601e47268d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T19:17:20.395 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.379+0000 7f6098bbf700 1 --2- 172.21.15.40:0/496825202 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5601e4726460 0x5601e47268d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:17:20.395 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.379+0000 7f6093fff700 1 --2- 172.21.15.40:0/496825202 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5601e4726460 0x5601e47268d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:20.395 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.379+0000 7f6093fff700 1 -- 172.21.15.40:0/496825202 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5601e44a3d90 con 0x5601e4724f10 2023-07-14T19:17:20.396 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.379+0000 7f6098bbf700 1 --2- 172.21.15.40:0/496825202 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5601e4726460 0x5601e47268d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-14T19:17:20.396 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.379+0000 7f6093fff700 1 --2- 172.21.15.40:0/496825202 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5601e4724f10 0x5601e4725320 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f608409b110 tx=0x7f608409adf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d1e96eb2a122bee7 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:17:20.396 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.379+0000 7f607a7fc700 1 -- 172.21.15.40:0/496825202 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f608409f440 con 0x5601e4724f10 2023-07-14T19:17:20.396 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.379+0000 7f607a7fc700 1 -- 172.21.15.40:0/496825202 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f60840a37d0 con 0x5601e4724f10 2023-07-14T19:17:20.396 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.379+0000 7f607a7fc700 1 -- 172.21.15.40:0/496825202 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60840a10b0 con 0x5601e4724f10 2023-07-14T19:17:20.396 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.380+0000 7f60a67bab80 1 -- 172.21.15.40:0/496825202 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5601e4724f10 msgr2=0x5601e4725320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:17:20.396 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.380+0000 7f60a67bab80 1 --2- 172.21.15.40:0/496825202 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5601e4724f10 0x5601e4725320 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f608409b110 tx=0x7f608409adf0 comp rx=0 tx=0).stop 2023-07-14T19:17:20.397 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.380+0000 7f60a67bab80 1 -- 172.21.15.40:0/496825202 shutdown_connections 2023-07-14T19:17:20.397 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.380+0000 7f60a67bab80 1 --2- 172.21.15.40:0/496825202 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5601e4726460 0x5601e47268d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:20.397 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.380+0000 7f60a67bab80 1 --2- 172.21.15.40:0/496825202 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5601e4724f10 0x5601e4725320 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:20.397 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.380+0000 7f60a67bab80 1 --2- 172.21.15.40:0/496825202 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5601e4725860 0x5601e4725e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:20.397 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.380+0000 7f60a67bab80 1 -- 172.21.15.40:0/496825202 >> 172.21.15.40:0/496825202 conn(0x5601e471a630 msgr2=0x5601e471f980 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:17:20.397 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.380+0000 7f60a67bab80 1 -- 172.21.15.40:0/496825202 shutdown_connections 2023-07-14T19:17:20.397 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.380+0000 7f60a67bab80 1 -- 172.21.15.40:0/496825202 wait complete. 2023-07-14T19:17:20.398 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.380+0000 7f60a67bab80 1 -- start start 2023-07-14T19:17:20.398 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.380+0000 7f60a67bab80 1 Processor -- start 2023-07-14T19:17:20.398 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.380+0000 7f60a67bab80 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5601e4724f10 0x5601e472e680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:17:20.398 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.381+0000 7f60a67bab80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5601e4725860 0x5601e472ebc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:17:20.398 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.381+0000 7f60a67bab80 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5601e4726460 0x5601e4735c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:17:20.398 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.381+0000 7f60a67bab80 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x5601e4615a70 con 0x5601e4724f10 2023-07-14T19:17:20.398 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.381+0000 7f60a67bab80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5601e4615770 con 0x5601e4725860 2023-07-14T19:17:20.398 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.381+0000 7f60a67bab80 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x5601e44f98d0 con 0x5601e4726460 2023-07-14T19:17:20.399 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.381+0000 7f6093fff700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5601e4724f10 0x5601e472e680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:17:20.399 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.381+0000 7f60a67bab80 10 client.0 _ll_register_callbacks cb 0x5601e4613710 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-14T19:17:20.399 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.381+0000 7f6093fff700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5601e4724f10 0x5601e472e680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:34286/0 (socket says 172.21.15.40:34286) 2023-07-14T19:17:20.399 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.381+0000 7f6093fff700 1 -- 172.21.15.40:0/2263009382 learned_addr learned my addr 172.21.15.40:0/2263009382 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-07-14T19:17:20.399 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.381+0000 7f60937fe700 1 --2- 172.21.15.40:0/2263009382 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5601e4725860 0x5601e472ebc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:17:20.399 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.381+0000 7f6098bbf700 1 --2- 172.21.15.40:0/2263009382 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5601e4726460 0x5601e4735c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:17:20.399 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.381+0000 7f6093fff700 1 -- 172.21.15.40:0/2263009382 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5601e4726460 msgr2=0x5601e4735c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T19:17:20.399 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.381+0000 7f6093fff700 1 --2- 172.21.15.40:0/2263009382 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5601e4726460 0x5601e4735c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:20.400 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.381+0000 7f6093fff700 1 -- 172.21.15.40:0/2263009382 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5601e4725860 msgr2=0x5601e472ebc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T19:17:20.400 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.381+0000 7f6093fff700 1 --2- 172.21.15.40:0/2263009382 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5601e4725860 0x5601e472ebc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:20.400 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.381+0000 7f6093fff700 1 -- 172.21.15.40:0/2263009382 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f609419de10 con 0x5601e4724f10 2023-07-14T19:17:20.400 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.381+0000 7f6098bbf700 1 --2- 172.21.15.40:0/2263009382 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5601e4726460 0x5601e4735c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-14T19:17:20.400 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.381+0000 7f60937fe700 1 --2- 172.21.15.40:0/2263009382 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5601e4725860 0x5601e472ebc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T19:17:20.400 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.381+0000 7f6093fff700 1 --2- 172.21.15.40:0/2263009382 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5601e4724f10 0x5601e472e680 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f6084097910 tx=0x7f6084005c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5eea104c36014392 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:17:20.401 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.381+0000 7f60737fe700 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2023-07-14T19:17:20.401 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.381+0000 7f60737fe700 1 -- 172.21.15.40:0/2263009382 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60840a10e0 con 0x5601e4724f10 2023-07-14T19:17:20.401 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.381+0000 7f60737fe700 1 -- 172.21.15.40:0/2263009382 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(83..83 src has 1..83) v4 ==== 8269+0+0 (secure 0 0 0) 0x7f60840a1240 con 0x5601e4724f10 2023-07-14T19:17:20.401 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.381+0000 7f60a67bab80 1 -- 172.21.15.40:0/2263009382 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5601e45cf4a0 con 0x5601e4724f10 2023-07-14T19:17:20.401 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.381+0000 7f60737fe700 1 -- 172.21.15.40:0/2263009382 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7f60840a3340 con 0x5601e4724f10 2023-07-14T19:17:20.401 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.381+0000 7f60a67bab80 10 client.15258 fetch_fsmap learned FSMap version 26 2023-07-14T19:17:20.401 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.381+0000 7f60a67bab80 1 -- 172.21.15.40:0/2263009382 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) v3 -- 0x5601e448cc10 con 0x5601e4724f10 2023-07-14T19:17:20.401 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.382+0000 7f60737fe700 1 -- 172.21.15.40:0/2263009382 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x5601e45cf4a0 con 0x5601e4724f10 2023-07-14T19:17:20.402 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.382+0000 7f60737fe700 1 -- 172.21.15.40:0/2263009382 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 26) v1 ==== 7462+0+0 (secure 0 0 0) 0x7f6084022720 con 0x5601e4724f10 2023-07-14T19:17:20.402 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.382+0000 7f60a67bab80 10 client.15258 fetch_fsmap finished waiting for FSMap version 26 2023-07-14T19:17:20.402 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.382+0000 7f60a67bab80 10 client.15258 resolve_mds: resolved name 'i' to daemon mds.i 2023-07-14T19:17:20.402 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.382+0000 7f60a67bab80 20 client.15258 populate_metadata read hostname 'smithi040' 2023-07-14T19:17:20.402 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.382+0000 7f60a67bab80 1 --2- 172.21.15.40:0/2263009382 >> [v2:172.21.15.191:6838/1977507302,v1:172.21.15.191:6839/1977507302] conn(0x5601e4739960 0x5601e4738aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:17:20.402 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.382+0000 7f60a67bab80 4 client.15258 mds_command: new command op to 24508 tid=0[{"prefix": "get_command_descriptions"}] 2023-07-14T19:17:20.402 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.382+0000 7f60a67bab80 1 -- 172.21.15.40:0/2263009382 --> [v2:172.21.15.191:6838/1977507302,v1:172.21.15.191:6839/1977507302] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5601e46f3d50 con 0x5601e4739960 2023-07-14T19:17:20.402 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.382+0000 7f60937fe700 1 --2- 172.21.15.40:0/2263009382 >> [v2:172.21.15.191:6838/1977507302,v1:172.21.15.191:6839/1977507302] conn(0x5601e4739960 0x5601e4738aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:17:20.403 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.382+0000 7f60937fe700 1 --2- 172.21.15.40:0/2263009382 >> [v2:172.21.15.191:6838/1977507302,v1:172.21.15.191:6839/1977507302] conn(0x5601e4739960 0x5601e4738aa0 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=86757b627c1b6770 server_cookie=7179dd0ca95907c1 in_seq=0 out_seq=0 2023-07-14T19:17:20.403 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.382+0000 7f60737fe700 10 client.15258 ms_handle_connect on v2:172.21.15.191:6838/1977507302 2023-07-14T19:17:20.403 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.383+0000 7f60737fe700 1 -- 172.21.15.40:0/2263009382 <== mds.1 v2:172.21.15.191:6838/1977507302 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18017 (crc 0 0 0) 0x5601e46f3d50 con 0x5601e4739960 2023-07-14T19:17:20.403 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.383+0000 7f60737fe700 10 client.15258 handle_command_reply: tid=0 2023-07-14T19:17:20.403 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.383+0000 7f60a67bab80 1 client.15258 shutdown 2023-07-14T19:17:20.403 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.383+0000 7f60a67bab80 1 -- 172.21.15.40:0/2263009382 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5601e4724f10 msgr2=0x5601e472e680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:17:20.403 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.383+0000 7f60a67bab80 1 --2- 172.21.15.40:0/2263009382 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5601e4724f10 0x5601e472e680 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f6084097910 tx=0x7f6084005c00 comp rx=0 tx=0).stop 2023-07-14T19:17:20.404 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.383+0000 7f60a67bab80 1 -- 172.21.15.40:0/2263009382 shutdown_connections 2023-07-14T19:17:20.404 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.383+0000 7f60a67bab80 1 --2- 172.21.15.40:0/2263009382 >> [v2:172.21.15.191:6838/1977507302,v1:172.21.15.191:6839/1977507302] conn(0x5601e4739960 0x5601e4738aa0 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:20.404 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.383+0000 7f60a67bab80 1 --2- 172.21.15.40:0/2263009382 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5601e4726460 0x5601e4735c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:20.404 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.383+0000 7f60737fe700 0 client.15258 ms_handle_reset on v2:172.21.15.191:6838/1977507302 2023-07-14T19:17:20.404 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.383+0000 7f60a67bab80 1 --2- 172.21.15.40:0/2263009382 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5601e4724f10 0x5601e472e680 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:20.404 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.383+0000 7f60a67bab80 1 --2- 172.21.15.40:0/2263009382 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5601e4725860 0x5601e472ebc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:20.404 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.383+0000 7f60a67bab80 1 -- 172.21.15.40:0/2263009382 >> 172.21.15.40:0/2263009382 conn(0x5601e471a630 msgr2=0x5601e471e600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:17:20.404 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.383+0000 7f60a67bab80 1 -- 172.21.15.40:0/2263009382 shutdown_connections 2023-07-14T19:17:20.405 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.383+0000 7f60a67bab80 1 -- 172.21.15.40:0/2263009382 wait complete. 2023-07-14T19:17:20.405 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.383+0000 7f60a67bab80 20 client.15258 trim_cache size 0 max 16384 2023-07-14T19:17:20.409 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.401+0000 7f60a67bab80 1 Processor -- start 2023-07-14T19:17:20.409 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.401+0000 7f60a67bab80 1 -- start start 2023-07-14T19:17:20.409 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.401+0000 7f60a67bab80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5601e4724f10 0x5601e472d350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:17:20.409 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.401+0000 7f60a67bab80 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5601e4725860 0x5601e4735bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:17:20.410 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.401+0000 7f60a67bab80 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5601e4726460 0x5601e472c840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:17:20.410 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.401+0000 7f60a67bab80 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x5601e471c6d0 con 0x5601e4726460 2023-07-14T19:17:20.410 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.401+0000 7f60a67bab80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5601e4726310 con 0x5601e4724f10 2023-07-14T19:17:20.410 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.401+0000 7f60a67bab80 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x5601e473a630 con 0x5601e4725860 2023-07-14T19:17:20.410 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.401+0000 7f6098bbf700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5601e4726460 0x5601e472c840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:17:20.410 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.401+0000 7f6098bbf700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5601e4726460 0x5601e472c840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:34298/0 (socket says 172.21.15.40:34298) 2023-07-14T19:17:20.410 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.401+0000 7f6098bbf700 1 -- 172.21.15.40:0/1894967917 learned_addr learned my addr 172.21.15.40:0/1894967917 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-07-14T19:17:20.411 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.402+0000 7f6093fff700 1 --2- 172.21.15.40:0/1894967917 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5601e4724f10 0x5601e472d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:17:20.411 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.402+0000 7f60937fe700 1 --2- 172.21.15.40:0/1894967917 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5601e4725860 0x5601e4735bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:17:20.411 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.402+0000 7f6098bbf700 1 -- 172.21.15.40:0/1894967917 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5601e4725860 msgr2=0x5601e4735bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T19:17:20.411 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.402+0000 7f6098bbf700 1 --2- 172.21.15.40:0/1894967917 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5601e4725860 0x5601e4735bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:20.411 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.402+0000 7f6098bbf700 1 -- 172.21.15.40:0/1894967917 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5601e4724f10 msgr2=0x5601e472d350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T19:17:20.411 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.402+0000 7f6098bbf700 1 --2- 172.21.15.40:0/1894967917 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5601e4724f10 0x5601e472d350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:20.411 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.402+0000 7f6098bbf700 1 -- 172.21.15.40:0/1894967917 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5601e44ca200 con 0x5601e4726460 2023-07-14T19:17:20.412 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.402+0000 7f6098bbf700 1 --2- 172.21.15.40:0/1894967917 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5601e4726460 0x5601e472c840 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f608c007600 tx=0x7f608c00b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8b0a4fa5f7bf5226 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:17:20.412 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.402+0000 7f607affd700 1 -- 172.21.15.40:0/1894967917 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f608c00e040 con 0x5601e4726460 2023-07-14T19:17:20.412 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.402+0000 7f607affd700 1 -- 172.21.15.40:0/1894967917 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f608c00f040 con 0x5601e4726460 2023-07-14T19:17:20.412 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.402+0000 7f607affd700 1 -- 172.21.15.40:0/1894967917 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f608c016ad0 con 0x5601e4726460 2023-07-14T19:17:20.412 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.403+0000 7f60a67bab80 1 -- 172.21.15.40:0/1894967917 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5601e4726460 msgr2=0x5601e472c840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:17:20.412 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.403+0000 7f60a67bab80 1 --2- 172.21.15.40:0/1894967917 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5601e4726460 0x5601e472c840 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f608c007600 tx=0x7f608c00b040 comp rx=0 tx=0).stop 2023-07-14T19:17:20.412 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.403+0000 7f60a67bab80 1 -- 172.21.15.40:0/1894967917 shutdown_connections 2023-07-14T19:17:20.412 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.403+0000 7f60a67bab80 1 --2- 172.21.15.40:0/1894967917 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5601e4726460 0x5601e472c840 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:20.413 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.403+0000 7f60a67bab80 1 --2- 172.21.15.40:0/1894967917 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5601e4724f10 0x5601e472d350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:20.413 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.403+0000 7f60a67bab80 1 --2- 172.21.15.40:0/1894967917 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5601e4725860 0x5601e4735bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:20.413 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.403+0000 7f60a67bab80 1 -- 172.21.15.40:0/1894967917 >> 172.21.15.40:0/1894967917 conn(0x5601e471a630 msgr2=0x5601e4738aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:17:20.413 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.403+0000 7f60a67bab80 1 -- 172.21.15.40:0/1894967917 shutdown_connections 2023-07-14T19:17:20.413 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.403+0000 7f60a67bab80 1 -- 172.21.15.40:0/1894967917 wait complete. 2023-07-14T19:17:20.413 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.403+0000 7f60a67bab80 1 -- start start 2023-07-14T19:17:20.413 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.403+0000 7f60a67bab80 1 Processor -- start 2023-07-14T19:17:20.414 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.403+0000 7f60a67bab80 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5601e4724f10 0x5601e4730e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:17:20.414 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.403+0000 7f60a67bab80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5601e4725860 0x5601e4731340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:17:20.414 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.403+0000 7f60a67bab80 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5601e4726460 0x5601e47341e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:17:20.414 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.403+0000 7f60a67bab80 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x5601e4734720 con 0x5601e4724f10 2023-07-14T19:17:20.414 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.403+0000 7f60a67bab80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5601e4734890 con 0x5601e4725860 2023-07-14T19:17:20.414 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.403+0000 7f60a67bab80 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x5601e4734a50 con 0x5601e4726460 2023-07-14T19:17:20.414 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.403+0000 7f60a67bab80 10 client.0 _ll_register_callbacks cb 0x7f608409b000 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-14T19:17:20.414 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.404+0000 7f6093fff700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5601e4724f10 0x5601e4730e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:17:20.415 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.404+0000 7f6093fff700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5601e4724f10 0x5601e4730e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:34300/0 (socket says 172.21.15.40:34300) 2023-07-14T19:17:20.415 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.404+0000 7f60937fe700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5601e4725860 0x5601e4731340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:17:20.415 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.404+0000 7f6093fff700 1 -- 172.21.15.40:0/3545853702 learned_addr learned my addr 172.21.15.40:0/3545853702 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-07-14T19:17:20.415 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.404+0000 7f6098bbf700 1 --2- 172.21.15.40:0/3545853702 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5601e4726460 0x5601e47341e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:17:20.415 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.404+0000 7f6093fff700 1 -- 172.21.15.40:0/3545853702 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5601e4726460 msgr2=0x5601e47341e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T19:17:20.415 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.404+0000 7f6093fff700 1 --2- 172.21.15.40:0/3545853702 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5601e4726460 0x5601e47341e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:20.415 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.404+0000 7f6093fff700 1 -- 172.21.15.40:0/3545853702 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5601e4725860 msgr2=0x5601e4731340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T19:17:20.415 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.404+0000 7f6093fff700 1 --2- 172.21.15.40:0/3545853702 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5601e4725860 0x5601e4731340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:20.416 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.404+0000 7f6093fff700 1 -- 172.21.15.40:0/3545853702 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f608c0092f0 con 0x5601e4724f10 2023-07-14T19:17:20.416 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.404+0000 7f6098bbf700 1 --2- 172.21.15.40:0/3545853702 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5601e4726460 0x5601e47341e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T19:17:20.416 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.404+0000 7f60937fe700 1 --2- 172.21.15.40:0/3545853702 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5601e4725860 0x5601e4731340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-14T19:17:20.416 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.404+0000 7f6093fff700 1 --2- 172.21.15.40:0/3545853702 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5601e4724f10 0x5601e4730e00 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f6084097e80 tx=0x7f60840a13e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5c76897f62a162cc server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:17:20.416 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.404+0000 7f60737fe700 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2023-07-14T19:17:20.416 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.404+0000 7f60737fe700 1 -- 172.21.15.40:0/3545853702 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60840cbdb0 con 0x5601e4724f10 2023-07-14T19:17:20.417 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.404+0000 7f60737fe700 1 -- 172.21.15.40:0/3545853702 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(83..83 src has 1..83) v4 ==== 8269+0+0 (secure 0 0 0) 0x7f608400fdb0 con 0x5601e4724f10 2023-07-14T19:17:20.417 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.404+0000 7f60a67bab80 1 -- 172.21.15.40:0/3545853702 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f60840a37d0 con 0x5601e4724f10 2023-07-14T19:17:20.417 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.404+0000 7f60737fe700 1 -- 172.21.15.40:0/3545853702 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6084022b30 con 0x5601e4724f10 2023-07-14T19:17:20.417 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.404+0000 7f60a67bab80 10 client.15264 fetch_fsmap learned FSMap version 26 2023-07-14T19:17:20.417 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.404+0000 7f60a67bab80 1 -- 172.21.15.40:0/3545853702 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) v3 -- 0x5601e44a2260 con 0x5601e4724f10 2023-07-14T19:17:20.417 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.405+0000 7f60737fe700 1 -- 172.21.15.40:0/3545853702 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f60840a37d0 con 0x5601e4724f10 2023-07-14T19:17:20.417 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.405+0000 7f60737fe700 1 -- 172.21.15.40:0/3545853702 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 26) v1 ==== 7462+0+0 (secure 0 0 0) 0x7f608409a370 con 0x5601e4724f10 2023-07-14T19:17:20.417 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.405+0000 7f60a67bab80 10 client.15264 fetch_fsmap finished waiting for FSMap version 26 2023-07-14T19:17:20.418 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.405+0000 7f60a67bab80 10 client.15264 resolve_mds: resolved name 'i' to daemon mds.i 2023-07-14T19:17:20.418 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.405+0000 7f60a67bab80 20 client.15264 populate_metadata read hostname 'smithi040' 2023-07-14T19:17:20.418 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.405+0000 7f60a67bab80 1 --2- 172.21.15.40:0/3545853702 >> [v2:172.21.15.191:6838/1977507302,v1:172.21.15.191:6839/1977507302] conn(0x5601e4791fd0 0x5601e4735c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:17:20.418 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.405+0000 7f60a67bab80 4 client.15264 mds_command: new command op to 24508 tid=0[{"prefix": "status"}] 2023-07-14T19:17:20.418 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.405+0000 7f60a67bab80 1 -- 172.21.15.40:0/3545853702 --> [v2:172.21.15.191:6838/1977507302,v1:172.21.15.191:6839/1977507302] -- command(tid 0: {"prefix": "status"}) v1 -- 0x5601e466ee60 con 0x5601e4791fd0 2023-07-14T19:17:20.418 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.405+0000 7f60937fe700 1 --2- 172.21.15.40:0/3545853702 >> [v2:172.21.15.191:6838/1977507302,v1:172.21.15.191:6839/1977507302] conn(0x5601e4791fd0 0x5601e4735c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:17:20.418 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.405+0000 7f60737fe700 10 client.15264 ms_handle_connect on v2:172.21.15.191:6838/1977507302 2023-07-14T19:17:20.418 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.405+0000 7f60937fe700 1 --2- 172.21.15.40:0/3545853702 >> [v2:172.21.15.191:6838/1977507302,v1:172.21.15.191:6839/1977507302] conn(0x5601e4791fd0 0x5601e4735c40 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=b22a025eff6cf1 server_cookie=48127a67c9fba8f1 in_seq=0 out_seq=0 2023-07-14T19:17:20.419 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.406+0000 7f60737fe700 1 -- 172.21.15.40:0/3545853702 <== mds.1 v2:172.21.15.191:6838/1977507302 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+332 (crc 0 0 0) 0x5601e466ee60 con 0x5601e4791fd0 2023-07-14T19:17:20.419 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.406+0000 7f60737fe700 10 client.15264 handle_command_reply: tid=0 2023-07-14T19:17:20.419 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.406+0000 7f60a67bab80 1 client.15264 shutdown 2023-07-14T19:17:20.419 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.406+0000 7f60a67bab80 1 -- 172.21.15.40:0/3545853702 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5601e4724f10 msgr2=0x5601e4730e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:17:20.419 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.406+0000 7f60a67bab80 1 --2- 172.21.15.40:0/3545853702 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5601e4724f10 0x5601e4730e00 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f6084097e80 tx=0x7f60840a13e0 comp rx=0 tx=0).stop 2023-07-14T19:17:20.419 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.406+0000 7f60a67bab80 1 -- 172.21.15.40:0/3545853702 shutdown_connections 2023-07-14T19:17:20.419 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.406+0000 7f60a67bab80 1 --2- 172.21.15.40:0/3545853702 >> [v2:172.21.15.191:6838/1977507302,v1:172.21.15.191:6839/1977507302] conn(0x5601e4791fd0 0x5601e4735c40 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:20.420 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.406+0000 7f60a67bab80 1 --2- 172.21.15.40:0/3545853702 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x5601e4726460 0x5601e47341e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:20.420 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.406+0000 7f60737fe700 0 client.15264 ms_handle_reset on v2:172.21.15.191:6838/1977507302 2023-07-14T19:17:20.420 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.406+0000 7f60a67bab80 1 --2- 172.21.15.40:0/3545853702 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5601e4724f10 0x5601e4730e00 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:20.420 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.406+0000 7f60a67bab80 1 --2- 172.21.15.40:0/3545853702 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5601e4725860 0x5601e4731340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:20.420 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.406+0000 7f60a67bab80 1 -- 172.21.15.40:0/3545853702 >> 172.21.15.40:0/3545853702 conn(0x5601e471a630 msgr2=0x5601e472d350 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:17:20.420 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.406+0000 7f60a67bab80 1 -- 172.21.15.40:0/3545853702 shutdown_connections 2023-07-14T19:17:20.421 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.406+0000 7f60a67bab80 1 -- 172.21.15.40:0/3545853702 wait complete. 2023-07-14T19:17:20.421 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.406+0000 7f60a67bab80 20 client.15264 trim_cache size 0 max 16384 2023-07-14T19:17:20.421 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2023-07-14T19:17:20.421 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_fsid": "adc52f04-2279-11ee-9b33-001a4aab830c", 2023-07-14T19:17:20.422 INFO:teuthology.orchestra.run.smithi040.stdout: "whoami": 1, 2023-07-14T19:17:20.422 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 24508, 2023-07-14T19:17:20.422 INFO:teuthology.orchestra.run.smithi040.stdout: "want_state": "up:active", 2023-07-14T19:17:20.423 INFO:teuthology.orchestra.run.smithi040.stdout: "state": "up:active", 2023-07-14T19:17:20.423 INFO:teuthology.orchestra.run.smithi040.stdout: "fs_name": "cephfs", 2023-07-14T19:17:20.423 INFO:teuthology.orchestra.run.smithi040.stdout: "rank_uptime": 78.844368767999995, 2023-07-14T19:17:20.423 INFO:teuthology.orchestra.run.smithi040.stdout: "mdsmap_epoch": 26, 2023-07-14T19:17:20.424 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap_epoch": 83, 2023-07-14T19:17:20.424 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap_epoch_barrier": 81, 2023-07-14T19:17:20.424 INFO:teuthology.orchestra.run.smithi040.stdout: "uptime": 94.541771894999997 2023-07-14T19:17:20.424 INFO:teuthology.orchestra.run.smithi040.stdout:} 2023-07-14T19:17:20.426 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.406+0000 7f609a622700 1 -- 172.21.15.40:0/187175802 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f607c0a11e0 msgr2=0x7f607c0a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:17:20.426 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.406+0000 7f609a622700 1 --2- 172.21.15.40:0/187175802 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f607c0a11e0 0x7f607c0a3690 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f6088005fd0 tx=0x7f6088005f60 comp rx=0 tx=0).stop 2023-07-14T19:17:20.426 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.406+0000 7f609a622700 1 -- 172.21.15.40:0/187175802 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f60941006f0 msgr2=0x7f6094195920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:17:20.426 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.406+0000 7f609a622700 1 --2- 172.21.15.40:0/187175802 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f60941006f0 0x7f6094195920 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f60840059d0 tx=0x7f60840047a0 comp rx=0 tx=0).stop 2023-07-14T19:17:20.426 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.407+0000 7f609a622700 1 -- 172.21.15.40:0/187175802 shutdown_connections 2023-07-14T19:17:20.426 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.407+0000 7f609a622700 1 --2- 172.21.15.40:0/187175802 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f607c0a11e0 0x7f607c0a3690 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:20.427 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.407+0000 7f609a622700 1 --2- 172.21.15.40:0/187175802 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6094105930 0x7f609419ced0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:20.427 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.407+0000 7f609a622700 1 --2- 172.21.15.40:0/187175802 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f60941006f0 0x7f6094195920 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:20.427 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.407+0000 7f609a622700 1 --2- 172.21.15.40:0/187175802 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6094103010 0x7f6094195e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:20.427 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.407+0000 7f609a622700 1 -- 172.21.15.40:0/187175802 >> 172.21.15.40:0/187175802 conn(0x7f60940fa130 msgr2=0x7f6094101350 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:17:20.427 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.407+0000 7f609a622700 1 -- 172.21.15.40:0/187175802 shutdown_connections 2023-07-14T19:17:20.427 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.407+0000 7f609a622700 1 -- 172.21.15.40:0/187175802 wait complete. 2023-07-14T19:17:20.428 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2023-07-14T19:17:20.554 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.553+0000 7f1657ee4700 1 -- 172.21.15.40:0/614728469 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f16501006c0 msgr2=0x7f1650106f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:17:20.555 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.553+0000 7f1657ee4700 1 --2- 172.21.15.40:0/614728469 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f16501006c0 0x7f1650106f80 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f164800b0a0 tx=0x7f164800b3b0 comp rx=0 tx=0).stop 2023-07-14T19:17:20.555 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.553+0000 7f1657ee4700 1 -- 172.21.15.40:0/614728469 shutdown_connections 2023-07-14T19:17:20.555 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.553+0000 7f1657ee4700 1 --2- 172.21.15.40:0/614728469 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f16500feb40 0x7f16500fef50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:20.555 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.553+0000 7f1657ee4700 1 --2- 172.21.15.40:0/614728469 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f16501006c0 0x7f1650106f80 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:20.555 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.553+0000 7f1657ee4700 1 --2- 172.21.15.40:0/614728469 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f16500ffd30 0x7f1650100180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:20.555 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.553+0000 7f1657ee4700 1 -- 172.21.15.40:0/614728469 >> 172.21.15.40:0/614728469 conn(0x7f16500fa160 msgr2=0x7f16500fc590 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:17:20.555 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.553+0000 7f1657ee4700 1 -- 172.21.15.40:0/614728469 shutdown_connections 2023-07-14T19:17:20.556 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.553+0000 7f1657ee4700 1 -- 172.21.15.40:0/614728469 wait complete. 2023-07-14T19:17:20.557 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.553+0000 7f1657ee4700 1 Processor -- start 2023-07-14T19:17:20.558 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.553+0000 7f1657ee4700 1 -- start start 2023-07-14T19:17:20.558 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.553+0000 7f1657ee4700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f16500feb40 0x7f1650069320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:17:20.558 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.553+0000 7f1657ee4700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f16500ffd30 0x7f165019a7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:17:20.558 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.554+0000 7f1657ee4700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f16501006c0 0x7f165019ce40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:17:20.558 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.554+0000 7f1657ee4700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f165019d610 con 0x7f16501006c0 2023-07-14T19:17:20.558 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.554+0000 7f1657ee4700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f165019d780 con 0x7f16500feb40 2023-07-14T19:17:20.559 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.554+0000 7f1657ee4700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f165019d940 con 0x7f16500ffd30 2023-07-14T19:17:20.559 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.554+0000 7f1656481700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f16501006c0 0x7f165019ce40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:17:20.559 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.554+0000 7f1656481700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f16501006c0 0x7f165019ce40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:34318/0 (socket says 172.21.15.40:34318) 2023-07-14T19:17:20.559 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.554+0000 7f1656481700 1 -- 172.21.15.40:0/2320907117 learned_addr learned my addr 172.21.15.40:0/2320907117 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-07-14T19:17:20.559 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.554+0000 7f1655c80700 1 --2- 172.21.15.40:0/2320907117 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f16500feb40 0x7f1650069320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:17:20.559 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.554+0000 7f165547f700 1 --2- 172.21.15.40:0/2320907117 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f16500ffd30 0x7f165019a7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:17:20.559 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.554+0000 7f1656481700 1 -- 172.21.15.40:0/2320907117 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f16500ffd30 msgr2=0x7f165019a7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:17:20.560 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.554+0000 7f1656481700 1 --2- 172.21.15.40:0/2320907117 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f16500ffd30 0x7f165019a7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:20.560 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.554+0000 7f1656481700 1 -- 172.21.15.40:0/2320907117 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f16500feb40 msgr2=0x7f1650069320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:17:20.560 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.554+0000 7f1656481700 1 --2- 172.21.15.40:0/2320907117 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f16500feb40 0x7f1650069320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:20.560 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.554+0000 7f1656481700 1 -- 172.21.15.40:0/2320907117 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1648009d00 con 0x7f16501006c0 2023-07-14T19:17:20.560 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.554+0000 7f1656481700 1 --2- 172.21.15.40:0/2320907117 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f16501006c0 0x7f165019ce40 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f1648004d50 tx=0x7f1648004d80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:17:20.560 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.554+0000 7f1646ffd700 1 -- 172.21.15.40:0/2320907117 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f164800e040 con 0x7f16501006c0 2023-07-14T19:17:20.560 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.554+0000 7f1646ffd700 1 -- 172.21.15.40:0/2320907117 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f164801d940 con 0x7f16501006c0 2023-07-14T19:17:20.561 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.554+0000 7f1646ffd700 1 -- 172.21.15.40:0/2320907117 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f164801b850 con 0x7f16501006c0 2023-07-14T19:17:20.561 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.554+0000 7f1657ee4700 1 -- 172.21.15.40:0/2320907117 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f165019dbc0 con 0x7f16501006c0 2023-07-14T19:17:20.561 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.554+0000 7f1657ee4700 1 -- 172.21.15.40:0/2320907117 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f165019e040 con 0x7f16501006c0 2023-07-14T19:17:20.565 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.557+0000 7f1646ffd700 1 -- 172.21.15.40:0/2320907117 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7f164801dab0 con 0x7f16501006c0 2023-07-14T19:17:20.565 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.557+0000 7f1646ffd700 1 --2- 172.21.15.40:0/2320907117 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f163c0a1260 0x7f163c0a3710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:17:20.565 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.557+0000 7f1655c80700 1 --2- 172.21.15.40:0/2320907117 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f163c0a1260 0x7f163c0a3710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:17:20.565 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.557+0000 7f1646ffd700 1 -- 172.21.15.40:0/2320907117 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 8269+0+0 (secure 0 0 0) 0x7f16480ce340 con 0x7f16501006c0 2023-07-14T19:17:20.566 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.558+0000 7f1655c80700 1 --2- 172.21.15.40:0/2320907117 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f163c0a1260 0x7f163c0a3710 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f16500ffb90 tx=0x7f164c00b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:17:20.566 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.558+0000 7f166407cb80 1 Processor -- start 2023-07-14T19:17:20.566 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.558+0000 7f166407cb80 1 -- start start 2023-07-14T19:17:20.566 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.559+0000 7f166407cb80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x565221343f10 0x565221344320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:17:20.566 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.559+0000 7f166407cb80 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x565221344860 0x565221344e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:17:20.566 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.559+0000 7f166407cb80 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x565221345460 0x5652213458d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:17:20.566 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.559+0000 7f166407cb80 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x565221248c60 con 0x565221344860 2023-07-14T19:17:20.566 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.559+0000 7f166407cb80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x56522129b9f0 con 0x565221343f10 2023-07-14T19:17:20.567 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.559+0000 7f166407cb80 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x565221294e80 con 0x565221345460 2023-07-14T19:17:20.567 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.559+0000 7f165547f700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x565221344860 0x565221344e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:17:20.567 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.559+0000 7f165547f700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x565221344860 0x565221344e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:34324/0 (socket says 172.21.15.40:34324) 2023-07-14T19:17:20.567 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.559+0000 7f165547f700 1 -- 172.21.15.40:0/700895040 learned_addr learned my addr 172.21.15.40:0/700895040 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-07-14T19:17:20.567 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.559+0000 7f1655c80700 1 --2- 172.21.15.40:0/700895040 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x565221343f10 0x565221344320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:17:20.567 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.559+0000 7f1656481700 1 --2- 172.21.15.40:0/700895040 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x565221345460 0x5652213458d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:17:20.568 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.559+0000 7f165547f700 1 -- 172.21.15.40:0/700895040 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x565221345460 msgr2=0x5652213458d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T19:17:20.568 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.559+0000 7f165547f700 1 --2- 172.21.15.40:0/700895040 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x565221345460 0x5652213458d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:20.568 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.559+0000 7f165547f700 1 -- 172.21.15.40:0/700895040 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x565221343f10 msgr2=0x565221344320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T19:17:20.568 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.559+0000 7f165547f700 1 --2- 172.21.15.40:0/700895040 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x565221343f10 0x565221344320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:20.568 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.559+0000 7f165547f700 1 -- 172.21.15.40:0/700895040 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5652210c2d90 con 0x565221344860 2023-07-14T19:17:20.568 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.559+0000 7f1655c80700 1 --2- 172.21.15.40:0/700895040 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x565221343f10 0x565221344320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T19:17:20.569 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.559+0000 7f165547f700 1 --2- 172.21.15.40:0/700895040 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x565221344860 0x565221344e80 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f1640007c90 tx=0x7f16400103d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=34663b867603c16d server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:17:20.569 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.559+0000 7f163bfff700 1 -- 172.21.15.40:0/700895040 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1640010d60 con 0x565221344860 2023-07-14T19:17:20.569 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.559+0000 7f163bfff700 1 -- 172.21.15.40:0/700895040 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f1640010ec0 con 0x565221344860 2023-07-14T19:17:20.569 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.559+0000 7f163bfff700 1 -- 172.21.15.40:0/700895040 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f164001a470 con 0x565221344860 2023-07-14T19:17:20.569 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.560+0000 7f166407cb80 1 -- 172.21.15.40:0/700895040 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x565221344860 msgr2=0x565221344e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:17:20.569 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.560+0000 7f166407cb80 1 --2- 172.21.15.40:0/700895040 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x565221344860 0x565221344e80 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f1640007c90 tx=0x7f16400103d0 comp rx=0 tx=0).stop 2023-07-14T19:17:20.569 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.560+0000 7f166407cb80 1 -- 172.21.15.40:0/700895040 shutdown_connections 2023-07-14T19:17:20.569 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.560+0000 7f166407cb80 1 --2- 172.21.15.40:0/700895040 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x565221345460 0x5652213458d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:20.570 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.560+0000 7f166407cb80 1 --2- 172.21.15.40:0/700895040 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x565221343f10 0x565221344320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:20.570 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.560+0000 7f166407cb80 1 --2- 172.21.15.40:0/700895040 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x565221344860 0x565221344e80 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:20.570 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.560+0000 7f166407cb80 1 -- 172.21.15.40:0/700895040 >> 172.21.15.40:0/700895040 conn(0x565221339630 msgr2=0x56522133e980 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:17:20.570 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.560+0000 7f166407cb80 1 -- 172.21.15.40:0/700895040 shutdown_connections 2023-07-14T19:17:20.570 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.560+0000 7f166407cb80 1 -- 172.21.15.40:0/700895040 wait complete. 2023-07-14T19:17:20.570 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.561+0000 7f166407cb80 1 -- start start 2023-07-14T19:17:20.570 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.561+0000 7f166407cb80 1 Processor -- start 2023-07-14T19:17:20.571 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.561+0000 7f166407cb80 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x565221343f10 0x56522134d930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:17:20.571 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.561+0000 7f166407cb80 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x565221344860 0x56522134de70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:17:20.571 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.561+0000 7f166407cb80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x565221345460 0x565221354eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:17:20.571 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.561+0000 7f166407cb80 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x565221234a70 con 0x565221344860 2023-07-14T19:17:20.571 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.561+0000 7f166407cb80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x565221234770 con 0x565221345460 2023-07-14T19:17:20.571 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.561+0000 7f166407cb80 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x5652211188d0 con 0x565221343f10 2023-07-14T19:17:20.571 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.561+0000 7f165547f700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x565221344860 0x56522134de70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:17:20.571 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.561+0000 7f166407cb80 10 client.0 _ll_register_callbacks cb 0x565221232710 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-14T19:17:20.572 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.561+0000 7f165547f700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x565221344860 0x56522134de70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:34330/0 (socket says 172.21.15.40:34330) 2023-07-14T19:17:20.572 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.561+0000 7f165547f700 1 -- 172.21.15.40:0/3653366865 learned_addr learned my addr 172.21.15.40:0/3653366865 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-07-14T19:17:20.572 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.562+0000 7f1656481700 1 --2- 172.21.15.40:0/3653366865 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x565221345460 0x565221354eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:17:20.572 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.562+0000 7f1655c80700 1 --2- 172.21.15.40:0/3653366865 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x565221343f10 0x56522134d930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:17:20.572 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.562+0000 7f165547f700 1 -- 172.21.15.40:0/3653366865 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x565221343f10 msgr2=0x56522134d930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T19:17:20.572 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.562+0000 7f165547f700 1 --2- 172.21.15.40:0/3653366865 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x565221343f10 0x56522134d930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:20.572 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.562+0000 7f165547f700 1 -- 172.21.15.40:0/3653366865 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x565221345460 msgr2=0x565221354eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T19:17:20.573 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.562+0000 7f165547f700 1 --2- 172.21.15.40:0/3653366865 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x565221345460 0x565221354eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:20.573 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.562+0000 7f165547f700 1 -- 172.21.15.40:0/3653366865 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f16400078c0 con 0x565221344860 2023-07-14T19:17:20.573 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.562+0000 7f1655c80700 1 --2- 172.21.15.40:0/3653366865 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x565221343f10 0x56522134d930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-14T19:17:20.573 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.562+0000 7f165547f700 1 --2- 172.21.15.40:0/3653366865 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x565221344860 0x56522134de70 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f1640007c60 tx=0x7f16400038b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3e4ec6ec52914913 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:17:20.573 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.562+0000 7f16397fa700 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2023-07-14T19:17:20.573 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.562+0000 7f16397fa700 1 -- 172.21.15.40:0/3653366865 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f164001f070 con 0x565221344860 2023-07-14T19:17:20.573 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.562+0000 7f166407cb80 1 -- 172.21.15.40:0/3653366865 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x565221312d50 con 0x565221344860 2023-07-14T19:17:20.573 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.562+0000 7f16397fa700 1 -- 172.21.15.40:0/3653366865 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(83..83 src has 1..83) v4 ==== 8269+0+0 (secure 0 0 0) 0x7f1640010e30 con 0x565221344860 2023-07-14T19:17:20.574 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.562+0000 7f16397fa700 1 -- 172.21.15.40:0/3653366865 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1640004050 con 0x565221344860 2023-07-14T19:17:20.574 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.562+0000 7f166407cb80 10 client.15276 fetch_fsmap learned FSMap version 26 2023-07-14T19:17:20.574 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.562+0000 7f166407cb80 1 -- 172.21.15.40:0/3653366865 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) v3 -- 0x5652210abc10 con 0x565221344860 2023-07-14T19:17:20.574 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.562+0000 7f16397fa700 1 -- 172.21.15.40:0/3653366865 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f1640032080 con 0x565221344860 2023-07-14T19:17:20.574 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.563+0000 7f16397fa700 1 -- 172.21.15.40:0/3653366865 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 26) v1 ==== 7462+0+0 (secure 0 0 0) 0x7f164002c030 con 0x565221344860 2023-07-14T19:17:20.574 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.563+0000 7f166407cb80 10 client.15276 fetch_fsmap finished waiting for FSMap version 26 2023-07-14T19:17:20.574 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.563+0000 7f166407cb80 10 client.15276 resolve_mds: resolved name 'b' to daemon mds.b 2023-07-14T19:17:20.575 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.563+0000 7f166407cb80 20 client.15276 populate_metadata read hostname 'smithi040' 2023-07-14T19:17:20.575 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.563+0000 7f166407cb80 1 --2- 172.21.15.40:0/3653366865 >> [v2:172.21.15.130:6838/2145479535,v1:172.21.15.130:6839/2145479535] conn(0x565221358aa0 0x565221357f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:17:20.575 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.563+0000 7f166407cb80 4 client.15276 mds_command: new command op to 24539 tid=0[{"prefix": "get_command_descriptions"}] 2023-07-14T19:17:20.575 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.563+0000 7f166407cb80 1 -- 172.21.15.40:0/3653366865 --> [v2:172.21.15.130:6838/2145479535,v1:172.21.15.130:6839/2145479535] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x56522128de60 con 0x565221358aa0 2023-07-14T19:17:20.575 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.563+0000 7f1655c80700 1 --2- 172.21.15.40:0/3653366865 >> [v2:172.21.15.130:6838/2145479535,v1:172.21.15.130:6839/2145479535] conn(0x565221358aa0 0x565221357f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:17:20.575 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.563+0000 7f16397fa700 10 client.15276 ms_handle_connect on v2:172.21.15.130:6838/2145479535 2023-07-14T19:17:20.575 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.563+0000 7f1655c80700 1 --2- 172.21.15.40:0/3653366865 >> [v2:172.21.15.130:6838/2145479535,v1:172.21.15.130:6839/2145479535] conn(0x565221358aa0 0x565221357f50 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4393406662861b65 server_cookie=f0e47bbb0d603e49 in_seq=0 out_seq=0 2023-07-14T19:17:20.576 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.564+0000 7f16397fa700 1 -- 172.21.15.40:0/3653366865 <== mds.0 v2:172.21.15.130:6838/2145479535 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18017 (crc 0 0 0) 0x56522128de60 con 0x565221358aa0 2023-07-14T19:17:20.577 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.564+0000 7f16397fa700 10 client.15276 handle_command_reply: tid=0 2023-07-14T19:17:20.577 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.564+0000 7f166407cb80 1 client.15276 shutdown 2023-07-14T19:17:20.577 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.564+0000 7f166407cb80 1 -- 172.21.15.40:0/3653366865 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x565221344860 msgr2=0x56522134de70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:17:20.577 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.564+0000 7f166407cb80 1 --2- 172.21.15.40:0/3653366865 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x565221344860 0x56522134de70 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f1640007c60 tx=0x7f16400038b0 comp rx=0 tx=0).stop 2023-07-14T19:17:20.577 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.564+0000 7f166407cb80 1 -- 172.21.15.40:0/3653366865 shutdown_connections 2023-07-14T19:17:20.577 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.564+0000 7f166407cb80 1 --2- 172.21.15.40:0/3653366865 >> [v2:172.21.15.130:6838/2145479535,v1:172.21.15.130:6839/2145479535] conn(0x565221358aa0 0x565221357f50 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:20.577 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.564+0000 7f166407cb80 1 --2- 172.21.15.40:0/3653366865 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x565221343f10 0x56522134d930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:20.578 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.564+0000 7f16397fa700 0 client.15276 ms_handle_reset on v2:172.21.15.130:6838/2145479535 2023-07-14T19:17:20.578 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.564+0000 7f166407cb80 1 --2- 172.21.15.40:0/3653366865 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x565221345460 0x565221354eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:20.578 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.564+0000 7f166407cb80 1 --2- 172.21.15.40:0/3653366865 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x565221344860 0x56522134de70 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:20.578 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.564+0000 7f166407cb80 1 -- 172.21.15.40:0/3653366865 >> 172.21.15.40:0/3653366865 conn(0x565221339630 msgr2=0x56522133d600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:17:20.578 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.564+0000 7f166407cb80 1 -- 172.21.15.40:0/3653366865 shutdown_connections 2023-07-14T19:17:20.578 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.564+0000 7f166407cb80 1 -- 172.21.15.40:0/3653366865 wait complete. 2023-07-14T19:17:20.579 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.564+0000 7f166407cb80 20 client.15276 trim_cache size 0 max 16384 2023-07-14T19:17:20.584 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.582+0000 7f166407cb80 1 Processor -- start 2023-07-14T19:17:20.584 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.582+0000 7f166407cb80 1 -- start start 2023-07-14T19:17:20.585 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.582+0000 7f166407cb80 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x565221343f10 0x565221357f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:17:20.585 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.582+0000 7f166407cb80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x565221344860 0x565221354e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:17:20.585 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.582+0000 7f166407cb80 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x565221345460 0x56522134b830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:17:20.585 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.582+0000 7f166407cb80 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x56522134eb20 con 0x565221343f10 2023-07-14T19:17:20.585 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.582+0000 7f166407cb80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x565221263790 con 0x565221344860 2023-07-14T19:17:20.585 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.582+0000 7f166407cb80 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x565221124de0 con 0x565221345460 2023-07-14T19:17:20.585 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.582+0000 7f1655c80700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x565221343f10 0x565221357f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:17:20.585 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.582+0000 7f1655c80700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x565221343f10 0x565221357f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:34336/0 (socket says 172.21.15.40:34336) 2023-07-14T19:17:20.586 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.582+0000 7f1655c80700 1 -- 172.21.15.40:0/3894085606 learned_addr learned my addr 172.21.15.40:0/3894085606 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-07-14T19:17:20.586 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.582+0000 7f165547f700 1 --2- 172.21.15.40:0/3894085606 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x565221344860 0x565221354e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:17:20.586 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.582+0000 7f1656481700 1 --2- 172.21.15.40:0/3894085606 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x565221345460 0x56522134b830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:17:20.586 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.582+0000 7f1655c80700 1 -- 172.21.15.40:0/3894085606 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x565221345460 msgr2=0x56522134b830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T19:17:20.586 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.582+0000 7f1655c80700 1 --2- 172.21.15.40:0/3894085606 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x565221345460 0x56522134b830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:20.586 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.582+0000 7f1655c80700 1 -- 172.21.15.40:0/3894085606 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x565221344860 msgr2=0x565221354e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T19:17:20.586 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.582+0000 7f1655c80700 1 --2- 172.21.15.40:0/3894085606 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x565221344860 0x565221354e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:20.587 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.582+0000 7f1655c80700 1 -- 172.21.15.40:0/3894085606 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5652210e9200 con 0x565221343f10 2023-07-14T19:17:20.587 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.582+0000 7f165547f700 1 --2- 172.21.15.40:0/3894085606 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x565221344860 0x565221354e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-14T19:17:20.587 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.582+0000 7f1656481700 1 --2- 172.21.15.40:0/3894085606 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x565221345460 0x56522134b830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-14T19:17:20.587 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.583+0000 7f1655c80700 1 --2- 172.21.15.40:0/3894085606 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x565221343f10 0x565221357f50 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f164c020760 tx=0x7f164c013000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e11eb5b749e0e4f3 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:17:20.587 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.583+0000 7f1644ff9700 1 -- 172.21.15.40:0/3894085606 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f164c025040 con 0x565221343f10 2023-07-14T19:17:20.587 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.583+0000 7f1644ff9700 1 -- 172.21.15.40:0/3894085606 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f164c015070 con 0x565221343f10 2023-07-14T19:17:20.587 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.583+0000 7f1644ff9700 1 -- 172.21.15.40:0/3894085606 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f164c0118a0 con 0x565221343f10 2023-07-14T19:17:20.590 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.583+0000 7f166407cb80 1 -- 172.21.15.40:0/3894085606 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x565221343f10 msgr2=0x565221357f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:17:20.591 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.583+0000 7f166407cb80 1 --2- 172.21.15.40:0/3894085606 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x565221343f10 0x565221357f50 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f164c020760 tx=0x7f164c013000 comp rx=0 tx=0).stop 2023-07-14T19:17:20.591 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.583+0000 7f166407cb80 1 -- 172.21.15.40:0/3894085606 shutdown_connections 2023-07-14T19:17:20.591 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.583+0000 7f166407cb80 1 --2- 172.21.15.40:0/3894085606 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x565221345460 0x56522134b830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:20.591 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.583+0000 7f166407cb80 1 --2- 172.21.15.40:0/3894085606 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x565221343f10 0x565221357f50 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:20.591 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.583+0000 7f166407cb80 1 --2- 172.21.15.40:0/3894085606 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x565221344860 0x565221354e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:20.591 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.583+0000 7f166407cb80 1 -- 172.21.15.40:0/3894085606 >> 172.21.15.40:0/3894085606 conn(0x565221339630 msgr2=0x56522134c650 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:17:20.592 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.583+0000 7f166407cb80 1 -- 172.21.15.40:0/3894085606 shutdown_connections 2023-07-14T19:17:20.592 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.583+0000 7f166407cb80 1 -- 172.21.15.40:0/3894085606 wait complete. 2023-07-14T19:17:20.592 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.584+0000 7f166407cb80 1 -- start start 2023-07-14T19:17:20.592 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.584+0000 7f166407cb80 1 Processor -- start 2023-07-14T19:17:20.592 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.584+0000 7f166407cb80 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x565221343f10 0x56522133e620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:17:20.592 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.584+0000 7f166407cb80 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x565221344860 0x565221354e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:17:20.592 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.584+0000 7f166407cb80 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x565221345460 0x5652213ac720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:17:20.593 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.584+0000 7f166407cb80 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x56522133eea0 con 0x565221343f10 2023-07-14T19:17:20.593 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.584+0000 7f166407cb80 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x5652213acc60 con 0x565221345460 2023-07-14T19:17:20.593 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.584+0000 7f166407cb80 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x5652213acda0 con 0x565221344860 2023-07-14T19:17:20.593 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.584+0000 7f166407cb80 10 client.0 _ll_register_callbacks cb 0x7f1640007ae0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-14T19:17:20.593 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.584+0000 7f1655c80700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x565221343f10 0x56522133e620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:17:20.593 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.584+0000 7f1655c80700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x565221343f10 0x56522133e620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:34342/0 (socket says 172.21.15.40:34342) 2023-07-14T19:17:20.593 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.584+0000 7f1655c80700 1 -- 172.21.15.40:0/3671886196 learned_addr learned my addr 172.21.15.40:0/3671886196 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-07-14T19:17:20.594 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.584+0000 7f1656481700 1 --2- 172.21.15.40:0/3671886196 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x565221345460 0x5652213ac720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:17:20.594 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.584+0000 7f165547f700 1 --2- 172.21.15.40:0/3671886196 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x565221344860 0x565221354e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:17:20.594 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.584+0000 7f1655c80700 1 -- 172.21.15.40:0/3671886196 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x565221344860 msgr2=0x565221354e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T19:17:20.594 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.584+0000 7f1655c80700 1 --2- 172.21.15.40:0/3671886196 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x565221344860 0x565221354e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:20.594 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.584+0000 7f1655c80700 1 -- 172.21.15.40:0/3671886196 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x565221345460 msgr2=0x5652213ac720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T19:17:20.594 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.584+0000 7f1655c80700 1 --2- 172.21.15.40:0/3671886196 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x565221345460 0x5652213ac720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:20.594 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.584+0000 7f1655c80700 1 -- 172.21.15.40:0/3671886196 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f164c020440 con 0x565221343f10 2023-07-14T19:17:20.595 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.584+0000 7f1656481700 1 --2- 172.21.15.40:0/3671886196 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x565221345460 0x5652213ac720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T19:17:20.595 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.585+0000 7f165547f700 1 --2- 172.21.15.40:0/3671886196 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x565221344860 0x565221354e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-14T19:17:20.595 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.585+0000 7f1655c80700 1 --2- 172.21.15.40:0/3671886196 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x565221343f10 0x56522133e620 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f164c0114b0 tx=0x7f164c00bd90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c706db220a211a78 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:17:20.595 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.585+0000 7f16397fa700 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2023-07-14T19:17:20.595 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.585+0000 7f16397fa700 1 -- 172.21.15.40:0/3671886196 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f164c0289c0 con 0x565221343f10 2023-07-14T19:17:20.595 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.585+0000 7f16397fa700 1 -- 172.21.15.40:0/3671886196 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(83..83 src has 1..83) v4 ==== 8269+0+0 (secure 0 0 0) 0x7f164c017070 con 0x565221343f10 2023-07-14T19:17:20.595 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.585+0000 7f166407cb80 1 -- 172.21.15.40:0/3671886196 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5652211ee4a0 con 0x565221343f10 2023-07-14T19:17:20.595 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.585+0000 7f16397fa700 1 -- 172.21.15.40:0/3671886196 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7f164c028b20 con 0x565221343f10 2023-07-14T19:17:20.596 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.585+0000 7f166407cb80 10 client.15282 fetch_fsmap learned FSMap version 26 2023-07-14T19:17:20.596 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.585+0000 7f166407cb80 1 -- 172.21.15.40:0/3671886196 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) v3 -- 0x5652210c1260 con 0x565221343f10 2023-07-14T19:17:20.596 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.585+0000 7f16397fa700 1 -- 172.21.15.40:0/3671886196 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x5652211ee4a0 con 0x565221343f10 2023-07-14T19:17:20.596 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.585+0000 7f16397fa700 1 -- 172.21.15.40:0/3671886196 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 26) v1 ==== 7462+0+0 (secure 0 0 0) 0x7f164c0115e0 con 0x565221343f10 2023-07-14T19:17:20.596 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.585+0000 7f166407cb80 10 client.15282 fetch_fsmap finished waiting for FSMap version 26 2023-07-14T19:17:20.596 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.585+0000 7f166407cb80 10 client.15282 resolve_mds: resolved name 'b' to daemon mds.b 2023-07-14T19:17:20.596 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.585+0000 7f166407cb80 20 client.15282 populate_metadata read hostname 'smithi040' 2023-07-14T19:17:20.597 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.585+0000 7f166407cb80 1 --2- 172.21.15.40:0/3671886196 >> [v2:172.21.15.130:6838/2145479535,v1:172.21.15.130:6839/2145479535] conn(0x565221358aa0 0x5652213ae3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:17:20.597 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.585+0000 7f166407cb80 4 client.15282 mds_command: new command op to 24539 tid=0[{"prefix": "status"}] 2023-07-14T19:17:20.597 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.586+0000 7f166407cb80 1 -- 172.21.15.40:0/3671886196 --> [v2:172.21.15.130:6838/2145479535,v1:172.21.15.130:6839/2145479535] -- command(tid 0: {"prefix": "status"}) v1 -- 0x565221250000 con 0x565221358aa0 2023-07-14T19:17:20.597 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.586+0000 7f165547f700 1 --2- 172.21.15.40:0/3671886196 >> [v2:172.21.15.130:6838/2145479535,v1:172.21.15.130:6839/2145479535] conn(0x565221358aa0 0x5652213ae3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:17:20.597 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.586+0000 7f16397fa700 10 client.15282 ms_handle_connect on v2:172.21.15.130:6838/2145479535 2023-07-14T19:17:20.597 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.586+0000 7f165547f700 1 --2- 172.21.15.40:0/3671886196 >> [v2:172.21.15.130:6838/2145479535,v1:172.21.15.130:6839/2145479535] conn(0x565221358aa0 0x5652213ae3b0 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f1492f40b25973c7 server_cookie=9172c4edc5f51789 in_seq=0 out_seq=0 2023-07-14T19:17:20.597 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.586+0000 7f16397fa700 1 -- 172.21.15.40:0/3671886196 <== mds.0 v2:172.21.15.130:6838/2145479535 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+332 (crc 0 0 0) 0x565221250000 con 0x565221358aa0 2023-07-14T19:17:20.597 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.586+0000 7f16397fa700 10 client.15282 handle_command_reply: tid=0 2023-07-14T19:17:20.598 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.586+0000 7f166407cb80 1 client.15282 shutdown 2023-07-14T19:17:20.598 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.586+0000 7f166407cb80 1 -- 172.21.15.40:0/3671886196 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x565221343f10 msgr2=0x56522133e620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:17:20.598 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.586+0000 7f166407cb80 1 --2- 172.21.15.40:0/3671886196 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x565221343f10 0x56522133e620 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f164c0114b0 tx=0x7f164c00bd90 comp rx=0 tx=0).stop 2023-07-14T19:17:20.598 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.587+0000 7f166407cb80 1 -- 172.21.15.40:0/3671886196 shutdown_connections 2023-07-14T19:17:20.598 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.587+0000 7f166407cb80 1 --2- 172.21.15.40:0/3671886196 >> [v2:172.21.15.130:6838/2145479535,v1:172.21.15.130:6839/2145479535] conn(0x565221358aa0 0x5652213ae3b0 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:20.598 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.587+0000 7f166407cb80 1 --2- 172.21.15.40:0/3671886196 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x565221345460 0x5652213ac720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:20.599 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.587+0000 7f166407cb80 1 --2- 172.21.15.40:0/3671886196 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x565221343f10 0x56522133e620 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:20.599 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.587+0000 7f166407cb80 1 --2- 172.21.15.40:0/3671886196 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x565221344860 0x565221354e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:20.599 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.587+0000 7f166407cb80 1 -- 172.21.15.40:0/3671886196 >> 172.21.15.40:0/3671886196 conn(0x565221339630 msgr2=0x565221357f50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:17:20.599 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.587+0000 7f16397fa700 0 client.15282 ms_handle_reset on v2:172.21.15.130:6838/2145479535 2023-07-14T19:17:20.599 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.587+0000 7f166407cb80 1 -- 172.21.15.40:0/3671886196 shutdown_connections 2023-07-14T19:17:20.599 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.587+0000 7f166407cb80 1 -- 172.21.15.40:0/3671886196 wait complete. 2023-07-14T19:17:20.599 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.587+0000 7f166407cb80 20 client.15282 trim_cache size 0 max 16384 2023-07-14T19:17:20.600 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2023-07-14T19:17:20.600 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_fsid": "adc52f04-2279-11ee-9b33-001a4aab830c", 2023-07-14T19:17:20.600 INFO:teuthology.orchestra.run.smithi040.stdout: "whoami": 0, 2023-07-14T19:17:20.600 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 24539, 2023-07-14T19:17:20.600 INFO:teuthology.orchestra.run.smithi040.stdout: "want_state": "up:active", 2023-07-14T19:17:20.601 INFO:teuthology.orchestra.run.smithi040.stdout: "state": "up:active", 2023-07-14T19:17:20.601 INFO:teuthology.orchestra.run.smithi040.stdout: "fs_name": "cephfs", 2023-07-14T19:17:20.601 INFO:teuthology.orchestra.run.smithi040.stdout: "rank_uptime": 82.026193504999995, 2023-07-14T19:17:20.601 INFO:teuthology.orchestra.run.smithi040.stdout: "mdsmap_epoch": 26, 2023-07-14T19:17:20.601 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap_epoch": 83, 2023-07-14T19:17:20.601 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap_epoch_barrier": 81, 2023-07-14T19:17:20.601 INFO:teuthology.orchestra.run.smithi040.stdout: "uptime": 87.566024038999998 2023-07-14T19:17:20.602 INFO:teuthology.orchestra.run.smithi040.stdout:} 2023-07-14T19:17:20.603 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.587+0000 7f1657ee4700 1 -- 172.21.15.40:0/2320907117 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f163c0a1260 msgr2=0x7f163c0a3710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:17:20.603 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.587+0000 7f1657ee4700 1 --2- 172.21.15.40:0/2320907117 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f163c0a1260 0x7f163c0a3710 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f16500ffb90 tx=0x7f164c00b410 comp rx=0 tx=0).stop 2023-07-14T19:17:20.603 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.587+0000 7f1657ee4700 1 -- 172.21.15.40:0/2320907117 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f16501006c0 msgr2=0x7f165019ce40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:17:20.603 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.587+0000 7f1657ee4700 1 --2- 172.21.15.40:0/2320907117 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f16501006c0 0x7f165019ce40 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f1648004d50 tx=0x7f1648004d80 comp rx=0 tx=0).stop 2023-07-14T19:17:20.603 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.587+0000 7f1657ee4700 1 -- 172.21.15.40:0/2320907117 shutdown_connections 2023-07-14T19:17:20.603 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.587+0000 7f1657ee4700 1 --2- 172.21.15.40:0/2320907117 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f163c0a1260 0x7f163c0a3710 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:20.604 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.587+0000 7f1657ee4700 1 --2- 172.21.15.40:0/2320907117 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f16501006c0 0x7f165019ce40 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:20.604 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.587+0000 7f1657ee4700 1 --2- 172.21.15.40:0/2320907117 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f16500feb40 0x7f1650069320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:20.604 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.587+0000 7f1657ee4700 1 --2- 172.21.15.40:0/2320907117 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f16500ffd30 0x7f165019a7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:20.604 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.587+0000 7f1657ee4700 1 -- 172.21.15.40:0/2320907117 >> 172.21.15.40:0/2320907117 conn(0x7f16500fa160 msgr2=0x7f16500fc360 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:17:20.604 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.587+0000 7f1657ee4700 1 -- 172.21.15.40:0/2320907117 shutdown_connections 2023-07-14T19:17:20.604 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:20.587+0000 7f1657ee4700 1 -- 172.21.15.40:0/2320907117 wait complete. 2023-07-14T19:17:20.605 INFO:teuthology.run_tasks:Running task kclient... 2023-07-14T19:17:20.629 INFO:tasks.kclient:Mounting kernel clients... 2023-07-14T19:17:20.630 INFO:tasks.kclient:config is {'client.0': {}} 2023-07-14T19:17:20.630 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-14T19:17:20.630 DEBUG:teuthology.orchestra.run.smithi040:> ip netns list 2023-07-14T19:17:20.649 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-14T19:17:20.649 DEBUG:teuthology.orchestra.run.smithi040:> sudo ip link delete ceph-brx 2023-07-14T19:17:20.718 INFO:teuthology.orchestra.run.smithi040.stderr:Cannot find device "ceph-brx" 2023-07-14T19:17:20.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-07-14T19:17:20.720 INFO:tasks.kclient:client.0 config is {'mntopts': ['ms_mode=secure', 'wsync'], 'syntax': 'v2', 'mount_subvol_num': 1} 2023-07-14T19:17:20.720 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2023-07-14T19:17:20.720 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2023-07-14T19:17:20.721 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell --fsid adc52f04-2279-11ee-9b33-001a4aab830c -- ceph fs ls 2023-07-14T19:17:21.337 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/adc52f04-2279-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T19:17:22.643 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:22.640+0000 7f9c9af72700 1 -- 172.21.15.40:0/3495353797 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9c94103ed0 msgr2=0x7f9c9410ba50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:17:22.643 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:22.640+0000 7f9c9af72700 1 --2- 172.21.15.40:0/3495353797 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9c94103ed0 0x7f9c9410ba50 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f9c8c00b0d0 tx=0x7f9c8c00b3e0 comp rx=0 tx=0).stop 2023-07-14T19:17:22.644 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:22.640+0000 7f9c9af72700 1 -- 172.21.15.40:0/3495353797 shutdown_connections 2023-07-14T19:17:22.644 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:22.640+0000 7f9c9af72700 1 --2- 172.21.15.40:0/3495353797 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9c940fee30 0x7f9c940ff200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:22.644 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:22.640+0000 7f9c9af72700 1 --2- 172.21.15.40:0/3495353797 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9c94103ed0 0x7f9c9410ba50 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:22.644 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:22.640+0000 7f9c9af72700 1 --2- 172.21.15.40:0/3495353797 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9c940ff7d0 0x7f9c941037a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:22.645 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:22.640+0000 7f9c9af72700 1 -- 172.21.15.40:0/3495353797 >> 172.21.15.40:0/3495353797 conn(0x7f9c940fa6b0 msgr2=0x7f9c940fcac0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:17:22.645 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:22.640+0000 7f9c9af72700 1 -- 172.21.15.40:0/3495353797 shutdown_connections 2023-07-14T19:17:22.645 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:22.640+0000 7f9c9af72700 1 -- 172.21.15.40:0/3495353797 wait complete. 2023-07-14T19:17:22.645 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:22.641+0000 7f9c9af72700 1 Processor -- start 2023-07-14T19:17:22.645 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:22.641+0000 7f9c9af72700 1 -- start start 2023-07-14T19:17:22.645 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:22.641+0000 7f9c9af72700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9c940fee30 0x7f9c94195ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:17:22.646 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:22.641+0000 7f9c9af72700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9c940ff7d0 0x7f9c94196400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:17:22.646 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:22.641+0000 7f9c9af72700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9c94103ed0 0x7f9c9419a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:17:22.646 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:22.641+0000 7f9c9af72700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c9419ae80 con 0x7f9c940ff7d0 2023-07-14T19:17:22.646 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:22.641+0000 7f9c9af72700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c9419aff0 con 0x7f9c940fee30 2023-07-14T19:17:22.646 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:22.641+0000 7f9c9af72700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c9419b1b0 con 0x7f9c94103ed0 2023-07-14T19:17:22.646 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:22.641+0000 7f9c93fff700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9c940ff7d0 0x7f9c94196400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:17:22.646 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:22.641+0000 7f9c93fff700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9c940ff7d0 0x7f9c94196400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:34356/0 (socket says 172.21.15.40:34356) 2023-07-14T19:17:22.647 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:22.641+0000 7f9c93fff700 1 -- 172.21.15.40:0/2030223476 learned_addr learned my addr 172.21.15.40:0/2030223476 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-07-14T19:17:22.647 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:22.641+0000 7f9c98d0e700 1 --2- 172.21.15.40:0/2030223476 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9c940fee30 0x7f9c94195ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:17:22.647 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:22.641+0000 7f9c93fff700 1 -- 172.21.15.40:0/2030223476 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9c94103ed0 msgr2=0x7f9c9419a870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:17:22.647 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:22.641+0000 7f9c93fff700 1 --2- 172.21.15.40:0/2030223476 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9c94103ed0 0x7f9c9419a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:22.647 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:22.641+0000 7f9c93fff700 1 -- 172.21.15.40:0/2030223476 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9c940fee30 msgr2=0x7f9c94195ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:17:22.647 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:22.641+0000 7f9c93fff700 1 --2- 172.21.15.40:0/2030223476 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9c940fee30 0x7f9c94195ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:22.647 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:22.641+0000 7f9c93fff700 1 -- 172.21.15.40:0/2030223476 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c8c009d00 con 0x7f9c940ff7d0 2023-07-14T19:17:22.647 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:22.642+0000 7f9c93fff700 1 --2- 172.21.15.40:0/2030223476 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9c940ff7d0 0x7f9c94196400 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f9c7c00d980 tx=0x7f9c7c00dd40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:17:22.648 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:22.642+0000 7f9c91ffb700 1 -- 172.21.15.40:0/2030223476 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c7c009b40 con 0x7f9c940ff7d0 2023-07-14T19:17:22.648 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:22.642+0000 7f9c91ffb700 1 -- 172.21.15.40:0/2030223476 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f9c7c00de60 con 0x7f9c940ff7d0 2023-07-14T19:17:22.648 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:22.642+0000 7f9c91ffb700 1 -- 172.21.15.40:0/2030223476 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c7c00f620 con 0x7f9c940ff7d0 2023-07-14T19:17:22.648 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:22.642+0000 7f9c9af72700 1 -- 172.21.15.40:0/2030223476 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9c9419b490 con 0x7f9c940ff7d0 2023-07-14T19:17:22.649 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:22.642+0000 7f9c9af72700 1 -- 172.21.15.40:0/2030223476 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9c941a2cf0 con 0x7f9c940ff7d0 2023-07-14T19:17:22.649 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:22.643+0000 7f9c9af72700 1 -- 172.21.15.40:0/2030223476 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9c941081b0 con 0x7f9c940ff7d0 2023-07-14T19:17:22.649 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:22.644+0000 7f9c91ffb700 1 -- 172.21.15.40:0/2030223476 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 18) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f9c7c010460 con 0x7f9c940ff7d0 2023-07-14T19:17:22.649 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:22.645+0000 7f9c91ffb700 1 --2- 172.21.15.40:0/2030223476 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f9c800a11d0 0x7f9c800a3680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:17:22.650 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:22.645+0000 7f9c98d0e700 1 --2- 172.21.15.40:0/2030223476 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f9c800a11d0 0x7f9c800a3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:17:22.650 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:22.645+0000 7f9c91ffb700 1 -- 172.21.15.40:0/2030223476 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 8269+0+0 (secure 0 0 0) 0x7f9c7c0cc3e0 con 0x7f9c940ff7d0 2023-07-14T19:17:22.650 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:22.645+0000 7f9c98d0e700 1 --2- 172.21.15.40:0/2030223476 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f9c800a11d0 0x7f9c800a3680 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f9c84005950 tx=0x7f9c840058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:17:22.651 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:22.649+0000 7f9c91ffb700 1 -- 172.21.15.40:0/2030223476 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f9c7c095b10 con 0x7f9c940ff7d0 2023-07-14T19:17:22.897 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:22.895+0000 7f9c9af72700 1 -- 172.21.15.40:0/2030223476 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f9c9404e340 con 0x7f9c940ff7d0 2023-07-14T19:17:22.898 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:22.895+0000 7f9c91ffb700 1 -- 172.21.15.40:0/2030223476 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v26) v1 ==== 53+0+73 (secure 0 0 0) 0x7f9c7c099130 con 0x7f9c940ff7d0 2023-07-14T19:17:22.898 INFO:teuthology.orchestra.run.smithi040.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2023-07-14T19:17:22.899 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:22.897+0000 7f9c9af72700 1 -- 172.21.15.40:0/2030223476 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f9c800a11d0 msgr2=0x7f9c800a3680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:17:22.900 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:22.897+0000 7f9c9af72700 1 --2- 172.21.15.40:0/2030223476 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f9c800a11d0 0x7f9c800a3680 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f9c84005950 tx=0x7f9c840058e0 comp rx=0 tx=0).stop 2023-07-14T19:17:22.900 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:22.897+0000 7f9c9af72700 1 -- 172.21.15.40:0/2030223476 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9c940ff7d0 msgr2=0x7f9c94196400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:17:22.900 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:22.897+0000 7f9c9af72700 1 --2- 172.21.15.40:0/2030223476 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9c940ff7d0 0x7f9c94196400 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f9c7c00d980 tx=0x7f9c7c00dd40 comp rx=0 tx=0).stop 2023-07-14T19:17:22.900 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:22.898+0000 7f9c9af72700 1 -- 172.21.15.40:0/2030223476 shutdown_connections 2023-07-14T19:17:22.901 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:22.898+0000 7f9c9af72700 1 --2- 172.21.15.40:0/2030223476 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f9c800a11d0 0x7f9c800a3680 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:22.901 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:22.898+0000 7f9c9af72700 1 --2- 172.21.15.40:0/2030223476 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9c94103ed0 0x7f9c9419a870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:22.901 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:22.898+0000 7f9c9af72700 1 --2- 172.21.15.40:0/2030223476 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9c940fee30 0x7f9c94195ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:22.902 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:22.898+0000 7f9c9af72700 1 --2- 172.21.15.40:0/2030223476 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9c940ff7d0 0x7f9c94196400 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:22.902 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:22.898+0000 7f9c9af72700 1 -- 172.21.15.40:0/2030223476 >> 172.21.15.40:0/2030223476 conn(0x7f9c940fa6b0 msgr2=0x7f9c94068d80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:17:22.902 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:22.898+0000 7f9c9af72700 1 -- 172.21.15.40:0/2030223476 shutdown_connections 2023-07-14T19:17:22.902 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:22.898+0000 7f9c9af72700 1 -- 172.21.15.40:0/2030223476 wait complete. 2023-07-14T19:17:22.926 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:17:22 smithi040 ceph-mon[96518]: pgmap v298: 97 pgs: 97 active+clean; 638 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 10 op/s 2023-07-14T19:17:22.926 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:17:22 smithi040 ceph-mon[96518]: mgrmap e18: x(active, since 7m), standbys: z, y 2023-07-14T19:17:22.972 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:17:22 smithi191 ceph-mon[98311]: pgmap v298: 97 pgs: 97 active+clean; 638 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 10 op/s 2023-07-14T19:17:22.972 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:17:22 smithi191 ceph-mon[98311]: mgrmap e18: x(active, since 7m), standbys: z, y 2023-07-14T19:17:22.992 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:17:22 smithi130 ceph-mon[97832]: pgmap v298: 97 pgs: 97 active+clean; 638 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 10 op/s 2023-07-14T19:17:22.993 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:17:22 smithi130 ceph-mon[97832]: mgrmap e18: x(active, since 7m), standbys: z, y 2023-07-14T19:17:23.386 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2023-07-14T19:17:23.386 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-07-14T19:17:23.387 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi040.front.sepia.ceph.com 2023-07-14T19:17:23.387 INFO:tasks.cephfs.mount:self.client.name = client.0 2023-07-14T19:17:23.387 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2023-07-14T19:17:23.388 INFO:tasks.cephfs.mount:self.cephfs_name = None 2023-07-14T19:17:23.388 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2023-07-14T19:17:23.389 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-07-14T19:17:23.389 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.0' 2023-07-14T19:17:23.390 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-14T19:17:23.390 DEBUG:teuthology.orchestra.run.smithi040:> ip addr 2023-07-14T19:17:23.410 INFO:teuthology.orchestra.run.smithi040.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-07-14T19:17:23.411 INFO:teuthology.orchestra.run.smithi040.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-07-14T19:17:23.411 INFO:teuthology.orchestra.run.smithi040.stdout: inet 127.0.0.1/8 scope host lo 2023-07-14T19:17:23.412 INFO:teuthology.orchestra.run.smithi040.stdout: valid_lft forever preferred_lft forever 2023-07-14T19:17:23.412 INFO:teuthology.orchestra.run.smithi040.stdout: inet6 ::1/128 scope host 2023-07-14T19:17:23.412 INFO:teuthology.orchestra.run.smithi040.stdout: valid_lft forever preferred_lft forever 2023-07-14T19:17:23.412 INFO:teuthology.orchestra.run.smithi040.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-07-14T19:17:23.412 INFO:teuthology.orchestra.run.smithi040.stdout: link/ether 0c:c4:7a:6b:fd:16 brd ff:ff:ff:ff:ff:ff 2023-07-14T19:17:23.412 INFO:teuthology.orchestra.run.smithi040.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-07-14T19:17:23.413 INFO:teuthology.orchestra.run.smithi040.stdout: link/ether 0c:c4:7a:6b:fd:17 brd ff:ff:ff:ff:ff:ff 2023-07-14T19:17:23.413 INFO:teuthology.orchestra.run.smithi040.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2023-07-14T19:17:23.413 INFO:teuthology.orchestra.run.smithi040.stdout: link/ether 0c:c4:7a:bd:12:cc brd ff:ff:ff:ff:ff:ff 2023-07-14T19:17:23.413 INFO:teuthology.orchestra.run.smithi040.stdout: inet 172.21.15.40/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2023-07-14T19:17:23.413 INFO:teuthology.orchestra.run.smithi040.stdout: valid_lft 41598sec preferred_lft 41598sec 2023-07-14T19:17:23.413 INFO:teuthology.orchestra.run.smithi040.stdout: inet6 fe80::ec4:7aff:febd:12cc/64 scope link 2023-07-14T19:17:23.413 INFO:teuthology.orchestra.run.smithi040.stdout: valid_lft forever preferred_lft forever 2023-07-14T19:17:23.414 INFO:teuthology.orchestra.run.smithi040.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-07-14T19:17:23.414 INFO:teuthology.orchestra.run.smithi040.stdout: link/ether 0c:c4:7a:bd:12:cd brd ff:ff:ff:ff:ff:ff 2023-07-14T19:17:23.414 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2023-07-14T19:17:23.414 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-14T19:17:23.415 DEBUG:teuthology.orchestra.run.smithi040:> (cd / && exec bash -c ' 2023-07-14T19:17:23.415 DEBUG:teuthology.orchestra.run.smithi040:> set -e 2023-07-14T19:17:23.415 DEBUG:teuthology.orchestra.run.smithi040:> sudo ip link add name ceph-brx type bridge 2023-07-14T19:17:23.416 DEBUG:teuthology.orchestra.run.smithi040:> sudo ip addr flush dev ceph-brx 2023-07-14T19:17:23.416 DEBUG:teuthology.orchestra.run.smithi040:> sudo ip link set ceph-brx up 2023-07-14T19:17:23.416 DEBUG:teuthology.orchestra.run.smithi040:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2023-07-14T19:17:23.416 DEBUG:teuthology.orchestra.run.smithi040:> ') 2023-07-14T19:17:23.585 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-14T19:17:23.586 DEBUG:teuthology.orchestra.run.smithi040:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-07-14T19:17:23.618 INFO:teuthology.orchestra.run.smithi040.stdout:1 2023-07-14T19:17:23.619 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-14T19:17:23.620 DEBUG:teuthology.orchestra.run.smithi040:> route 2023-07-14T19:17:23.694 INFO:teuthology.orchestra.run.smithi040.stdout:Kernel IP routing table 2023-07-14T19:17:23.695 INFO:teuthology.orchestra.run.smithi040.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-07-14T19:17:23.695 INFO:teuthology.orchestra.run.smithi040.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2023-07-14T19:17:23.695 INFO:teuthology.orchestra.run.smithi040.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2023-07-14T19:17:23.695 INFO:teuthology.orchestra.run.smithi040.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-07-14T19:17:23.696 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-14T19:17:23.696 DEBUG:teuthology.orchestra.run.smithi040:> (cd / && exec bash -c ' 2023-07-14T19:17:23.696 DEBUG:teuthology.orchestra.run.smithi040:> set -e 2023-07-14T19:17:23.696 DEBUG:teuthology.orchestra.run.smithi040:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2023-07-14T19:17:23.697 DEBUG:teuthology.orchestra.run.smithi040:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2023-07-14T19:17:23.697 DEBUG:teuthology.orchestra.run.smithi040:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2023-07-14T19:17:23.697 DEBUG:teuthology.orchestra.run.smithi040:> ') 2023-07-14T19:17:23.869 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:17:23 smithi040 ceph-mon[96518]: from='client.? 172.21.15.40:0/2030223476' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2023-07-14T19:17:23.951 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-14T19:17:23.952 DEBUG:teuthology.orchestra.run.smithi040:> ip netns list 2023-07-14T19:17:23.970 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-14T19:17:23.970 DEBUG:teuthology.orchestra.run.smithi040:> ip netns list-id 2023-07-14T19:17:23.972 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:17:23 smithi191 ceph-mon[98311]: from='client.? 172.21.15.40:0/2030223476' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2023-07-14T19:17:23.992 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:17:23 smithi130 ceph-mon[97832]: from='client.? 172.21.15.40:0/2030223476' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2023-07-14T19:17:24.027 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-14T19:17:24.027 DEBUG:teuthology.orchestra.run.smithi040:> (cd / && exec bash -c ' 2023-07-14T19:17:24.028 DEBUG:teuthology.orchestra.run.smithi040:> set -e 2023-07-14T19:17:24.028 DEBUG:teuthology.orchestra.run.smithi040:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.0 2023-07-14T19:17:24.028 DEBUG:teuthology.orchestra.run.smithi040:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.0 0 2023-07-14T19:17:24.028 DEBUG:teuthology.orchestra.run.smithi040:> ') 2023-07-14T19:17:24.125 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.0' with 192.168.0.1/16 2023-07-14T19:17:24.125 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-14T19:17:24.125 DEBUG:teuthology.orchestra.run.smithi040:> (cd / && exec bash -c ' 2023-07-14T19:17:24.126 DEBUG:teuthology.orchestra.run.smithi040:> set -e 2023-07-14T19:17:24.126 DEBUG:teuthology.orchestra.run.smithi040:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.0 type veth peer name brx.0 2023-07-14T19:17:24.126 DEBUG:teuthology.orchestra.run.smithi040:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2023-07-14T19:17:24.126 DEBUG:teuthology.orchestra.run.smithi040:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set veth0 up 2023-07-14T19:17:24.127 DEBUG:teuthology.orchestra.run.smithi040:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set lo up 2023-07-14T19:17:24.127 DEBUG:teuthology.orchestra.run.smithi040:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip route add default via 192.168.255.254 2023-07-14T19:17:24.127 DEBUG:teuthology.orchestra.run.smithi040:> ') 2023-07-14T19:17:24.360 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-14T19:17:24.361 DEBUG:teuthology.orchestra.run.smithi040:> (cd / && exec bash -c ' 2023-07-14T19:17:24.361 DEBUG:teuthology.orchestra.run.smithi040:> set -e 2023-07-14T19:17:24.361 DEBUG:teuthology.orchestra.run.smithi040:> sudo ip link set brx.0 up 2023-07-14T19:17:24.362 DEBUG:teuthology.orchestra.run.smithi040:> sudo ip link set dev brx.0 master ceph-brx 2023-07-14T19:17:24.362 DEBUG:teuthology.orchestra.run.smithi040:> ') 2023-07-14T19:17:24.462 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-07-14T19:17:24.462 DEBUG:teuthology.orchestra.run.smithi040:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2023-07-14T19:17:24.479 INFO:teuthology.orchestra.run.smithi040.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2023-07-14T19:17:24.479 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-07-14T19:17:24.480 DEBUG:teuthology.orchestra.run.smithi040:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2023-07-14T19:17:24.536 WARNING:tasks.cephfs.mount:You cannot specify both: cephfs_mntpt and mount_subvol_num 2023-07-14T19:17:24.537 INFO:tasks.cephfs.mount:Mounting subvol 1 for now 2023-07-14T19:17:24.537 INFO:tasks.cephfs.kernel_mount:mounting using device: 0@.cephfs=/volumes/_nogroup/sv_1/0ab40c84-70ae-4034-9a8f-8c6c6627b87a 2023-07-14T19:17:24.537 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-14T19:17:24.538 DEBUG:teuthology.orchestra.run.smithi040:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 0@.cephfs=/volumes/_nogroup/sv_1/0ab40c84-70ae-4034-9a8f-8c6c6627b87a /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,ms_mode=secure,wsync,nofallback 2023-07-14T19:17:24.634 INFO:teuthology.orchestra.run.smithi040.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,ms_mode=secure,wsync,nofallback 2023-07-14T19:17:24.634 INFO:teuthology.orchestra.run.smithi040.stdout:mount.ceph: options "norequire_active_mds,norbytes,ms_mode=secure,wsync". 2023-07-14T19:17:24.668 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:17:24 smithi040 ceph-mon[96518]: pgmap v299: 97 pgs: 97 active+clean; 638 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2023-07-14T19:17:24.768 INFO:teuthology.orchestra.run.smithi040.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,ms_mode=secure,wsync,nofallback 2023-07-14T19:17:24.768 INFO:teuthology.orchestra.run.smithi040.stdout:mount.ceph: options "norequire_active_mds,norbytes,ms_mode=secure,wsync". 2023-07-14T19:17:24.769 INFO:teuthology.orchestra.run.smithi040.stdout:mount.ceph: resolved to: "172.21.15.40:3300,172.21.15.130:3300,172.21.15.191:3300" 2023-07-14T19:17:24.769 INFO:teuthology.orchestra.run.smithi040.stdout:mount.ceph: trying mount with new device syntax: 0@adc52f04-2279-11ee-9b33-001a4aab830c.cephfs=/volumes/_nogroup/sv_1/0ab40c84-70ae-4034-9a8f-8c6c6627b87a 2023-07-14T19:17:24.769 INFO:teuthology.orchestra.run.smithi040.stdout:mount.ceph: options "norequire_active_mds,norbytes,ms_mode=secure,wsync,name=0,key=0,mon_addr=172.21.15.40:3300/172.21.15.130:3300/172.21.15.191:3300" will pass to kernel 2023-07-14T19:17:24.769 INFO:teuthology.orchestra.run.smithi040.stdout:mount.ceph: v2 mount support check returned 2 2023-07-14T19:17:24.769 INFO:teuthology.orchestra.run.smithi040.stdout:mount.ceph: kernel does not support v2 syntax 2023-07-14T19:17:24.769 INFO:teuthology.orchestra.run.smithi040.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.40:3300,172.21.15.130:3300,172.21.15.191:3300:/volumes/_nogroup/sv_1/0ab40c84-70ae-4034-9a8f-8c6c6627b87a 2023-07-14T19:17:24.769 INFO:teuthology.orchestra.run.smithi040.stdout:mount.ceph: options "norequire_active_mds,norbytes,ms_mode=secure,wsync,name=0,key=0,mds_namespace=cephfs,fsid=adc52f04-2279-11ee-9b33-001a4aab830c" will pass to kernel 2023-07-14T19:17:24.776 INFO:tasks.cephfs.kernel_mount:mount command passed 2023-07-14T19:17:24.776 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-14T19:17:24.777 DEBUG:teuthology.orchestra.run.smithi040:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2023-07-14T19:17:24.832 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-14T19:17:24.832 DEBUG:teuthology.orchestra.run.smithi040:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2023-07-14T19:17:24.866 INFO:teuthology.orchestra.run.smithi040.stdout:client25012 2023-07-14T19:17:24.867 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2023-07-14T19:17:24.867 DEBUG:teuthology.orchestra.run.smithi040:> dd if=/proc/self/mounts of=/dev/stdout 2023-07-14T19:17:24.925 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-14T19:17:24.926 DEBUG:teuthology.orchestra.run.smithi040:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2023-07-14T19:17:24.971 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:17:24 smithi191 ceph-mon[98311]: pgmap v299: 97 pgs: 97 active+clean; 638 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2023-07-14T19:17:24.992 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:17:24 smithi130 ceph-mon[97832]: pgmap v299: 97 pgs: 97 active+clean; 638 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2023-07-14T19:17:24.999 INFO:teuthology.orchestra.run.smithi040.stdout:client25012 2023-07-14T19:17:24.999 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-14T19:17:25.000 DEBUG:teuthology.orchestra.run.smithi040:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2023-07-14T19:17:25.073 INFO:teuthology.orchestra.run.smithi040.stdout:client25012 2023-07-14T19:17:25.073 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-14T19:17:25.074 DEBUG:teuthology.orchestra.run.smithi040:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'sudo dd if=/sys/kernel/debug/ceph/adc52f04-2279-11ee-9b33-001a4aab830c.client25012/status') 2023-07-14T19:17:25.155 INFO:teuthology.orchestra.run.smithi040.stdout:instance: client.25012 (3)172.21.15.40:0/3905700381 2023-07-14T19:17:25.156 INFO:teuthology.orchestra.run.smithi040.stdout:blocklisted: false 2023-07-14T19:17:25.157 INFO:teuthology.orchestra.run.smithi040.stderr:0+1 records in 2023-07-14T19:17:25.158 INFO:teuthology.orchestra.run.smithi040.stderr:0+1 records out 2023-07-14T19:17:25.158 INFO:teuthology.orchestra.run.smithi040.stderr:71 bytes copied, 2.2635e-05 s, 3.1 MB/s 2023-07-14T19:17:25.158 INFO:teuthology.run_tasks:Running task exec... 2023-07-14T19:17:25.168 INFO:teuthology.task.exec:Executing custom commands... 2023-07-14T19:17:25.169 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi040.front.sepia.ceph.com 2023-07-14T19:17:25.169 DEBUG:teuthology.orchestra.run.smithi040:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs set cephfs balance_automate false' 2023-07-14T19:17:25.328 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:25.324+0000 7fe11ebfc700 1 -- 172.21.15.40:0/2638580006 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe118100770 msgr2=0x7fe118107010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:17:25.328 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:25.324+0000 7fe11ebfc700 1 --2- 172.21.15.40:0/2638580006 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe118100770 0x7fe118107010 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7fe110009ec0 tx=0x7fe11000b330 comp rx=0 tx=0).stop 2023-07-14T19:17:25.328 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:25.324+0000 7fe11ebfc700 1 -- 172.21.15.40:0/2638580006 shutdown_connections 2023-07-14T19:17:25.328 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:25.324+0000 7fe11ebfc700 1 --2- 172.21.15.40:0/2638580006 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe1180feb30 0x7fe1180fef40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:25.329 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:25.324+0000 7fe11ebfc700 1 --2- 172.21.15.40:0/2638580006 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe118100770 0x7fe118107010 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:25.329 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:25.324+0000 7fe11ebfc700 1 --2- 172.21.15.40:0/2638580006 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe1180ffdc0 0x7fe118100230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:25.329 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:25.324+0000 7fe11ebfc700 1 -- 172.21.15.40:0/2638580006 >> 172.21.15.40:0/2638580006 conn(0x7fe1180fa130 msgr2=0x7fe1180fc580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:17:25.329 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:25.324+0000 7fe11ebfc700 1 -- 172.21.15.40:0/2638580006 shutdown_connections 2023-07-14T19:17:25.329 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:25.324+0000 7fe11ebfc700 1 -- 172.21.15.40:0/2638580006 wait complete. 2023-07-14T19:17:25.330 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:25.325+0000 7fe11ebfc700 1 Processor -- start 2023-07-14T19:17:25.330 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:25.325+0000 7fe11ebfc700 1 -- start start 2023-07-14T19:17:25.330 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:25.325+0000 7fe11ebfc700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe1180feb30 0x7fe118198810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:17:25.330 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:25.325+0000 7fe11ebfc700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe1180ffdc0 0x7fe11819ad60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:17:25.331 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:25.325+0000 7fe11ebfc700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe118100770 0x7fe11819d2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:17:25.331 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:25.325+0000 7fe11ebfc700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe11819d880 con 0x7fe1180feb30 2023-07-14T19:17:25.331 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:25.325+0000 7fe11ebfc700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe11819d9c0 con 0x7fe118100770 2023-07-14T19:17:25.332 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:25.325+0000 7fe11ebfc700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe11819db80 con 0x7fe1180ffdc0 2023-07-14T19:17:25.332 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:25.325+0000 7fe11c998700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe1180feb30 0x7fe118198810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:17:25.332 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:25.325+0000 7fe11c998700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe1180feb30 0x7fe118198810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:34380/0 (socket says 172.21.15.40:34380) 2023-07-14T19:17:25.332 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:25.325+0000 7fe11c998700 1 -- 172.21.15.40:0/3053872968 learned_addr learned my addr 172.21.15.40:0/3053872968 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-07-14T19:17:25.333 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:25.325+0000 7fe11d199700 1 --2- 172.21.15.40:0/3053872968 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe118100770 0x7fe11819d2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:17:25.333 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:25.325+0000 7fe117fff700 1 --2- 172.21.15.40:0/3053872968 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe1180ffdc0 0x7fe11819ad60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:17:25.333 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:25.325+0000 7fe11c998700 1 -- 172.21.15.40:0/3053872968 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe1180ffdc0 msgr2=0x7fe11819ad60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:17:25.334 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:25.325+0000 7fe11c998700 1 --2- 172.21.15.40:0/3053872968 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe1180ffdc0 0x7fe11819ad60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:25.334 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:25.325+0000 7fe11c998700 1 -- 172.21.15.40:0/3053872968 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe118100770 msgr2=0x7fe11819d2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:17:25.334 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:25.325+0000 7fe11c998700 1 --2- 172.21.15.40:0/3053872968 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe118100770 0x7fe11819d2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:25.335 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:25.325+0000 7fe11c998700 1 -- 172.21.15.40:0/3053872968 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe110005e70 con 0x7fe1180feb30 2023-07-14T19:17:25.335 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:25.326+0000 7fe11c998700 1 --2- 172.21.15.40:0/3053872968 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe1180feb30 0x7fe118198810 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7fe10c00d950 tx=0x7fe10c00dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:17:25.335 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:25.326+0000 7fe115ffb700 1 -- 172.21.15.40:0/3053872968 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe10c009940 con 0x7fe1180feb30 2023-07-14T19:17:25.335 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:25.326+0000 7fe115ffb700 1 -- 172.21.15.40:0/3053872968 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe10c010460 con 0x7fe1180feb30 2023-07-14T19:17:25.336 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:25.326+0000 7fe11ebfc700 1 -- 172.21.15.40:0/3053872968 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe11819de60 con 0x7fe1180feb30 2023-07-14T19:17:25.336 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:25.326+0000 7fe115ffb700 1 -- 172.21.15.40:0/3053872968 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe10c00f620 con 0x7fe1180feb30 2023-07-14T19:17:25.336 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:25.326+0000 7fe11ebfc700 1 -- 172.21.15.40:0/3053872968 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe11819e280 con 0x7fe1180feb30 2023-07-14T19:17:25.338 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:25.327+0000 7fe11ebfc700 1 -- 172.21.15.40:0/3053872968 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe11804e490 con 0x7fe1180feb30 2023-07-14T19:17:25.338 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:25.328+0000 7fe115ffb700 1 -- 172.21.15.40:0/3053872968 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 18) v1 ==== 135715+0+0 (secure 0 0 0) 0x7fe10c0108d0 con 0x7fe1180feb30 2023-07-14T19:17:25.339 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:25.328+0000 7fe115ffb700 1 --2- 172.21.15.40:0/3053872968 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7fe1080a1150 0x7fe1080a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:17:25.339 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:25.329+0000 7fe115ffb700 1 -- 172.21.15.40:0/3053872968 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 8269+0+0 (secure 0 0 0) 0x7fe10c0ccb50 con 0x7fe1180feb30 2023-07-14T19:17:25.339 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:25.329+0000 7fe117fff700 1 --2- 172.21.15.40:0/3053872968 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7fe1080a1150 0x7fe1080a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:17:25.339 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:25.329+0000 7fe117fff700 1 --2- 172.21.15.40:0/3053872968 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7fe1080a1150 0x7fe1080a3600 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fe118067ee0 tx=0x7fe10400b5b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:17:25.340 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:25.333+0000 7fe115ffb700 1 -- 172.21.15.40:0/3053872968 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fe10c096220 con 0x7fe1180feb30 2023-07-14T19:17:25.581 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:25.579+0000 7fe11ebfc700 1 -- 172.21.15.40:0/3053872968 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"} v 0) v1 -- 0x7fe118106160 con 0x7fe1180feb30 2023-07-14T19:17:25.599 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:25.597+0000 7fe115ffb700 1 -- 172.21.15.40:0/3053872968 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]=0 v27) v1 ==== 118+0+0 (secure 0 0 0) 0x7fe10c0200b0 con 0x7fe1180feb30 2023-07-14T19:17:25.601 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:25.599+0000 7fe11ebfc700 1 -- 172.21.15.40:0/3053872968 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7fe1080a1150 msgr2=0x7fe1080a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:17:25.602 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:25.599+0000 7fe11ebfc700 1 --2- 172.21.15.40:0/3053872968 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7fe1080a1150 0x7fe1080a3600 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fe118067ee0 tx=0x7fe10400b5b0 comp rx=0 tx=0).stop 2023-07-14T19:17:25.602 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:25.599+0000 7fe11ebfc700 1 -- 172.21.15.40:0/3053872968 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe1180feb30 msgr2=0x7fe118198810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:17:25.602 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:25.599+0000 7fe11ebfc700 1 --2- 172.21.15.40:0/3053872968 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe1180feb30 0x7fe118198810 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7fe10c00d950 tx=0x7fe10c00dc60 comp rx=0 tx=0).stop 2023-07-14T19:17:25.602 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:25.599+0000 7fe11ebfc700 1 -- 172.21.15.40:0/3053872968 shutdown_connections 2023-07-14T19:17:25.602 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:25.599+0000 7fe11ebfc700 1 --2- 172.21.15.40:0/3053872968 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7fe1080a1150 0x7fe1080a3600 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:25.603 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:25.599+0000 7fe11ebfc700 1 --2- 172.21.15.40:0/3053872968 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe118100770 0x7fe11819d2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:25.603 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:25.599+0000 7fe11ebfc700 1 --2- 172.21.15.40:0/3053872968 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe1180feb30 0x7fe118198810 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:25.603 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:25.599+0000 7fe11ebfc700 1 --2- 172.21.15.40:0/3053872968 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe1180ffdc0 0x7fe11819ad60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:25.603 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:25.599+0000 7fe11ebfc700 1 -- 172.21.15.40:0/3053872968 >> 172.21.15.40:0/3053872968 conn(0x7fe1180fa130 msgr2=0x7fe1181032a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:17:25.603 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:25.599+0000 7fe11ebfc700 1 -- 172.21.15.40:0/3053872968 shutdown_connections 2023-07-14T19:17:25.603 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:25.600+0000 7fe11ebfc700 1 -- 172.21.15.40:0/3053872968 wait complete. 2023-07-14T19:17:25.613 DEBUG:teuthology.orchestra.run.smithi040:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs subvolumegroup pin cephfs _nogroup random 0.01' 2023-07-14T19:17:25.742 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:25.740+0000 7f9e99998700 1 -- 172.21.15.40:0/3563309930 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9e940ffc60 msgr2=0x7f9e94100070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:17:25.742 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:25.740+0000 7f9e99998700 1 --2- 172.21.15.40:0/3563309930 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9e940ffc60 0x7f9e94100070 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f9e88009990 tx=0x7f9e88009ca0 comp rx=0 tx=0).stop 2023-07-14T19:17:25.742 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:25.740+0000 7f9e99998700 1 -- 172.21.15.40:0/3563309930 shutdown_connections 2023-07-14T19:17:25.743 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:25.740+0000 7f9e99998700 1 --2- 172.21.15.40:0/3563309930 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9e941017e0 0x7f9e941080a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:25.743 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:25.740+0000 7f9e99998700 1 --2- 172.21.15.40:0/3563309930 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9e940ffc60 0x7f9e94100070 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:25.743 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:25.740+0000 7f9e99998700 1 --2- 172.21.15.40:0/3563309930 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9e94100e50 0x7f9e941012a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:25.743 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:25.740+0000 7f9e99998700 1 -- 172.21.15.40:0/3563309930 >> 172.21.15.40:0/3563309930 conn(0x7f9e940fb260 msgr2=0x7f9e940fd6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:17:25.743 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:25.740+0000 7f9e99998700 1 -- 172.21.15.40:0/3563309930 shutdown_connections 2023-07-14T19:17:25.743 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:25.740+0000 7f9e99998700 1 -- 172.21.15.40:0/3563309930 wait complete. 2023-07-14T19:17:25.743 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:25.741+0000 7f9e99998700 1 Processor -- start 2023-07-14T19:17:25.745 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:25.741+0000 7f9e99998700 1 -- start start 2023-07-14T19:17:25.746 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:25.741+0000 7f9e99998700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9e940ffc60 0x7f9e94195ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:17:25.746 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:25.741+0000 7f9e99998700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9e94100e50 0x7f9e94196000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:17:25.746 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:25.741+0000 7f9e99998700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9e941017e0 0x7f9e94196e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:17:25.746 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:25.741+0000 7f9e99998700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e941974e0 con 0x7f9e940ffc60 2023-07-14T19:17:25.746 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:25.741+0000 7f9e99998700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e94197650 con 0x7f9e941017e0 2023-07-14T19:17:25.746 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:25.741+0000 7f9e99998700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e94197810 con 0x7f9e94100e50 2023-07-14T19:17:25.747 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:25.741+0000 7f9e92ffd700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9e940ffc60 0x7f9e94195ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:17:25.747 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:25.741+0000 7f9e92ffd700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9e940ffc60 0x7f9e94195ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:34394/0 (socket says 172.21.15.40:34394) 2023-07-14T19:17:25.747 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:25.741+0000 7f9e92ffd700 1 -- 172.21.15.40:0/1012743270 learned_addr learned my addr 172.21.15.40:0/1012743270 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-07-14T19:17:25.747 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:25.741+0000 7f9e927fc700 1 --2- 172.21.15.40:0/1012743270 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9e94100e50 0x7f9e94196000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:17:25.747 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:25.741+0000 7f9e92ffd700 1 -- 172.21.15.40:0/1012743270 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9e94100e50 msgr2=0x7f9e94196000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:17:25.747 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:25.741+0000 7f9e92ffd700 1 --2- 172.21.15.40:0/1012743270 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9e94100e50 0x7f9e94196000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:25.748 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:25.741+0000 7f9e92ffd700 1 -- 172.21.15.40:0/1012743270 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9e941017e0 msgr2=0x7f9e94196e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:17:25.748 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:25.741+0000 7f9e92ffd700 1 --2- 172.21.15.40:0/1012743270 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9e941017e0 0x7f9e94196e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:25.748 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:25.741+0000 7f9e92ffd700 1 -- 172.21.15.40:0/1012743270 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e880095f0 con 0x7f9e940ffc60 2023-07-14T19:17:25.748 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:25.741+0000 7f9e927fc700 1 --2- 172.21.15.40:0/1012743270 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f9e94100e50 0x7f9e94196000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T19:17:25.748 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:25.742+0000 7f9e92ffd700 1 --2- 172.21.15.40:0/1012743270 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9e940ffc60 0x7f9e94195ac0 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f9e880098e0 tx=0x7f9e88005e80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:17:25.748 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:25.742+0000 7f9e98996700 1 -- 172.21.15.40:0/1012743270 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e8801d070 con 0x7f9e940ffc60 2023-07-14T19:17:25.748 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:25.742+0000 7f9e99998700 1 -- 172.21.15.40:0/1012743270 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9e94197a90 con 0x7f9e940ffc60 2023-07-14T19:17:25.749 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:25.742+0000 7f9e98996700 1 -- 172.21.15.40:0/1012743270 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f9e88004ca0 con 0x7f9e940ffc60 2023-07-14T19:17:25.749 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:25.742+0000 7f9e98996700 1 -- 172.21.15.40:0/1012743270 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e880054b0 con 0x7f9e940ffc60 2023-07-14T19:17:25.749 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:25.742+0000 7f9e99998700 1 -- 172.21.15.40:0/1012743270 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e94197d00 con 0x7f9e940ffc60 2023-07-14T19:17:25.749 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:25.743+0000 7f9e99998700 1 -- 172.21.15.40:0/1012743270 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9e9404e420 con 0x7f9e940ffc60 2023-07-14T19:17:25.749 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:25.744+0000 7f9e98996700 1 -- 172.21.15.40:0/1012743270 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 18) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f9e8800fc40 con 0x7f9e940ffc60 2023-07-14T19:17:25.750 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:25.744+0000 7f9e98996700 1 --2- 172.21.15.40:0/1012743270 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f9e7c0a1150 0x7f9e7c0a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:17:25.750 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:25.744+0000 7f9e98996700 1 -- 172.21.15.40:0/1012743270 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 8269+0+0 (secure 0 0 0) 0x7f9e880ce1b0 con 0x7f9e940ffc60 2023-07-14T19:17:25.751 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:25.744+0000 7f9e927fc700 1 --2- 172.21.15.40:0/1012743270 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f9e7c0a1150 0x7f9e7c0a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:17:25.751 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:25.745+0000 7f9e927fc700 1 --2- 172.21.15.40:0/1012743270 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f9e7c0a1150 0x7f9e7c0a3600 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f9e84005fd0 tx=0x7f9e84005f60 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:17:25.751 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:25.748+0000 7f9e98996700 1 -- 172.21.15.40:0/1012743270 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f9e88097970 con 0x7f9e940ffc60 2023-07-14T19:17:25.970 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:17:25 smithi040 ceph-mon[96518]: from='client.? 172.21.15.40:0/3053872968' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]: dispatch 2023-07-14T19:17:25.971 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:17:25 smithi191 ceph-mon[98311]: from='client.? 172.21.15.40:0/3053872968' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]: dispatch 2023-07-14T19:17:25.986 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:25.985+0000 7f9e99998700 1 -- 172.21.15.40:0/1012743270 --> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] -- mgr_command(tid 0: {"prefix": "fs subvolumegroup pin", "vol_name": "cephfs", "group_name": "_nogroup", "pin_type": "random", "pin_setting": "0.01", "target": ["mon-mgr", ""]}) v1 -- 0x7f9e94060a80 con 0x7f9e7c0a1150 2023-07-14T19:17:25.992 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:17:25 smithi130 ceph-mon[97832]: from='client.? 172.21.15.40:0/3053872968' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]: dispatch 2023-07-14T19:17:26.007 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:26.006+0000 7f9e98996700 1 -- 172.21.15.40:0/1012743270 <== mgr.14152 v2:172.21.15.40:6800/1693027118 1 ==== mgr_command_reply(tid 0: -1 Operation not permitted for group '_nogroup' as it is an internal group.) v1 ==== 80+0+0 (secure 0 0 0) 0x7f9e94060a80 con 0x7f9e7c0a1150 2023-07-14T19:17:26.009 INFO:teuthology.orchestra.run.smithi040.stderr:Error EPERM: Operation not permitted for group '_nogroup' as it is an internal group. 2023-07-14T19:17:26.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-07-14T19:17:26.013 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_aa6a80e6cc655fff48452a7eca02070e61d074fc/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_aa6a80e6cc655fff48452a7eca02070e61d074fc/teuthology/run_tasks.py", line 84, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_aa6a80e6cc655fff48452a7eca02070e61d074fc/teuthology/task/exec.py", line 66, in task remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_aa6a80e6cc655fff48452a7eca02070e61d074fc/teuthology/orchestra/remote.py", line 522, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_aa6a80e6cc655fff48452a7eca02070e61d074fc/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_aa6a80e6cc655fff48452a7eca02070e61d074fc/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_aa6a80e6cc655fff48452a7eca02070e61d074fc/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi040 with status 1: "sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs subvolumegroup pin cephfs _nogroup random 0.01'" 2023-07-14T19:17:26.076 ERROR:teuthology.run_tasks: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=8dd2d741af5b472b961c3ef46ed37445 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_aa6a80e6cc655fff48452a7eca02070e61d074fc/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_aa6a80e6cc655fff48452a7eca02070e61d074fc/teuthology/run_tasks.py", line 84, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_aa6a80e6cc655fff48452a7eca02070e61d074fc/teuthology/task/exec.py", line 66, in task remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_aa6a80e6cc655fff48452a7eca02070e61d074fc/teuthology/orchestra/remote.py", line 522, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_aa6a80e6cc655fff48452a7eca02070e61d074fc/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_aa6a80e6cc655fff48452a7eca02070e61d074fc/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_aa6a80e6cc655fff48452a7eca02070e61d074fc/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi040 with status 1: "sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs subvolumegroup pin cephfs _nogroup random 0.01'" 2023-07-14T19:17:26.078 DEBUG:teuthology.run_tasks:Unwinding manager kclient 2023-07-14T19:17:26.087 INFO:tasks.kclient:Unmounting kernel clients... 2023-07-14T19:17:26.088 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2023-07-14T19:17:26.088 DEBUG:teuthology.orchestra.run.smithi040:> dd if=/proc/self/mounts of=/dev/stdout 2023-07-14T19:17:26.112 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2023-07-14T19:17:26.113 DEBUG:teuthology.orchestra.run.smithi040:> dd if=/proc/self/mounts of=/dev/stdout 2023-07-14T19:17:26.171 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-07-14T19:17:26.339 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:26.337+0000 7f54fb902700 1 -- 172.21.15.40:0/3844908190 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f54f40ffcf0 msgr2=0x7f54f4100140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:17:26.340 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:26.337+0000 7f54fb902700 1 --2- 172.21.15.40:0/3844908190 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f54f40ffcf0 0x7f54f4100140 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f54e4009990 tx=0x7f54e4009ca0 comp rx=0 tx=0).stop 2023-07-14T19:17:26.340 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:26.337+0000 7f54fb902700 1 -- 172.21.15.40:0/3844908190 shutdown_connections 2023-07-14T19:17:26.340 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:26.337+0000 7f54fb902700 1 --2- 172.21.15.40:0/3844908190 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f54f4100680 0x7f54f4106f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:26.340 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:26.337+0000 7f54fb902700 1 --2- 172.21.15.40:0/3844908190 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f54f40feb00 0x7f54f40fef10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:26.340 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:26.337+0000 7f54fb902700 1 --2- 172.21.15.40:0/3844908190 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f54f40ffcf0 0x7f54f4100140 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:26.341 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:26.337+0000 7f54fb902700 1 -- 172.21.15.40:0/3844908190 >> 172.21.15.40:0/3844908190 conn(0x7f54f40fa100 msgr2=0x7f54f40fc550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:17:26.341 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:26.337+0000 7f54fb902700 1 -- 172.21.15.40:0/3844908190 shutdown_connections 2023-07-14T19:17:26.341 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:26.337+0000 7f54fb902700 1 -- 172.21.15.40:0/3844908190 wait complete. 2023-07-14T19:17:26.341 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:26.338+0000 7f54fb902700 1 Processor -- start 2023-07-14T19:17:26.341 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:26.338+0000 7f54fb902700 1 -- start start 2023-07-14T19:17:26.343 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:26.338+0000 7f54fb902700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f54f40feb00 0x7f54f4102a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:17:26.344 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:26.338+0000 7f54fb902700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f54f40ffcf0 0x7f54f41010d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:17:26.344 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:26.338+0000 7f54fb902700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f54f4100680 0x7f54f4101610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:17:26.345 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:26.338+0000 7f54fb902700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54f4101db0 con 0x7f54f40feb00 2023-07-14T19:17:26.345 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:26.338+0000 7f54fb902700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54f4101f20 con 0x7f54f40ffcf0 2023-07-14T19:17:26.345 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:26.338+0000 7f54fb902700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54f419f8e0 con 0x7f54f4100680 2023-07-14T19:17:26.346 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:26.338+0000 7f54f969e700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f54f40feb00 0x7f54f4102a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:17:26.346 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:26.338+0000 7f54f969e700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f54f40feb00 0x7f54f4102a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:34428/0 (socket says 172.21.15.40:34428) 2023-07-14T19:17:26.346 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:26.338+0000 7f54f969e700 1 -- 172.21.15.40:0/1439736300 learned_addr learned my addr 172.21.15.40:0/1439736300 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-07-14T19:17:26.346 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:26.338+0000 7f54f8e9d700 1 --2- 172.21.15.40:0/1439736300 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f54f40ffcf0 0x7f54f41010d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:17:26.346 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:26.338+0000 7f54f9e9f700 1 --2- 172.21.15.40:0/1439736300 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f54f4100680 0x7f54f4101610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:17:26.347 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:26.338+0000 7f54f969e700 1 -- 172.21.15.40:0/1439736300 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f54f4100680 msgr2=0x7f54f4101610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:17:26.347 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:26.338+0000 7f54f969e700 1 --2- 172.21.15.40:0/1439736300 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f54f4100680 0x7f54f4101610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:26.347 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:26.338+0000 7f54f969e700 1 -- 172.21.15.40:0/1439736300 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f54f40ffcf0 msgr2=0x7f54f41010d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:17:26.347 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:26.338+0000 7f54f969e700 1 --2- 172.21.15.40:0/1439736300 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f54f40ffcf0 0x7f54f41010d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:26.347 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:26.338+0000 7f54f969e700 1 -- 172.21.15.40:0/1439736300 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54e40095f0 con 0x7f54f40feb00 2023-07-14T19:17:26.348 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:26.339+0000 7f54f969e700 1 --2- 172.21.15.40:0/1439736300 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f54f40feb00 0x7f54f4102a90 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f54f000c990 tx=0x7f54f000cca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:17:26.348 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:26.339+0000 7f54ea7fc700 1 -- 172.21.15.40:0/1439736300 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54f0007940 con 0x7f54f40feb00 2023-07-14T19:17:26.348 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:26.339+0000 7f54ea7fc700 1 -- 172.21.15.40:0/1439736300 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f54f000f450 con 0x7f54f40feb00 2023-07-14T19:17:26.348 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:26.339+0000 7f54fb902700 1 -- 172.21.15.40:0/1439736300 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f54f419fbc0 con 0x7f54f40feb00 2023-07-14T19:17:26.348 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:26.339+0000 7f54ea7fc700 1 -- 172.21.15.40:0/1439736300 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54f0018660 con 0x7f54f40feb00 2023-07-14T19:17:26.349 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:26.339+0000 7f54fb902700 1 -- 172.21.15.40:0/1439736300 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f54f41a0120 con 0x7f54f40feb00 2023-07-14T19:17:26.349 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:26.340+0000 7f54fb902700 1 -- 172.21.15.40:0/1439736300 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f54f410a2d0 con 0x7f54f40feb00 2023-07-14T19:17:26.349 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:26.341+0000 7f54ea7fc700 1 -- 172.21.15.40:0/1439736300 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 18) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f54f000f8c0 con 0x7f54f40feb00 2023-07-14T19:17:26.349 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:26.341+0000 7f54ea7fc700 1 --2- 172.21.15.40:0/1439736300 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f54e00a1220 0x7f54e00a36d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:17:26.349 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:26.341+0000 7f54ea7fc700 1 -- 172.21.15.40:0/1439736300 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 8269+0+0 (secure 0 0 0) 0x7f54f00cc790 con 0x7f54f40feb00 2023-07-14T19:17:26.350 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:26.341+0000 7f54f8e9d700 1 --2- 172.21.15.40:0/1439736300 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f54e00a1220 0x7f54e00a36d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:17:26.350 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:26.342+0000 7f54f8e9d700 1 --2- 172.21.15.40:0/1439736300 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f54e00a1220 0x7f54e00a36d0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f54e400b5c0 tx=0x7f54e4005960 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:17:26.350 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:26.346+0000 7f54ea7fc700 1 -- 172.21.15.40:0/1439736300 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f54f0095ec0 con 0x7f54f40feb00 2023-07-14T19:17:26.471 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Jul 14 19:17:25 smithi040 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-mgr-x[96762]: 2023-07-14T19:17:25.990+0000 7f2f93b57700 -1 client.0 error registering admin socket command: (17) File exists 2023-07-14T19:17:26.471 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Jul 14 19:17:25 smithi040 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-mgr-x[96762]: 2023-07-14T19:17:25.990+0000 7f2f93b57700 -1 client.0 error registering admin socket command: (17) File exists 2023-07-14T19:17:26.471 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Jul 14 19:17:25 smithi040 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-mgr-x[96762]: 2023-07-14T19:17:25.990+0000 7f2f93b57700 -1 client.0 error registering admin socket command: (17) File exists 2023-07-14T19:17:26.472 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Jul 14 19:17:25 smithi040 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-mgr-x[96762]: 2023-07-14T19:17:25.990+0000 7f2f93b57700 -1 client.0 error registering admin socket command: (17) File exists 2023-07-14T19:17:26.472 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Jul 14 19:17:25 smithi040 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-mgr-x[96762]: 2023-07-14T19:17:25.990+0000 7f2f93b57700 -1 client.0 error registering admin socket command: (17) File exists 2023-07-14T19:17:26.472 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Jul 14 19:17:26 smithi040 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-mgr-x[96762]: 2023-07-14T19:17:26.006+0000 7f2f93b57700 -1 mgr.server reply reply (1) Operation not permitted Operation not permitted for group '_nogroup' as it is an internal group. 2023-07-14T19:17:26.547 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:26.546+0000 7f54fb902700 1 -- 172.21.15.40:0/1439736300 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f54f41a06d0 con 0x7f54f40feb00 2023-07-14T19:17:26.549 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:26.547+0000 7f54ea7fc700 1 -- 172.21.15.40:0/1439736300 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v83) v1 ==== 74+0+22647 (secure 0 0 0) 0x7f54f00994e0 con 0x7f54f40feb00 2023-07-14T19:17:26.549 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-07-14T19:17:26.550 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":83,"fsid":"adc52f04-2279-11ee-9b33-001a4aab830c","created":"2023-07-14T19:08:56.893730+0000","modified":"2023-07-14T19:16:52.763512+0000","last_up_change":"2023-07-14T19:15:16.632979+0000","last_in_change":"2023-07-14T19:14:58.961702+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-07-14T19:12:03.271333+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-07-14T19:15:56.073464+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-07-14T19:15:57.018726+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"0d79356a-95d2-49ad-857b-2092ae9f7e13","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":3599703125},{"type":"v1","addr":"172.21.15.40:6803","nonce":3599703125}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":3599703125},{"type":"v1","addr":"172.21.15.40:6805","nonce":3599703125}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":3599703125},{"type":"v1","addr":"172.21.15.40:6809","nonce":3599703125}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":3599703125},{"type":"v1","addr":"172.21.15.40:6807","nonce":3599703125}]},"public_addr":"172.21.15.40:6803/3599703125","cluster_addr":"172.21.15.40:6805/3599703125","heartbeat_back_addr":"172.21.15.40:6809/3599703125","heartbeat_front_addr":"172.21.15.40:6807/3599703125","state":["exists","up"]},{"osd":1,"uuid":"40f8a3d1-a5e4-473c-b9f2-7600ec22be0d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6800","nonce":2512760702},{"type":"v1","addr":"172.21.15.130:6801","nonce":2512760702}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6802","nonce":2512760702},{"type":"v1","addr":"172.21.15.130:6803","nonce":2512760702}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6806","nonce":2512760702},{"type":"v1","addr":"172.21.15.130:6807","nonce":2512760702}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6804","nonce":2512760702},{"type":"v1","addr":"172.21.15.130:6805","nonce":2512760702}]},"public_addr":"172.21.15.130:6801/2512760702","cluster_addr":"172.21.15.130:6803/2512760702","heartbeat_back_addr":"172.21.15.130:6807/2512760702","heartbeat_front_addr":"172.21.15.130:6805/2512760702","state":["exists","up"]},{"osd":2,"uuid":"5a157891-cb7a-4ea6-9239-f48a7c67c003","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6800","nonce":589569135},{"type":"v1","addr":"172.21.15.191:6801","nonce":589569135}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6802","nonce":589569135},{"type":"v1","addr":"172.21.15.191:6803","nonce":589569135}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6806","nonce":589569135},{"type":"v1","addr":"172.21.15.191:6807","nonce":589569135}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6804","nonce":589569135},{"type":"v1","addr":"172.21.15.191:6805","nonce":589569135}]},"public_addr":"172.21.15.191:6801/589569135","cluster_addr":"172.21.15.191:6803/589569135","heartbeat_back_addr":"172.21.15.191:6807/589569135","heartbeat_front_addr":"172.21.15.191:6805/589569135","state":["exists","up"]},{"osd":3,"uuid":"8020e7dc-ac40-487f-857a-8da40bccb459","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":996287860},{"type":"v1","addr":"172.21.15.40:6811","nonce":996287860}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6812","nonce":996287860},{"type":"v1","addr":"172.21.15.40:6813","nonce":996287860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6816","nonce":996287860},{"type":"v1","addr":"172.21.15.40:6817","nonce":996287860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":996287860},{"type":"v1","addr":"172.21.15.40:6815","nonce":996287860}]},"public_addr":"172.21.15.40:6811/996287860","cluster_addr":"172.21.15.40:6813/996287860","heartbeat_back_addr":"172.21.15.40:6817/996287860","heartbeat_front_addr":"172.21.15.40:6815/996287860","state":["exists","up"]},{"osd":4,"uuid":"c5501b35-f62d-482d-8d22-115c88f0298e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6808","nonce":3507448720},{"type":"v1","addr":"172.21.15.130:6809","nonce":3507448720}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6810","nonce":3507448720},{"type":"v1","addr":"172.21.15.130:6811","nonce":3507448720}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6814","nonce":3507448720},{"type":"v1","addr":"172.21.15.130:6815","nonce":3507448720}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6812","nonce":3507448720},{"type":"v1","addr":"172.21.15.130:6813","nonce":3507448720}]},"public_addr":"172.21.15.130:6809/3507448720","cluster_addr":"172.21.15.130:6811/3507448720","heartbeat_back_addr":"172.21.15.130:6815/3507448720","heartbeat_front_addr":"172.21.15.130:6813/3507448720","state":["exists","up"]},{"osd":5,"uuid":"7770b6d6-5874-493a-986a-84db482cee42","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6808","nonce":1781944663},{"type":"v1","addr":"172.21.15.191:6809","nonce":1781944663}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6810","nonce":1781944663},{"type":"v1","addr":"172.21.15.191:6811","nonce":1781944663}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6814","nonce":1781944663},{"type":"v1","addr":"172.21.15.191:6815","nonce":1781944663}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6812","nonce":1781944663},{"type":"v1","addr":"172.21.15.191:6813","nonce":1781944663}]},"public_addr":"172.21.15.191:6809/1781944663","cluster_addr":"172.21.15.191:6811/1781944663","heartbeat_back_addr":"172.21.15.191:6815/1781944663","heartbeat_front_addr":"172.21.15.191:6813/1781944663","state":["exists","up"]},{"osd":6,"uuid":"ef65ad00-dfee-4300-9961-dfc8573aab06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6818","nonce":693186773},{"type":"v1","addr":"172.21.15.40:6819","nonce":693186773}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6820","nonce":693186773},{"type":"v1","addr":"172.21.15.40:6821","nonce":693186773}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6824","nonce":693186773},{"type":"v1","addr":"172.21.15.40:6825","nonce":693186773}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6822","nonce":693186773},{"type":"v1","addr":"172.21.15.40:6823","nonce":693186773}]},"public_addr":"172.21.15.40:6819/693186773","cluster_addr":"172.21.15.40:6821/693186773","heartbeat_back_addr":"172.21.15.40:6825/693186773","heartbeat_front_addr":"172.21.15.40:6823/693186773","state":["exists","up"]},{"osd":7,"uuid":"1700fbfa-cbfc-4f76-97be-727a3b100740","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":50,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6816","nonce":2384019528},{"type":"v1","addr":"172.21.15.130:6817","nonce":2384019528}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6818","nonce":2384019528},{"type":"v1","addr":"172.21.15.130:6819","nonce":2384019528}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6822","nonce":2384019528},{"type":"v1","addr":"172.21.15.130:6823","nonce":2384019528}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6820","nonce":2384019528},{"type":"v1","addr":"172.21.15.130:6821","nonce":2384019528}]},"public_addr":"172.21.15.130:6817/2384019528","cluster_addr":"172.21.15.130:6819/2384019528","heartbeat_back_addr":"172.21.15.130:6823/2384019528","heartbeat_front_addr":"172.21.15.130:6821/2384019528","state":["exists","up"]},{"osd":8,"uuid":"b76d4033-c1e2-457b-8571-c8a49fe443f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":57,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6816","nonce":2386818417},{"type":"v1","addr":"172.21.15.191:6817","nonce":2386818417}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6818","nonce":2386818417},{"type":"v1","addr":"172.21.15.191:6819","nonce":2386818417}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6822","nonce":2386818417},{"type":"v1","addr":"172.21.15.191:6823","nonce":2386818417}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6820","nonce":2386818417},{"type":"v1","addr":"172.21.15.191:6821","nonce":2386818417}]},"public_addr":"172.21.15.191:6817/2386818417","cluster_addr":"172.21.15.191:6819/2386818417","heartbeat_back_addr":"172.21.15.191:6823/2386818417","heartbeat_front_addr":"172.21.15.191:6821/2386818417","state":["exists","up"]},{"osd":9,"uuid":"df31bf2f-e30e-41b6-b2b9-0e26a07a88bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":64,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6826","nonce":2228380263},{"type":"v1","addr":"172.21.15.40:6827","nonce":2228380263}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6828","nonce":2228380263},{"type":"v1","addr":"172.21.15.40:6829","nonce":2228380263}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":2228380263},{"type":"v1","addr":"172.21.15.40:6833","nonce":2228380263}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6830","nonce":2228380263},{"type":"v1","addr":"172.21.15.40:6831","nonce":2228380263}]},"public_addr":"172.21.15.40:6827/2228380263","cluster_addr":"172.21.15.40:6829/2228380263","heartbeat_back_addr":"172.21.15.40:6833/2228380263","heartbeat_front_addr":"172.21.15.40:6831/2228380263","state":["exists","up"]},{"osd":10,"uuid":"4a93d621-49ee-4ad7-9e49-750614faf912","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6824","nonce":2417067798},{"type":"v1","addr":"172.21.15.130:6825","nonce":2417067798}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6826","nonce":2417067798},{"type":"v1","addr":"172.21.15.130:6827","nonce":2417067798}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6830","nonce":2417067798},{"type":"v1","addr":"172.21.15.130:6831","nonce":2417067798}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6828","nonce":2417067798},{"type":"v1","addr":"172.21.15.130:6829","nonce":2417067798}]},"public_addr":"172.21.15.130:6825/2417067798","cluster_addr":"172.21.15.130:6827/2417067798","heartbeat_back_addr":"172.21.15.130:6831/2417067798","heartbeat_front_addr":"172.21.15.130:6829/2417067798","state":["exists","up"]},{"osd":11,"uuid":"f43a2a6f-82f3-42ac-b77c-4f1b34da6918","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":76,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6824","nonce":2886059446},{"type":"v1","addr":"172.21.15.191:6825","nonce":2886059446}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6826","nonce":2886059446},{"type":"v1","addr":"172.21.15.191:6827","nonce":2886059446}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6830","nonce":2886059446},{"type":"v1","addr":"172.21.15.191:6831","nonce":2886059446}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6828","nonce":2886059446},{"type":"v1","addr":"172.21.15.191:6829","nonce":2886059446}]},"public_addr":"172.21.15.191:6825/2886059446","cluster_addr":"172.21.15.191:6827/2886059446","heartbeat_back_addr":"172.21.15.191:6831/2886059446","heartbeat_front_addr":"172.21.15.191:6829/2886059446","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:11:18.320495+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:11:39.901199+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:12:00.869042+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:12:23.292742+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:12:44.587019+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:13:06.494893+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:13:28.378374+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:13:50.033491+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:14:10.937062+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:14:32.358498+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:14:53.666584+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:15:14.837566+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"2.b","mappings":[{"from":8,"to":11}]},{"pgid":"2.c","mappings":[{"from":2,"to":4}]},{"pgid":"2.1f","mappings":[{"from":8,"to":1}]},{"pgid":"3.e","mappings":[{"from":4,"to":7}]},{"pgid":"3.17","mappings":[{"from":4,"to":8}]},{"pgid":"3.1a","mappings":[{"from":5,"to":6}]},{"pgid":"3.20","mappings":[{"from":5,"to":9}]},{"pgid":"3.27","mappings":[{"from":3,"to":9}]},{"pgid":"3.2c","mappings":[{"from":4,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.40:6801/3698757264":"2023-07-15T19:09:50.732458+0000","172.21.15.40:6800/3698757264":"2023-07-15T19:09:50.732458+0000","172.21.15.40:6800/2569015667":"2023-07-15T19:09:24.441543+0000","172.21.15.40:0/3653961003":"2023-07-15T19:09:24.441543+0000","172.21.15.40:0/3568861680":"2023-07-15T19:09:24.441543+0000","172.21.15.40:0/3249773172":"2023-07-15T19:09:50.732458+0000","172.21.15.40:0/4084215480":"2023-07-15T19:09:24.441543+0000","172.21.15.40:0/1163941760":"2023-07-15T19:09:50.732458+0000","172.21.15.40:6801/2569015667":"2023-07-15T19:09:24.441543+0000","172.21.15.40:0/2326639960":"2023-07-15T19:09:50.732458+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-07-14T19:17:26.551 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:26.549+0000 7f54fb902700 1 -- 172.21.15.40:0/1439736300 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f54e00a1220 msgr2=0x7f54e00a36d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:17:26.551 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:26.549+0000 7f54fb902700 1 --2- 172.21.15.40:0/1439736300 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f54e00a1220 0x7f54e00a36d0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f54e400b5c0 tx=0x7f54e4005960 comp rx=0 tx=0).stop 2023-07-14T19:17:26.551 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:26.549+0000 7f54fb902700 1 -- 172.21.15.40:0/1439736300 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f54f40feb00 msgr2=0x7f54f4102a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:17:26.551 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:26.549+0000 7f54fb902700 1 --2- 172.21.15.40:0/1439736300 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f54f40feb00 0x7f54f4102a90 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f54f000c990 tx=0x7f54f000cca0 comp rx=0 tx=0).stop 2023-07-14T19:17:26.552 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:26.549+0000 7f54fb902700 1 -- 172.21.15.40:0/1439736300 shutdown_connections 2023-07-14T19:17:26.552 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:26.549+0000 7f54fb902700 1 --2- 172.21.15.40:0/1439736300 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f54e00a1220 0x7f54e00a36d0 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:26.552 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:26.549+0000 7f54fb902700 1 --2- 172.21.15.40:0/1439736300 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f54f4100680 0x7f54f4101610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:26.552 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:26.549+0000 7f54fb902700 1 --2- 172.21.15.40:0/1439736300 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f54f40feb00 0x7f54f4102a90 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:26.553 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:26.549+0000 7f54fb902700 1 --2- 172.21.15.40:0/1439736300 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f54f40ffcf0 0x7f54f41010d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:26.553 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:26.549+0000 7f54fb902700 1 -- 172.21.15.40:0/1439736300 >> 172.21.15.40:0/1439736300 conn(0x7f54f40fa100 msgr2=0x7f54f40677a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:17:26.553 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:26.549+0000 7f54fb902700 1 -- 172.21.15.40:0/1439736300 shutdown_connections 2023-07-14T19:17:26.553 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:26.549+0000 7f54fb902700 1 -- 172.21.15.40:0/1439736300 wait complete. 2023-07-14T19:17:26.564 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-07-14T19:12:03.271333+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-07-14T19:15:56.073464+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '80', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-07-14T19:15:57.018726+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '81', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-07-14T19:17:26.565 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-07-14T19:17:26.696 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:26.694+0000 7f0cf566e700 1 -- 172.21.15.40:0/3565405583 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0cf0103050 msgr2=0x7f0cf01034c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:17:26.697 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:26.694+0000 7f0cf566e700 1 --2- 172.21.15.40:0/3565405583 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0cf0103050 0x7f0cf01034c0 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f0ce4009990 tx=0x7f0ce4009ca0 comp rx=0 tx=0).stop 2023-07-14T19:17:26.697 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:26.695+0000 7f0cf566e700 1 -- 172.21.15.40:0/3565405583 shutdown_connections 2023-07-14T19:17:26.697 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:26.695+0000 7f0cf566e700 1 --2- 172.21.15.40:0/3565405583 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0cf0103a00 0x7f0cf010a330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:26.698 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:26.695+0000 7f0cf566e700 1 --2- 172.21.15.40:0/3565405583 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0cf0101e70 0x7f0cf0102280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:26.698 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:26.695+0000 7f0cf566e700 1 --2- 172.21.15.40:0/3565405583 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0cf0103050 0x7f0cf01034c0 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:26.698 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:26.695+0000 7f0cf566e700 1 -- 172.21.15.40:0/3565405583 >> 172.21.15.40:0/3565405583 conn(0x7f0cf00fd4c0 msgr2=0x7f0cf00ff910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:17:26.698 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:26.695+0000 7f0cf566e700 1 -- 172.21.15.40:0/3565405583 shutdown_connections 2023-07-14T19:17:26.698 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:26.695+0000 7f0cf566e700 1 -- 172.21.15.40:0/3565405583 wait complete. 2023-07-14T19:17:26.700 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:26.695+0000 7f0cf566e700 1 Processor -- start 2023-07-14T19:17:26.701 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:26.696+0000 7f0cf566e700 1 -- start start 2023-07-14T19:17:26.701 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:26.696+0000 7f0cf566e700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0cf0101e70 0x7f0cf0197be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:17:26.701 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:26.696+0000 7f0cf566e700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0cf0103050 0x7f0cf0198120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:17:26.701 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:26.696+0000 7f0cf566e700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0cf0103a00 0x7f0cf019f190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:17:26.702 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:26.696+0000 7f0cf566e700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0cf019f760 con 0x7f0cf0103050 2023-07-14T19:17:26.702 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:26.696+0000 7f0cf566e700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0cf019f8a0 con 0x7f0cf0101e70 2023-07-14T19:17:26.702 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:26.696+0000 7f0cf566e700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0cf019fa30 con 0x7f0cf0103a00 2023-07-14T19:17:26.702 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:26.696+0000 7f0cee7fc700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0cf0103050 0x7f0cf0198120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:17:26.702 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:26.696+0000 7f0cee7fc700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0cf0103050 0x7f0cf0198120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:34444/0 (socket says 172.21.15.40:34444) 2023-07-14T19:17:26.703 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:26.696+0000 7f0cee7fc700 1 -- 172.21.15.40:0/1381476529 learned_addr learned my addr 172.21.15.40:0/1381476529 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-07-14T19:17:26.703 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:26.696+0000 7f0cef7fe700 1 --2- 172.21.15.40:0/1381476529 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0cf0103a00 0x7f0cf019f190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:17:26.703 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:26.696+0000 7f0ceeffd700 1 --2- 172.21.15.40:0/1381476529 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0cf0101e70 0x7f0cf0197be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:17:26.703 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:26.696+0000 7f0cee7fc700 1 -- 172.21.15.40:0/1381476529 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0cf0103a00 msgr2=0x7f0cf019f190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:17:26.704 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:26.696+0000 7f0cee7fc700 1 --2- 172.21.15.40:0/1381476529 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0cf0103a00 0x7f0cf019f190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:26.704 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:26.696+0000 7f0cee7fc700 1 -- 172.21.15.40:0/1381476529 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0cf0101e70 msgr2=0x7f0cf0197be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:17:26.704 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:26.696+0000 7f0cee7fc700 1 --2- 172.21.15.40:0/1381476529 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0cf0101e70 0x7f0cf0197be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:26.704 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:26.696+0000 7f0cee7fc700 1 -- 172.21.15.40:0/1381476529 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ce40095f0 con 0x7f0cf0103050 2023-07-14T19:17:26.705 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:26.696+0000 7f0ceeffd700 1 --2- 172.21.15.40:0/1381476529 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0cf0101e70 0x7f0cf0197be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T19:17:26.705 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:26.696+0000 7f0cee7fc700 1 --2- 172.21.15.40:0/1381476529 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0cf0103050 0x7f0cf0198120 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f0ce40098e0 tx=0x7f0ce4005ef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:17:26.705 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:26.697+0000 7f0cd7fff700 1 -- 172.21.15.40:0/1381476529 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ce401d070 con 0x7f0cf0103050 2023-07-14T19:17:26.705 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:26.697+0000 7f0cd7fff700 1 -- 172.21.15.40:0/1381476529 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f0ce4004ca0 con 0x7f0cf0103050 2023-07-14T19:17:26.705 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:26.697+0000 7f0cf566e700 1 -- 172.21.15.40:0/1381476529 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0cf019fcb0 con 0x7f0cf0103050 2023-07-14T19:17:26.705 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:26.697+0000 7f0cd7fff700 1 -- 172.21.15.40:0/1381476529 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ce40055f0 con 0x7f0cf0103050 2023-07-14T19:17:26.706 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:26.697+0000 7f0cf566e700 1 -- 172.21.15.40:0/1381476529 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0cf01a0070 con 0x7f0cf0103050 2023-07-14T19:17:26.706 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:26.698+0000 7f0cf566e700 1 -- 172.21.15.40:0/1381476529 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0cf0108ed0 con 0x7f0cf0103050 2023-07-14T19:17:26.706 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:26.699+0000 7f0cd7fff700 1 -- 172.21.15.40:0/1381476529 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 18) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f0ce400fc70 con 0x7f0cf0103050 2023-07-14T19:17:26.707 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:26.699+0000 7f0cd7fff700 1 --2- 172.21.15.40:0/1381476529 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f0cd80a1150 0x7f0cd80a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:17:26.707 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:26.700+0000 7f0cd7fff700 1 -- 172.21.15.40:0/1381476529 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 8269+0+0 (secure 0 0 0) 0x7f0ce40ce210 con 0x7f0cf0103050 2023-07-14T19:17:26.708 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:26.700+0000 7f0ceeffd700 1 --2- 172.21.15.40:0/1381476529 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f0cd80a1150 0x7f0cd80a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:17:26.708 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:26.700+0000 7f0ceeffd700 1 --2- 172.21.15.40:0/1381476529 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f0cd80a1150 0x7f0cd80a3600 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f0ce0005950 tx=0x7f0ce00058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:17:26.708 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:26.703+0000 7f0cd7fff700 1 -- 172.21.15.40:0/1381476529 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f0ce4097970 con 0x7f0cf0103050 2023-07-14T19:17:26.909 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:26.907+0000 7f0cf566e700 1 -- 172.21.15.40:0/1381476529 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f0cf002cb30 con 0x7f0cf0103050 2023-07-14T19:17:26.909 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:26.907+0000 7f0cd7fff700 1 -- 172.21.15.40:0/1381476529 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v83) v1 ==== 93+0+10 (secure 0 0 0) 0x7f0ce409af90 con 0x7f0cf0103050 2023-07-14T19:17:26.910 INFO:teuthology.orchestra.run.smithi040.stdout:pg_num: 1 2023-07-14T19:17:26.911 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:26.909+0000 7f0cf566e700 1 -- 172.21.15.40:0/1381476529 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f0cd80a1150 msgr2=0x7f0cd80a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:17:26.911 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:26.909+0000 7f0cf566e700 1 --2- 172.21.15.40:0/1381476529 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f0cd80a1150 0x7f0cd80a3600 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f0ce0005950 tx=0x7f0ce00058e0 comp rx=0 tx=0).stop 2023-07-14T19:17:26.911 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:26.909+0000 7f0cf566e700 1 -- 172.21.15.40:0/1381476529 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0cf0103050 msgr2=0x7f0cf0198120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:17:26.912 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:26.909+0000 7f0cf566e700 1 --2- 172.21.15.40:0/1381476529 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0cf0103050 0x7f0cf0198120 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f0ce40098e0 tx=0x7f0ce4005ef0 comp rx=0 tx=0).stop 2023-07-14T19:17:26.912 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:26.909+0000 7f0cf566e700 1 -- 172.21.15.40:0/1381476529 shutdown_connections 2023-07-14T19:17:26.912 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:26.909+0000 7f0cf566e700 1 --2- 172.21.15.40:0/1381476529 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f0cd80a1150 0x7f0cd80a3600 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:26.912 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:26.909+0000 7f0cf566e700 1 --2- 172.21.15.40:0/1381476529 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0cf0103a00 0x7f0cf019f190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:26.912 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:26.909+0000 7f0cf566e700 1 --2- 172.21.15.40:0/1381476529 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0cf0101e70 0x7f0cf0197be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:26.913 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:26.909+0000 7f0cf566e700 1 --2- 172.21.15.40:0/1381476529 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0cf0103050 0x7f0cf0198120 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:26.913 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:26.909+0000 7f0cf566e700 1 -- 172.21.15.40:0/1381476529 >> 172.21.15.40:0/1381476529 conn(0x7f0cf00fd4c0 msgr2=0x7f0cf00ff7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:17:26.913 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:26.909+0000 7f0cf566e700 1 -- 172.21.15.40:0/1381476529 shutdown_connections 2023-07-14T19:17:26.913 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:26.910+0000 7f0cf566e700 1 -- 172.21.15.40:0/1381476529 wait complete. 2023-07-14T19:17:26.922 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:17:26 smithi040 ceph-mon[96518]: pgmap v300: 97 pgs: 97 active+clean; 638 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 0 B/s wr, 9 op/s 2023-07-14T19:17:26.923 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:17:26 smithi040 ceph-mon[96518]: from='client.? 172.21.15.40:0/3053872968' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]': finished 2023-07-14T19:17:26.923 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:17:26 smithi040 ceph-mon[96518]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2023-07-14T19:17:26.923 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:17:26 smithi040 ceph-mon[96518]: from='client.? 172.21.15.40:0/1439736300' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T19:17:26.924 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-07-14T19:17:26.972 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:17:26 smithi191 ceph-mon[98311]: pgmap v300: 97 pgs: 97 active+clean; 638 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 0 B/s wr, 9 op/s 2023-07-14T19:17:26.972 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:17:26 smithi191 ceph-mon[98311]: from='client.? 172.21.15.40:0/3053872968' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]': finished 2023-07-14T19:17:26.972 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:17:26 smithi191 ceph-mon[98311]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2023-07-14T19:17:26.973 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:17:26 smithi191 ceph-mon[98311]: from='client.? 172.21.15.40:0/1439736300' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T19:17:26.992 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:17:26 smithi130 ceph-mon[97832]: pgmap v300: 97 pgs: 97 active+clean; 638 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 0 B/s wr, 9 op/s 2023-07-14T19:17:26.992 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:17:26 smithi130 ceph-mon[97832]: from='client.? 172.21.15.40:0/3053872968' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]': finished 2023-07-14T19:17:26.993 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:17:26 smithi130 ceph-mon[97832]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2023-07-14T19:17:26.993 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:17:26 smithi130 ceph-mon[97832]: from='client.? 172.21.15.40:0/1439736300' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T19:17:27.054 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:27.052+0000 7fcdb3fe1700 1 -- 172.21.15.40:0/2559362402 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcdac0ffd10 msgr2=0x7fcdac100160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:17:27.054 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:27.052+0000 7fcdb3fe1700 1 --2- 172.21.15.40:0/2559362402 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcdac0ffd10 0x7fcdac100160 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7fcda0009990 tx=0x7fcda0009ca0 comp rx=0 tx=0).stop 2023-07-14T19:17:27.054 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:27.052+0000 7fcdb3fe1700 1 -- 172.21.15.40:0/2559362402 shutdown_connections 2023-07-14T19:17:27.055 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:27.052+0000 7fcdb3fe1700 1 --2- 172.21.15.40:0/2559362402 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fcdac0feb20 0x7fcdac0fef30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:27.055 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:27.052+0000 7fcdb3fe1700 1 --2- 172.21.15.40:0/2559362402 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fcdac1006a0 0x7fcdac106f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:27.055 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:27.052+0000 7fcdb3fe1700 1 --2- 172.21.15.40:0/2559362402 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcdac0ffd10 0x7fcdac100160 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:27.055 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:27.052+0000 7fcdb3fe1700 1 -- 172.21.15.40:0/2559362402 >> 172.21.15.40:0/2559362402 conn(0x7fcdac0fa160 msgr2=0x7fcdac0fc570 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:17:27.055 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:27.052+0000 7fcdb3fe1700 1 -- 172.21.15.40:0/2559362402 shutdown_connections 2023-07-14T19:17:27.056 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:27.052+0000 7fcdb3fe1700 1 -- 172.21.15.40:0/2559362402 wait complete. 2023-07-14T19:17:27.057 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:27.053+0000 7fcdb3fe1700 1 Processor -- start 2023-07-14T19:17:27.058 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:27.053+0000 7fcdb3fe1700 1 -- start start 2023-07-14T19:17:27.058 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:27.053+0000 7fcdb3fe1700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcdac0feb20 0x7fcdac194870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:17:27.058 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:27.053+0000 7fcdb3fe1700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fcdac0ffd10 0x7fcdac194db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:17:27.058 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:27.053+0000 7fcdb3fe1700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fcdac1006a0 0x7fcdac19be20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:17:27.059 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:27.053+0000 7fcdb3fe1700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcdac19c3f0 con 0x7fcdac0feb20 2023-07-14T19:17:27.059 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:27.053+0000 7fcdb3fe1700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcdac19c530 con 0x7fcdac0ffd10 2023-07-14T19:17:27.059 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:27.053+0000 7fcdb3fe1700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcdac19c6c0 con 0x7fcdac1006a0 2023-07-14T19:17:27.059 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:27.053+0000 7fcdb1d7d700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcdac0feb20 0x7fcdac194870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:17:27.059 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:27.053+0000 7fcdb1d7d700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcdac0feb20 0x7fcdac194870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:34456/0 (socket says 172.21.15.40:34456) 2023-07-14T19:17:27.060 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:27.053+0000 7fcdb1d7d700 1 -- 172.21.15.40:0/2860924062 learned_addr learned my addr 172.21.15.40:0/2860924062 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-07-14T19:17:27.060 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:27.053+0000 7fcdb157c700 1 --2- 172.21.15.40:0/2860924062 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fcdac0ffd10 0x7fcdac194db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:17:27.060 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:27.053+0000 7fcdb257e700 1 --2- 172.21.15.40:0/2860924062 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fcdac1006a0 0x7fcdac19be20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:17:27.060 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:27.054+0000 7fcdb1d7d700 1 -- 172.21.15.40:0/2860924062 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fcdac1006a0 msgr2=0x7fcdac19be20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:17:27.060 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:27.054+0000 7fcdb1d7d700 1 --2- 172.21.15.40:0/2860924062 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fcdac1006a0 0x7fcdac19be20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:27.061 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:27.054+0000 7fcdb1d7d700 1 -- 172.21.15.40:0/2860924062 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fcdac0ffd10 msgr2=0x7fcdac194db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:17:27.061 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:27.054+0000 7fcdb1d7d700 1 --2- 172.21.15.40:0/2860924062 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fcdac0ffd10 0x7fcdac194db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:27.061 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:27.054+0000 7fcdb1d7d700 1 -- 172.21.15.40:0/2860924062 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcda00095f0 con 0x7fcdac0feb20 2023-07-14T19:17:27.061 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:27.054+0000 7fcdb257e700 1 --2- 172.21.15.40:0/2860924062 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fcdac1006a0 0x7fcdac19be20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-14T19:17:27.061 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:27.054+0000 7fcdb1d7d700 1 --2- 172.21.15.40:0/2860924062 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcdac0feb20 0x7fcdac194870 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7fcd9c00ada0 tx=0x7fcd9c00c330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:17:27.061 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:27.054+0000 7fcdaaffd700 1 -- 172.21.15.40:0/2860924062 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd9c00f070 con 0x7fcdac0feb20 2023-07-14T19:17:27.062 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:27.054+0000 7fcdaaffd700 1 -- 172.21.15.40:0/2860924062 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fcd9c00ae90 con 0x7fcdac0feb20 2023-07-14T19:17:27.062 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:27.054+0000 7fcdb3fe1700 1 -- 172.21.15.40:0/2860924062 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcdac19c9a0 con 0x7fcdac0feb20 2023-07-14T19:17:27.062 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:27.054+0000 7fcdaaffd700 1 -- 172.21.15.40:0/2860924062 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd9c004280 con 0x7fcdac0feb20 2023-07-14T19:17:27.062 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:27.054+0000 7fcdb3fe1700 1 -- 172.21.15.40:0/2860924062 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcdac19ceb0 con 0x7fcdac0feb20 2023-07-14T19:17:27.062 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:27.055+0000 7fcdb3fe1700 1 -- 172.21.15.40:0/2860924062 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcdac106060 con 0x7fcdac0feb20 2023-07-14T19:17:27.063 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:27.056+0000 7fcdaaffd700 1 -- 172.21.15.40:0/2860924062 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 18) v1 ==== 135715+0+0 (secure 0 0 0) 0x7fcd9c003a20 con 0x7fcdac0feb20 2023-07-14T19:17:27.063 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:27.057+0000 7fcdaaffd700 1 --2- 172.21.15.40:0/2860924062 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7fcd940a1150 0x7fcd940a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:17:27.064 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:27.057+0000 7fcdb157c700 1 --2- 172.21.15.40:0/2860924062 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7fcd940a1150 0x7fcd940a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:17:27.064 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:27.057+0000 7fcdaaffd700 1 -- 172.21.15.40:0/2860924062 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 8269+0+0 (secure 0 0 0) 0x7fcd9c0cbfe0 con 0x7fcdac0feb20 2023-07-14T19:17:27.064 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:27.057+0000 7fcdb157c700 1 --2- 172.21.15.40:0/2860924062 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7fcd940a1150 0x7fcd940a3600 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fcda000b5c0 tx=0x7fcda0005960 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:17:27.064 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:27.061+0000 7fcdaaffd700 1 -- 172.21.15.40:0/2860924062 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fcd9c095740 con 0x7fcdac0feb20 2023-07-14T19:17:27.261 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:27.260+0000 7fcdb3fe1700 1 -- 172.21.15.40:0/2860924062 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fcdac0612f0 con 0x7fcdac0feb20 2023-07-14T19:17:27.262 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:27.260+0000 7fcdaaffd700 1 -- 172.21.15.40:0/2860924062 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v83) v1 ==== 104+0+11 (secure 0 0 0) 0x7fcd9c098d60 con 0x7fcdac0feb20 2023-07-14T19:17:27.263 INFO:teuthology.orchestra.run.smithi040.stdout:pg_num: 32 2023-07-14T19:17:27.264 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:27.262+0000 7fcdb3fe1700 1 -- 172.21.15.40:0/2860924062 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7fcd940a1150 msgr2=0x7fcd940a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:17:27.264 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:27.262+0000 7fcdb3fe1700 1 --2- 172.21.15.40:0/2860924062 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7fcd940a1150 0x7fcd940a3600 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fcda000b5c0 tx=0x7fcda0005960 comp rx=0 tx=0).stop 2023-07-14T19:17:27.265 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:27.262+0000 7fcdb3fe1700 1 -- 172.21.15.40:0/2860924062 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcdac0feb20 msgr2=0x7fcdac194870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:17:27.265 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:27.262+0000 7fcdb3fe1700 1 --2- 172.21.15.40:0/2860924062 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcdac0feb20 0x7fcdac194870 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7fcd9c00ada0 tx=0x7fcd9c00c330 comp rx=0 tx=0).stop 2023-07-14T19:17:27.265 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:27.263+0000 7fcdb3fe1700 1 -- 172.21.15.40:0/2860924062 shutdown_connections 2023-07-14T19:17:27.265 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:27.263+0000 7fcdb3fe1700 1 --2- 172.21.15.40:0/2860924062 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7fcd940a1150 0x7fcd940a3600 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:27.265 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:27.263+0000 7fcdb3fe1700 1 --2- 172.21.15.40:0/2860924062 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fcdac1006a0 0x7fcdac19be20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:27.266 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:27.263+0000 7fcdb3fe1700 1 --2- 172.21.15.40:0/2860924062 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcdac0feb20 0x7fcdac194870 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:27.266 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:27.263+0000 7fcdb3fe1700 1 --2- 172.21.15.40:0/2860924062 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fcdac0ffd10 0x7fcdac194db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:27.266 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:27.263+0000 7fcdb3fe1700 1 -- 172.21.15.40:0/2860924062 >> 172.21.15.40:0/2860924062 conn(0x7fcdac0fa160 msgr2=0x7fcdac1031a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:17:27.266 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:27.263+0000 7fcdb3fe1700 1 -- 172.21.15.40:0/2860924062 shutdown_connections 2023-07-14T19:17:27.266 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:27.263+0000 7fcdb3fe1700 1 -- 172.21.15.40:0/2860924062 wait complete. 2023-07-14T19:17:27.276 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-07-14T19:17:27.405 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:27.403+0000 7fb480e32700 1 -- 172.21.15.40:0/827450123 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb47c0ffd00 msgr2=0x7fb47c100150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:17:27.406 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:27.403+0000 7fb480e32700 1 --2- 172.21.15.40:0/827450123 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb47c0ffd00 0x7fb47c100150 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7fb470009990 tx=0x7fb470009ca0 comp rx=0 tx=0).stop 2023-07-14T19:17:27.406 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:27.404+0000 7fb480e32700 1 -- 172.21.15.40:0/827450123 shutdown_connections 2023-07-14T19:17:27.406 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:27.404+0000 7fb480e32700 1 --2- 172.21.15.40:0/827450123 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb47c0feb10 0x7fb47c0fef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:27.407 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:27.404+0000 7fb480e32700 1 --2- 172.21.15.40:0/827450123 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb47c100690 0x7fb47c106f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:27.407 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:27.404+0000 7fb480e32700 1 --2- 172.21.15.40:0/827450123 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb47c0ffd00 0x7fb47c100150 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:27.407 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:27.404+0000 7fb480e32700 1 -- 172.21.15.40:0/827450123 >> 172.21.15.40:0/827450123 conn(0x7fb47c0fa130 msgr2=0x7fb47c0fc560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:17:27.407 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:27.404+0000 7fb480e32700 1 -- 172.21.15.40:0/827450123 shutdown_connections 2023-07-14T19:17:27.407 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:27.404+0000 7fb480e32700 1 -- 172.21.15.40:0/827450123 wait complete. 2023-07-14T19:17:27.410 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:27.405+0000 7fb480e32700 1 Processor -- start 2023-07-14T19:17:27.410 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:27.405+0000 7fb480e32700 1 -- start start 2023-07-14T19:17:27.410 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:27.405+0000 7fb480e32700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb47c0feb10 0x7fb47c193760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:17:27.410 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:27.405+0000 7fb480e32700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb47c0ffd00 0x7fb47c193ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:17:27.411 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:27.405+0000 7fb480e32700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb47c100690 0x7fb47c19ad10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:17:27.411 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:27.405+0000 7fb480e32700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb47c19b2e0 con 0x7fb47c0feb10 2023-07-14T19:17:27.411 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:27.405+0000 7fb480e32700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb47c19b420 con 0x7fb47c0ffd00 2023-07-14T19:17:27.411 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:27.405+0000 7fb480e32700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb47c19b5b0 con 0x7fb47c100690 2023-07-14T19:17:27.411 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:27.405+0000 7fb47a59c700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb47c0feb10 0x7fb47c193760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:17:27.412 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:27.405+0000 7fb47a59c700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb47c0feb10 0x7fb47c193760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:34470/0 (socket says 172.21.15.40:34470) 2023-07-14T19:17:27.412 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:27.405+0000 7fb47a59c700 1 -- 172.21.15.40:0/1074810457 learned_addr learned my addr 172.21.15.40:0/1074810457 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-07-14T19:17:27.412 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:27.405+0000 7fb47ad9d700 1 --2- 172.21.15.40:0/1074810457 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb47c100690 0x7fb47c19ad10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:17:27.412 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:27.405+0000 7fb479d9b700 1 --2- 172.21.15.40:0/1074810457 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb47c0ffd00 0x7fb47c193ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:17:27.413 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:27.405+0000 7fb47a59c700 1 -- 172.21.15.40:0/1074810457 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb47c100690 msgr2=0x7fb47c19ad10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:17:27.413 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:27.405+0000 7fb47a59c700 1 --2- 172.21.15.40:0/1074810457 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb47c100690 0x7fb47c19ad10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:27.413 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:27.405+0000 7fb47a59c700 1 -- 172.21.15.40:0/1074810457 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb47c0ffd00 msgr2=0x7fb47c193ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:17:27.413 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:27.405+0000 7fb47a59c700 1 --2- 172.21.15.40:0/1074810457 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb47c0ffd00 0x7fb47c193ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:27.414 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:27.405+0000 7fb47a59c700 1 -- 172.21.15.40:0/1074810457 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4700095f0 con 0x7fb47c0feb10 2023-07-14T19:17:27.414 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:27.406+0000 7fb47a59c700 1 --2- 172.21.15.40:0/1074810457 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb47c0feb10 0x7fb47c193760 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7fb46c00e950 tx=0x7fb46c00ec60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:17:27.414 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:27.406+0000 7fb46b7fe700 1 -- 172.21.15.40:0/1074810457 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb46c009770 con 0x7fb47c0feb10 2023-07-14T19:17:27.415 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:27.406+0000 7fb46b7fe700 1 -- 172.21.15.40:0/1074810457 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb46c00c7e0 con 0x7fb47c0feb10 2023-07-14T19:17:27.415 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:27.406+0000 7fb480e32700 1 -- 172.21.15.40:0/1074810457 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb47c19b890 con 0x7fb47c0feb10 2023-07-14T19:17:27.416 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:27.406+0000 7fb46b7fe700 1 -- 172.21.15.40:0/1074810457 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb46c003cf0 con 0x7fb47c0feb10 2023-07-14T19:17:27.416 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:27.406+0000 7fb480e32700 1 -- 172.21.15.40:0/1074810457 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb47c19bd50 con 0x7fb47c0feb10 2023-07-14T19:17:27.416 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:27.407+0000 7fb480e32700 1 -- 172.21.15.40:0/1074810457 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb47c1092b0 con 0x7fb47c0feb10 2023-07-14T19:17:27.417 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:27.408+0000 7fb46b7fe700 1 -- 172.21.15.40:0/1074810457 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 18) v1 ==== 135715+0+0 (secure 0 0 0) 0x7fb46c00c950 con 0x7fb47c0feb10 2023-07-14T19:17:27.417 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:27.408+0000 7fb46b7fe700 1 --2- 172.21.15.40:0/1074810457 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7fb4640a1150 0x7fb4640a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:17:27.419 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:27.409+0000 7fb479d9b700 1 --2- 172.21.15.40:0/1074810457 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7fb4640a1150 0x7fb4640a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:17:27.419 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:27.409+0000 7fb46b7fe700 1 -- 172.21.15.40:0/1074810457 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 8269+0+0 (secure 0 0 0) 0x7fb46c014070 con 0x7fb47c0feb10 2023-07-14T19:17:27.420 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:27.409+0000 7fb479d9b700 1 --2- 172.21.15.40:0/1074810457 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7fb4640a1150 0x7fb4640a3600 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fb470009550 tx=0x7fb470005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:17:27.420 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:27.413+0000 7fb46b7fe700 1 -- 172.21.15.40:0/1074810457 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fb46c099d20 con 0x7fb47c0feb10 2023-07-14T19:17:27.614 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:27.612+0000 7fb46b7fe700 1 -- 172.21.15.40:0/1074810457 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mgrmap(e 19) v1 ==== 135766+0+0 (secure 0 0 0) 0x7fb46c00cc30 con 0x7fb47c0feb10 2023-07-14T19:17:27.615 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:27.614+0000 7fb480e32700 1 -- 172.21.15.40:0/1074810457 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fb47c1059a0 con 0x7fb47c0feb10 2023-07-14T19:17:27.616 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:27.614+0000 7fb46b7fe700 1 -- 172.21.15.40:0/1074810457 <== mon.0 v2:172.21.15.40:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v83) v1 ==== 100+0+11 (secure 0 0 0) 0x7fb46c085680 con 0x7fb47c0feb10 2023-07-14T19:17:27.616 INFO:teuthology.orchestra.run.smithi040.stdout:pg_num: 64 2023-07-14T19:17:27.618 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:27.616+0000 7fb480e32700 1 -- 172.21.15.40:0/1074810457 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7fb4640a1150 msgr2=0x7fb4640a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:17:27.619 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:27.617+0000 7fb480e32700 1 --2- 172.21.15.40:0/1074810457 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7fb4640a1150 0x7fb4640a3600 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fb470009550 tx=0x7fb470005fd0 comp rx=0 tx=0).stop 2023-07-14T19:17:27.619 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:27.617+0000 7fb480e32700 1 -- 172.21.15.40:0/1074810457 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb47c0feb10 msgr2=0x7fb47c193760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:17:27.619 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:27.617+0000 7fb480e32700 1 --2- 172.21.15.40:0/1074810457 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb47c0feb10 0x7fb47c193760 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7fb46c00e950 tx=0x7fb46c00ec60 comp rx=0 tx=0).stop 2023-07-14T19:17:27.619 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:27.617+0000 7fb480e32700 1 -- 172.21.15.40:0/1074810457 shutdown_connections 2023-07-14T19:17:27.619 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:27.617+0000 7fb480e32700 1 --2- 172.21.15.40:0/1074810457 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7fb4640a1150 0x7fb4640a3600 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:27.620 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:27.617+0000 7fb480e32700 1 --2- 172.21.15.40:0/1074810457 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb47c100690 0x7fb47c19ad10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:27.620 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:27.617+0000 7fb480e32700 1 --2- 172.21.15.40:0/1074810457 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb47c0feb10 0x7fb47c193760 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:27.620 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:27.617+0000 7fb480e32700 1 --2- 172.21.15.40:0/1074810457 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb47c0ffd00 0x7fb47c193ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:27.620 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:27.617+0000 7fb480e32700 1 -- 172.21.15.40:0/1074810457 >> 172.21.15.40:0/1074810457 conn(0x7fb47c0fa130 msgr2=0x7fb47c0fc300 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:17:27.620 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:27.617+0000 7fb480e32700 1 -- 172.21.15.40:0/1074810457 shutdown_connections 2023-07-14T19:17:27.621 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:27.617+0000 7fb480e32700 1 -- 172.21.15.40:0/1074810457 wait complete. 2023-07-14T19:17:27.631 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-07-14T19:17:27.760 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:27.757+0000 7fbef2ea5700 1 -- 172.21.15.40:0/2502391438 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbeec103030 msgr2=0x7fbeec105410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:17:27.761 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:27.757+0000 7fbef2ea5700 1 --2- 172.21.15.40:0/2502391438 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbeec103030 0x7fbeec105410 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7fbee0009990 tx=0x7fbee0009ca0 comp rx=0 tx=0).stop 2023-07-14T19:17:27.761 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:27.757+0000 7fbef2ea5700 1 -- 172.21.15.40:0/2502391438 shutdown_connections 2023-07-14T19:17:27.762 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:27.757+0000 7fbef2ea5700 1 --2- 172.21.15.40:0/2502391438 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbeec105950 0x7fbeec107d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:27.763 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:27.757+0000 7fbef2ea5700 1 --2- 172.21.15.40:0/2502391438 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fbeec100710 0x7fbeec102af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:27.763 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:27.757+0000 7fbef2ea5700 1 --2- 172.21.15.40:0/2502391438 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbeec103030 0x7fbeec105410 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:27.763 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:27.757+0000 7fbef2ea5700 1 -- 172.21.15.40:0/2502391438 >> 172.21.15.40:0/2502391438 conn(0x7fbeec0fa130 msgr2=0x7fbeec0fc560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:17:27.764 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:27.757+0000 7fbef2ea5700 1 -- 172.21.15.40:0/2502391438 shutdown_connections 2023-07-14T19:17:27.765 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:27.757+0000 7fbef2ea5700 1 -- 172.21.15.40:0/2502391438 wait complete. 2023-07-14T19:17:27.765 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:27.757+0000 7fbef2ea5700 1 Processor -- start 2023-07-14T19:17:27.766 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:27.757+0000 7fbef2ea5700 1 -- start start 2023-07-14T19:17:27.766 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:27.758+0000 7fbef2ea5700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbeec100710 0x7fbeec195920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:17:27.766 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:27.758+0000 7fbef2ea5700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbeec103030 0x7fbeec195e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:17:27.767 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:27.758+0000 7fbef2ea5700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fbeec105950 0x7fbeec19ced0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:17:27.767 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:27.758+0000 7fbef2ea5700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbeec19d4a0 con 0x7fbeec100710 2023-07-14T19:17:27.767 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:27.758+0000 7fbef2ea5700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbeec19d5e0 con 0x7fbeec105950 2023-07-14T19:17:27.767 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:27.758+0000 7fbef2ea5700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbeec19d770 con 0x7fbeec103030 2023-07-14T19:17:27.768 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:27.758+0000 7fbef0c41700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbeec100710 0x7fbeec195920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:17:27.768 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:27.758+0000 7fbef0c41700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbeec100710 0x7fbeec195920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:34484/0 (socket says 172.21.15.40:34484) 2023-07-14T19:17:27.768 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:27.758+0000 7fbef0c41700 1 -- 172.21.15.40:0/3474102149 learned_addr learned my addr 172.21.15.40:0/3474102149 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-07-14T19:17:27.768 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:27.758+0000 7fbef1442700 1 --2- 172.21.15.40:0/3474102149 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fbeec105950 0x7fbeec19ced0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:17:27.769 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:27.758+0000 7fbeebfff700 1 --2- 172.21.15.40:0/3474102149 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbeec103030 0x7fbeec195e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:17:27.769 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:27.758+0000 7fbef0c41700 1 -- 172.21.15.40:0/3474102149 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbeec103030 msgr2=0x7fbeec195e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:17:27.769 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:27.758+0000 7fbef0c41700 1 --2- 172.21.15.40:0/3474102149 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbeec103030 0x7fbeec195e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:27.769 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:27.758+0000 7fbef0c41700 1 -- 172.21.15.40:0/3474102149 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fbeec105950 msgr2=0x7fbeec19ced0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:17:27.770 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:27.758+0000 7fbef0c41700 1 --2- 172.21.15.40:0/3474102149 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fbeec105950 0x7fbeec19ced0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:27.770 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:27.758+0000 7fbef0c41700 1 -- 172.21.15.40:0/3474102149 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbedc009c40 con 0x7fbeec100710 2023-07-14T19:17:27.770 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:27.758+0000 7fbef0c41700 1 --2- 172.21.15.40:0/3474102149 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbeec100710 0x7fbeec195920 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7fbedc00c370 tx=0x7fbedc00c680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:17:27.770 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:27.758+0000 7fbee9ffb700 1 -- 172.21.15.40:0/3474102149 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbedc017070 con 0x7fbeec100710 2023-07-14T19:17:27.770 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:27.758+0000 7fbee9ffb700 1 -- 172.21.15.40:0/3474102149 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fbedc00f040 con 0x7fbeec100710 2023-07-14T19:17:27.771 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:27.758+0000 7fbef2ea5700 1 -- 172.21.15.40:0/3474102149 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbee00095f0 con 0x7fbeec100710 2023-07-14T19:17:27.771 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:27.758+0000 7fbee9ffb700 1 -- 172.21.15.40:0/3474102149 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbedc00e050 con 0x7fbeec100710 2023-07-14T19:17:27.771 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:27.759+0000 7fbef2ea5700 1 -- 172.21.15.40:0/3474102149 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbeec19dd70 con 0x7fbeec100710 2023-07-14T19:17:27.772 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:27.760+0000 7fbef2ea5700 1 -- 172.21.15.40:0/3474102149 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbeec10b3c0 con 0x7fbeec100710 2023-07-14T19:17:27.772 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:27.760+0000 7fbee9ffb700 1 -- 172.21.15.40:0/3474102149 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 19) v1 ==== 135766+0+0 (secure 0 0 0) 0x7fbedc0044e0 con 0x7fbeec100710 2023-07-14T19:17:27.773 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:27.761+0000 7fbee9ffb700 1 --2- 172.21.15.40:0/3474102149 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7fbed40a11e0 0x7fbed40a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:17:27.773 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:27.761+0000 7fbeebfff700 1 --2- 172.21.15.40:0/3474102149 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7fbed40a11e0 0x7fbed40a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:17:27.773 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:27.761+0000 7fbee9ffb700 1 -- 172.21.15.40:0/3474102149 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 8269+0+0 (secure 0 0 0) 0x7fbedc0cc4c0 con 0x7fbeec100710 2023-07-14T19:17:27.773 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:27.762+0000 7fbeebfff700 1 --2- 172.21.15.40:0/3474102149 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7fbed40a11e0 0x7fbed40a3690 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fbee0006010 tx=0x7fbee000b260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:17:27.774 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:27.766+0000 7fbee9ffb700 1 -- 172.21.15.40:0/3474102149 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fbedc095a90 con 0x7fbeec100710 2023-07-14T19:17:27.970 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:17:27 smithi040 ceph-mon[96518]: from='client.15300 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup pin", "vol_name": "cephfs", "group_name": "_nogroup", "pin_type": "random", "pin_setting": "0.01", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T19:17:27.971 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:17:27 smithi040 ceph-mon[96518]: from='client.? 172.21.15.40:0/1381476529' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-07-14T19:17:27.971 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:17:27 smithi040 ceph-mon[96518]: from='client.? 172.21.15.40:0/2860924062' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-07-14T19:17:27.972 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:17:27 smithi191 ceph-mon[98311]: from='client.15300 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup pin", "vol_name": "cephfs", "group_name": "_nogroup", "pin_type": "random", "pin_setting": "0.01", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T19:17:27.972 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:17:27 smithi191 ceph-mon[98311]: from='client.? 172.21.15.40:0/1381476529' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-07-14T19:17:27.972 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:17:27 smithi191 ceph-mon[98311]: from='client.? 172.21.15.40:0/2860924062' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-07-14T19:17:27.984 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:27.982+0000 7fbef2ea5700 1 -- 172.21.15.40:0/3474102149 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fbeec19e3b0 con 0x7fbeec100710 2023-07-14T19:17:27.984 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:27.982+0000 7fbee9ffb700 1 -- 172.21.15.40:0/3474102149 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 10 entries v83) v1 ==== 81+0+592 (secure 0 0 0) 0x7fbedc0990b0 con 0x7fbeec100710 2023-07-14T19:17:27.985 INFO:teuthology.orchestra.run.smithi040.stdout:172.21.15.40:6801/3698757264 2023-07-15T19:09:50.732458+0000 2023-07-14T19:17:27.985 INFO:teuthology.orchestra.run.smithi040.stdout:172.21.15.40:6800/3698757264 2023-07-15T19:09:50.732458+0000 2023-07-14T19:17:27.985 INFO:teuthology.orchestra.run.smithi040.stdout:172.21.15.40:6800/2569015667 2023-07-15T19:09:24.441543+0000 2023-07-14T19:17:27.986 INFO:teuthology.orchestra.run.smithi040.stdout:172.21.15.40:0/3653961003 2023-07-15T19:09:24.441543+0000 2023-07-14T19:17:27.986 INFO:teuthology.orchestra.run.smithi040.stdout:172.21.15.40:0/3568861680 2023-07-15T19:09:24.441543+0000 2023-07-14T19:17:27.986 INFO:teuthology.orchestra.run.smithi040.stdout:172.21.15.40:0/3249773172 2023-07-15T19:09:50.732458+0000 2023-07-14T19:17:27.986 INFO:teuthology.orchestra.run.smithi040.stdout:172.21.15.40:0/4084215480 2023-07-15T19:09:24.441543+0000 2023-07-14T19:17:27.987 INFO:teuthology.orchestra.run.smithi040.stdout:172.21.15.40:0/1163941760 2023-07-15T19:09:50.732458+0000 2023-07-14T19:17:27.987 INFO:teuthology.orchestra.run.smithi040.stdout:172.21.15.40:6801/2569015667 2023-07-15T19:09:24.441543+0000 2023-07-14T19:17:27.987 INFO:teuthology.orchestra.run.smithi040.stdout:172.21.15.40:0/2326639960 2023-07-15T19:09:50.732458+0000 2023-07-14T19:17:27.988 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:27.984+0000 7fbef2ea5700 1 -- 172.21.15.40:0/3474102149 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7fbed40a11e0 msgr2=0x7fbed40a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:17:27.988 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:27.984+0000 7fbef2ea5700 1 --2- 172.21.15.40:0/3474102149 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7fbed40a11e0 0x7fbed40a3690 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fbee0006010 tx=0x7fbee000b260 comp rx=0 tx=0).stop 2023-07-14T19:17:27.988 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:27.984+0000 7fbef2ea5700 1 -- 172.21.15.40:0/3474102149 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbeec100710 msgr2=0x7fbeec195920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:17:27.989 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:27.984+0000 7fbef2ea5700 1 --2- 172.21.15.40:0/3474102149 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbeec100710 0x7fbeec195920 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7fbedc00c370 tx=0x7fbedc00c680 comp rx=0 tx=0).stop 2023-07-14T19:17:27.989 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:27.984+0000 7fbef2ea5700 1 -- 172.21.15.40:0/3474102149 shutdown_connections 2023-07-14T19:17:27.989 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:27.984+0000 7fbef2ea5700 1 --2- 172.21.15.40:0/3474102149 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7fbed40a11e0 0x7fbed40a3690 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:27.990 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:27.984+0000 7fbef2ea5700 1 --2- 172.21.15.40:0/3474102149 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fbeec105950 0x7fbeec19ced0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:27.990 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:27.984+0000 7fbef2ea5700 1 --2- 172.21.15.40:0/3474102149 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbeec100710 0x7fbeec195920 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:27.990 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:27.984+0000 7fbef2ea5700 1 --2- 172.21.15.40:0/3474102149 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fbeec103030 0x7fbeec195e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:27.990 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:27.984+0000 7fbef2ea5700 1 -- 172.21.15.40:0/3474102149 >> 172.21.15.40:0/3474102149 conn(0x7fbeec0fa130 msgr2=0x7fbeec101b70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:17:27.991 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:27.985+0000 7fbef2ea5700 1 -- 172.21.15.40:0/3474102149 shutdown_connections 2023-07-14T19:17:27.991 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:27.985+0000 7fbef2ea5700 1 -- 172.21.15.40:0/3474102149 wait complete. 2023-07-14T19:17:27.991 INFO:teuthology.orchestra.run.smithi040.stderr:listed 10 entries 2023-07-14T19:17:27.992 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:17:27 smithi130 ceph-mon[97832]: from='client.15300 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup pin", "vol_name": "cephfs", "group_name": "_nogroup", "pin_type": "random", "pin_setting": "0.01", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T19:17:27.992 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:17:27 smithi130 ceph-mon[97832]: from='client.? 172.21.15.40:0/1381476529' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-07-14T19:17:27.992 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:17:27 smithi130 ceph-mon[97832]: from='client.? 172.21.15.40:0/2860924062' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-07-14T19:17:27.998 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2023-07-14T19:17:27.999 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-14T19:17:28.000 DEBUG:teuthology.orchestra.run.smithi040:> sudo umount /home/ubuntu/cephtest/mnt.0 2023-07-14T19:17:28.049 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi040.front.sepia.ceph.com 2023-07-14T19:17:28.050 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-14T19:17:28.050 DEBUG:teuthology.orchestra.run.smithi040:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2023-07-14T19:17:28.072 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-14T19:17:28.073 DEBUG:teuthology.orchestra.run.smithi040:> ip netns list 2023-07-14T19:17:28.129 INFO:teuthology.orchestra.run.smithi040.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2023-07-14T19:17:28.130 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-14T19:17:28.130 DEBUG:teuthology.orchestra.run.smithi040:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.0 2023-07-14T19:17:28.207 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-14T19:17:28.207 DEBUG:teuthology.orchestra.run.smithi040:> sudo ip link delete ceph-brx 2023-07-14T19:17:28.308 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2023-07-14T19:17:28.326 INFO:tasks.cephadm:Teardown begin 2023-07-14T19:17:28.327 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2023-07-14T19:17:28.327 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-07-14T19:17:28.454 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:28.452+0000 7ff4326b0700 1 -- 172.21.15.40:0/2756195808 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff42c0ffcf0 msgr2=0x7ff42c100140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:17:28.455 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:28.452+0000 7ff4326b0700 1 --2- 172.21.15.40:0/2756195808 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff42c0ffcf0 0x7ff42c100140 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7ff420009990 tx=0x7ff420009ca0 comp rx=0 tx=0).stop 2023-07-14T19:17:28.455 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:28.452+0000 7ff4326b0700 1 -- 172.21.15.40:0/2756195808 shutdown_connections 2023-07-14T19:17:28.455 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:28.452+0000 7ff4326b0700 1 --2- 172.21.15.40:0/2756195808 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff42c0feb00 0x7ff42c0fef10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:28.455 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:28.452+0000 7ff4326b0700 1 --2- 172.21.15.40:0/2756195808 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff42c100680 0x7ff42c106f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:28.455 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:28.452+0000 7ff4326b0700 1 --2- 172.21.15.40:0/2756195808 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff42c0ffcf0 0x7ff42c100140 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:28.456 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:28.452+0000 7ff4326b0700 1 -- 172.21.15.40:0/2756195808 >> 172.21.15.40:0/2756195808 conn(0x7ff42c0fa100 msgr2=0x7ff42c0fc550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:17:28.456 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:28.452+0000 7ff4326b0700 1 -- 172.21.15.40:0/2756195808 shutdown_connections 2023-07-14T19:17:28.456 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:28.452+0000 7ff4326b0700 1 -- 172.21.15.40:0/2756195808 wait complete. 2023-07-14T19:17:28.456 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:28.453+0000 7ff4326b0700 1 Processor -- start 2023-07-14T19:17:28.457 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:28.453+0000 7ff4326b0700 1 -- start start 2023-07-14T19:17:28.458 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:28.453+0000 7ff4326b0700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff42c0feb00 0x7ff42c068330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:17:28.459 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:28.453+0000 7ff4326b0700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff42c0ffcf0 0x7ff42c19a570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:17:28.459 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:28.453+0000 7ff4326b0700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff42c100680 0x7ff42c19cbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:17:28.459 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:28.453+0000 7ff4326b0700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff42c19d390 con 0x7ff42c100680 2023-07-14T19:17:28.459 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:28.453+0000 7ff4326b0700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff42c19d500 con 0x7ff42c0ffcf0 2023-07-14T19:17:28.459 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:28.453+0000 7ff4326b0700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff42c19d6c0 con 0x7ff42c0feb00 2023-07-14T19:17:28.460 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:28.453+0000 7ff430c4d700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff42c100680 0x7ff42c19cbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:17:28.460 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:28.453+0000 7ff430c4d700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff42c100680 0x7ff42c19cbc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:34500/0 (socket says 172.21.15.40:34500) 2023-07-14T19:17:28.460 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:28.453+0000 7ff430c4d700 1 -- 172.21.15.40:0/2292415615 learned_addr learned my addr 172.21.15.40:0/2292415615 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-07-14T19:17:28.460 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:28.454+0000 7ff42b7fe700 1 --2- 172.21.15.40:0/2292415615 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff42c0ffcf0 0x7ff42c19a570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:17:28.461 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:28.454+0000 7ff42bfff700 1 --2- 172.21.15.40:0/2292415615 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff42c0feb00 0x7ff42c068330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:17:28.461 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:28.454+0000 7ff430c4d700 1 -- 172.21.15.40:0/2292415615 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff42c0feb00 msgr2=0x7ff42c068330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:17:28.461 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:28.454+0000 7ff430c4d700 1 --2- 172.21.15.40:0/2292415615 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff42c0feb00 0x7ff42c068330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:28.461 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:28.454+0000 7ff430c4d700 1 -- 172.21.15.40:0/2292415615 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff42c0ffcf0 msgr2=0x7ff42c19a570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:17:28.461 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:28.454+0000 7ff430c4d700 1 --2- 172.21.15.40:0/2292415615 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff42c0ffcf0 0x7ff42c19a570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:28.462 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:28.454+0000 7ff430c4d700 1 -- 172.21.15.40:0/2292415615 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4200095f0 con 0x7ff42c100680 2023-07-14T19:17:28.462 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:28.454+0000 7ff42bfff700 1 --2- 172.21.15.40:0/2292415615 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff42c0feb00 0x7ff42c068330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-14T19:17:28.462 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:28.454+0000 7ff430c4d700 1 --2- 172.21.15.40:0/2292415615 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff42c100680 0x7ff42c19cbc0 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7ff424007ee0 tx=0x7ff42400d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:17:28.463 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:28.454+0000 7ff4297fa700 1 -- 172.21.15.40:0/2292415615 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff424017070 con 0x7ff42c100680 2023-07-14T19:17:28.463 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:28.454+0000 7ff4326b0700 1 -- 172.21.15.40:0/2292415615 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff42c19d9a0 con 0x7ff42c100680 2023-07-14T19:17:28.463 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:28.454+0000 7ff4297fa700 1 -- 172.21.15.40:0/2292415615 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff42400f040 con 0x7ff42c100680 2023-07-14T19:17:28.463 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:28.454+0000 7ff4297fa700 1 -- 172.21.15.40:0/2292415615 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff424005740 con 0x7ff42c100680 2023-07-14T19:17:28.463 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:28.454+0000 7ff4326b0700 1 -- 172.21.15.40:0/2292415615 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff42c19deb0 con 0x7ff42c100680 2023-07-14T19:17:28.464 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:28.455+0000 7ff4326b0700 1 -- 172.21.15.40:0/2292415615 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff42c10a2d0 con 0x7ff42c100680 2023-07-14T19:17:28.464 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:28.456+0000 7ff4297fa700 1 -- 172.21.15.40:0/2292415615 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 19) v1 ==== 135766+0+0 (secure 0 0 0) 0x7ff42400dc60 con 0x7ff42c100680 2023-07-14T19:17:28.464 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:28.457+0000 7ff4297fa700 1 --2- 172.21.15.40:0/2292415615 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7ff4140a1220 0x7ff4140a36d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:17:28.464 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:28.457+0000 7ff42bfff700 1 --2- 172.21.15.40:0/2292415615 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7ff4140a1220 0x7ff4140a36d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:17:28.464 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:28.457+0000 7ff4297fa700 1 -- 172.21.15.40:0/2292415615 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 8269+0+0 (secure 0 0 0) 0x7ff4240cc270 con 0x7ff42c100680 2023-07-14T19:17:28.465 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:28.457+0000 7ff42bfff700 1 --2- 172.21.15.40:0/2292415615 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7ff4140a1220 0x7ff4140a36d0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7ff41c005fd0 tx=0x7ff41c005e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:17:28.465 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:28.461+0000 7ff4297fa700 1 -- 172.21.15.40:0/2292415615 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7ff424095840 con 0x7ff42c100680 2023-07-14T19:17:28.711 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:28.709+0000 7ff4326b0700 1 -- 172.21.15.40:0/2292415615 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff42c105910 con 0x7ff42c100680 2023-07-14T19:17:28.712 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:28.710+0000 7ff4297fa700 1 -- 172.21.15.40:0/2292415615 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) v1 ==== 94+0+10877 (secure 0 0 0) 0x7ff424098e60 con 0x7ff42c100680 2023-07-14T19:17:28.712 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-07-14T19:17:28.713 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14523,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6835/501143718","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6834","nonce":501143718},{"type":"v1","addr":"172.21.15.40:6835","nonce":501143718}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14535,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6837/2634476117","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6836","nonce":2634476117},{"type":"v1","addr":"172.21.15.40:6837","nonce":2634476117}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":27,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2023-07-14T19:15:57.650710+0000","modified":"2023-07-14T19:17:25.590847+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24539,"mds_1":24508,"mds_2":24506,"mds_3":24490,"mds_4":24482},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14586":{"gid":14586,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.40:6839/959937545","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6838","nonce":959937545},{"type":"v1","addr":"172.21.15.40:6839","nonce":959937545}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_14598":{"gid":14598,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.40:6841/1590486077","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6840","nonce":1590486077},{"type":"v1","addr":"172.21.15.40:6841","nonce":1590486077}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24433":{"gid":24433,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.191:6833/31509862","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":31509862},{"type":"v1","addr":"172.21.15.191:6833","nonce":31509862}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24452":{"gid":24452,"name":"k","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.130:6833/3677452688","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6832","nonce":3677452688},{"type":"v1","addr":"172.21.15.130:6833","nonce":3677452688}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24478":{"gid":24478,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.191:6835/3712674634","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":3712674634},{"type":"v1","addr":"172.21.15.191:6835","nonce":3712674634}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24482":{"gid":24482,"name":"e","rank":4,"incarnation":24,"state":"up:active","state_seq":8,"addr":"172.21.15.130:6835/3088490808","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6834","nonce":3088490808},{"type":"v1","addr":"172.21.15.130:6835","nonce":3088490808}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24490":{"gid":24490,"name":"c","rank":3,"incarnation":22,"state":"up:active","state_seq":7,"addr":"172.21.15.191:6837/2406892975","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6836","nonce":2406892975},{"type":"v1","addr":"172.21.15.191:6837","nonce":2406892975}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24506":{"gid":24506,"name":"h","rank":2,"incarnation":20,"state":"up:active","state_seq":6,"addr":"172.21.15.130:6837/2604019369","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6836","nonce":2604019369},{"type":"v1","addr":"172.21.15.130:6837","nonce":2604019369}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24508":{"gid":24508,"name":"i","rank":1,"incarnation":18,"state":"up:active","state_seq":5,"addr":"172.21.15.191:6839/1977507302","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":1977507302},{"type":"v1","addr":"172.21.15.191:6839","nonce":1977507302}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24539":{"gid":24539,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.130:6839/2145479535","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6838","nonce":2145479535},{"type":"v1","addr":"172.21.15.130:6839","nonce":2145479535}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-07-14T19:17:28.714 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:28.712+0000 7ff4326b0700 1 -- 172.21.15.40:0/2292415615 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7ff4140a1220 msgr2=0x7ff4140a36d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:17:28.714 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:28.712+0000 7ff4326b0700 1 --2- 172.21.15.40:0/2292415615 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7ff4140a1220 0x7ff4140a36d0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7ff41c005fd0 tx=0x7ff41c005e40 comp rx=0 tx=0).stop 2023-07-14T19:17:28.714 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:28.712+0000 7ff4326b0700 1 -- 172.21.15.40:0/2292415615 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff42c100680 msgr2=0x7ff42c19cbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:17:28.714 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:28.712+0000 7ff4326b0700 1 --2- 172.21.15.40:0/2292415615 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff42c100680 0x7ff42c19cbc0 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7ff424007ee0 tx=0x7ff42400d3b0 comp rx=0 tx=0).stop 2023-07-14T19:17:28.715 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:28.712+0000 7ff4326b0700 1 -- 172.21.15.40:0/2292415615 shutdown_connections 2023-07-14T19:17:28.715 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:28.712+0000 7ff4326b0700 1 --2- 172.21.15.40:0/2292415615 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7ff4140a1220 0x7ff4140a36d0 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:28.715 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:28.712+0000 7ff4326b0700 1 --2- 172.21.15.40:0/2292415615 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff42c0feb00 0x7ff42c068330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:28.715 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:28.712+0000 7ff4326b0700 1 --2- 172.21.15.40:0/2292415615 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff42c100680 0x7ff42c19cbc0 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:28.716 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:28.712+0000 7ff4326b0700 1 --2- 172.21.15.40:0/2292415615 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff42c0ffcf0 0x7ff42c19a570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:28.716 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:28.712+0000 7ff4326b0700 1 -- 172.21.15.40:0/2292415615 >> 172.21.15.40:0/2292415615 conn(0x7ff42c0fa100 msgr2=0x7ff42c0fc2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:17:28.716 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:28.712+0000 7ff4326b0700 1 -- 172.21.15.40:0/2292415615 shutdown_connections 2023-07-14T19:17:28.716 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:28.712+0000 7ff4326b0700 1 -- 172.21.15.40:0/2292415615 wait complete. 2023-07-14T19:17:28.717 INFO:teuthology.orchestra.run.smithi040.stderr:dumped fsmap epoch 27 2023-07-14T19:17:28.726 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-07-14T19:17:28.853 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:28.851+0000 7f7b70f4c700 1 -- 172.21.15.40:0/2699772101 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7b6c100680 msgr2=0x7f7b6c106f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:17:28.854 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:28.851+0000 7f7b70f4c700 1 --2- 172.21.15.40:0/2699772101 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7b6c100680 0x7f7b6c106f40 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f7b6400b0a0 tx=0x7f7b6400b3b0 comp rx=0 tx=0).stop 2023-07-14T19:17:28.854 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:28.851+0000 7f7b70f4c700 1 -- 172.21.15.40:0/2699772101 shutdown_connections 2023-07-14T19:17:28.855 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:28.851+0000 7f7b70f4c700 1 --2- 172.21.15.40:0/2699772101 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7b6c0feb00 0x7f7b6c0fef10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:28.855 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:28.851+0000 7f7b70f4c700 1 --2- 172.21.15.40:0/2699772101 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7b6c100680 0x7f7b6c106f40 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:28.855 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:28.851+0000 7f7b70f4c700 1 --2- 172.21.15.40:0/2699772101 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7b6c0ffcf0 0x7f7b6c100140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:28.856 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:28.851+0000 7f7b70f4c700 1 -- 172.21.15.40:0/2699772101 >> 172.21.15.40:0/2699772101 conn(0x7f7b6c0fa100 msgr2=0x7f7b6c0fc550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:17:28.856 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:28.851+0000 7f7b70f4c700 1 -- 172.21.15.40:0/2699772101 shutdown_connections 2023-07-14T19:17:28.857 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:28.851+0000 7f7b70f4c700 1 -- 172.21.15.40:0/2699772101 wait complete. 2023-07-14T19:17:28.860 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:28.852+0000 7f7b70f4c700 1 Processor -- start 2023-07-14T19:17:28.860 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:28.853+0000 7f7b70f4c700 1 -- start start 2023-07-14T19:17:28.860 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:28.853+0000 7f7b70f4c700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7b6c0feb00 0x7f7b6c102a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:17:28.861 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:28.853+0000 7f7b70f4c700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7b6c0ffcf0 0x7f7b6c1010d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:17:28.861 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:28.853+0000 7f7b70f4c700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7b6c100680 0x7f7b6c101610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:17:28.861 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:28.853+0000 7f7b70f4c700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b6c101db0 con 0x7f7b6c0feb00 2023-07-14T19:17:28.861 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:28.853+0000 7f7b70f4c700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b6c101f20 con 0x7f7b6c100680 2023-07-14T19:17:28.861 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:28.853+0000 7f7b70f4c700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b6c19fa50 con 0x7f7b6c0ffcf0 2023-07-14T19:17:28.862 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:28.853+0000 7f7b6a59c700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7b6c0feb00 0x7f7b6c102a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:17:28.862 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:28.853+0000 7f7b6a59c700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7b6c0feb00 0x7f7b6c102a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:34510/0 (socket says 172.21.15.40:34510) 2023-07-14T19:17:28.862 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:28.853+0000 7f7b6a59c700 1 -- 172.21.15.40:0/1445715125 learned_addr learned my addr 172.21.15.40:0/1445715125 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-07-14T19:17:28.862 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:28.853+0000 7f7b69d9b700 1 --2- 172.21.15.40:0/1445715125 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7b6c0ffcf0 0x7f7b6c1010d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:17:28.862 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:28.853+0000 7f7b6ad9d700 1 --2- 172.21.15.40:0/1445715125 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7b6c100680 0x7f7b6c101610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:17:28.863 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:28.853+0000 7f7b6a59c700 1 -- 172.21.15.40:0/1445715125 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7b6c0ffcf0 msgr2=0x7f7b6c1010d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:17:28.863 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:28.853+0000 7f7b6a59c700 1 --2- 172.21.15.40:0/1445715125 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7b6c0ffcf0 0x7f7b6c1010d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:28.863 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:28.853+0000 7f7b6a59c700 1 -- 172.21.15.40:0/1445715125 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7b6c100680 msgr2=0x7f7b6c101610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:17:28.863 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:28.853+0000 7f7b6a59c700 1 --2- 172.21.15.40:0/1445715125 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7b6c100680 0x7f7b6c101610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:28.864 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:28.853+0000 7f7b6a59c700 1 -- 172.21.15.40:0/1445715125 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b64009d00 con 0x7f7b6c0feb00 2023-07-14T19:17:28.864 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:28.853+0000 7f7b69d9b700 1 --2- 172.21.15.40:0/1445715125 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7b6c0ffcf0 0x7f7b6c1010d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-14T19:17:28.864 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:28.853+0000 7f7b6a59c700 1 --2- 172.21.15.40:0/1445715125 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7b6c0feb00 0x7f7b6c102a90 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f7b5c00e970 tx=0x7f7b5c00ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:17:28.864 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:28.853+0000 7f7b637fe700 1 -- 172.21.15.40:0/1445715125 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b5c009790 con 0x7f7b6c0feb00 2023-07-14T19:17:28.864 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:28.854+0000 7f7b637fe700 1 -- 172.21.15.40:0/1445715125 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f7b5c0044e0 con 0x7f7b6c0feb00 2023-07-14T19:17:28.865 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:28.854+0000 7f7b637fe700 1 -- 172.21.15.40:0/1445715125 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b5c003e30 con 0x7f7b6c0feb00 2023-07-14T19:17:28.865 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:28.854+0000 7f7b70f4c700 1 -- 172.21.15.40:0/1445715125 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7b6c19fd30 con 0x7f7b6c0feb00 2023-07-14T19:17:28.865 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:28.854+0000 7f7b70f4c700 1 -- 172.21.15.40:0/1445715125 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b6c1a0190 con 0x7f7b6c0feb00 2023-07-14T19:17:28.865 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:28.855+0000 7f7b70f4c700 1 -- 172.21.15.40:0/1445715125 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7b6c10a2e0 con 0x7f7b6c0feb00 2023-07-14T19:17:28.865 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:28.856+0000 7f7b637fe700 1 -- 172.21.15.40:0/1445715125 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 19) v1 ==== 135766+0+0 (secure 0 0 0) 0x7f7b5c003f90 con 0x7f7b6c0feb00 2023-07-14T19:17:28.866 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:28.856+0000 7f7b637fe700 1 --2- 172.21.15.40:0/1445715125 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f7b580a12a0 0x7f7b580a3750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:17:28.866 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:28.856+0000 7f7b69d9b700 1 --2- 172.21.15.40:0/1445715125 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f7b580a12a0 0x7f7b580a3750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:17:28.866 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:28.856+0000 7f7b637fe700 1 -- 172.21.15.40:0/1445715125 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 8269+0+0 (secure 0 0 0) 0x7f7b5c014070 con 0x7f7b6c0feb00 2023-07-14T19:17:28.866 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:28.857+0000 7f7b69d9b700 1 --2- 172.21.15.40:0/1445715125 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f7b580a12a0 0x7f7b580a3750 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f7b54005d90 tx=0x7f7b54005d00 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:17:28.867 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:28.860+0000 7f7b637fe700 1 -- 172.21.15.40:0/1445715125 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f7b5c096310 con 0x7f7b6c0feb00 2023-07-14T19:17:28.971 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:17:28 smithi040 ceph-mon[96518]: pgmap v301: 97 pgs: 97 active+clean; 639 KiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 511 B/s wr, 11 op/s 2023-07-14T19:17:28.972 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:17:28 smithi040 ceph-mon[96518]: mgrmap e19: x(active, since 7m), standbys: z, y 2023-07-14T19:17:28.972 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:17:28 smithi040 ceph-mon[96518]: from='client.? 172.21.15.40:0/1074810457' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-07-14T19:17:28.972 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:17:28 smithi040 ceph-mon[96518]: from='client.? 172.21.15.40:0/3474102149' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2023-07-14T19:17:28.974 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:17:28 smithi191 ceph-mon[98311]: pgmap v301: 97 pgs: 97 active+clean; 639 KiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 511 B/s wr, 11 op/s 2023-07-14T19:17:28.974 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:17:28 smithi191 ceph-mon[98311]: mgrmap e19: x(active, since 7m), standbys: z, y 2023-07-14T19:17:28.975 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:17:28 smithi191 ceph-mon[98311]: from='client.? 172.21.15.40:0/1074810457' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-07-14T19:17:28.976 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:17:28 smithi191 ceph-mon[98311]: from='client.? 172.21.15.40:0/3474102149' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2023-07-14T19:17:28.992 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:17:28 smithi130 ceph-mon[97832]: pgmap v301: 97 pgs: 97 active+clean; 639 KiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 511 B/s wr, 11 op/s 2023-07-14T19:17:28.993 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:17:28 smithi130 ceph-mon[97832]: mgrmap e19: x(active, since 7m), standbys: z, y 2023-07-14T19:17:28.993 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:17:28 smithi130 ceph-mon[97832]: from='client.? 172.21.15.40:0/1074810457' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-07-14T19:17:28.993 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:17:28 smithi130 ceph-mon[97832]: from='client.? 172.21.15.40:0/3474102149' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2023-07-14T19:17:29.106 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:29.105+0000 7f7b70f4c700 1 -- 172.21.15.40:0/1445715125 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7b6c1a0830 con 0x7f7b6c0feb00 2023-07-14T19:17:29.107 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:29.106+0000 7f7b637fe700 1 -- 172.21.15.40:0/1445715125 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) v1 ==== 94+0+10877 (secure 0 0 0) 0x7f7b5c099930 con 0x7f7b6c0feb00 2023-07-14T19:17:29.108 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-07-14T19:17:29.108 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14523,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6835/501143718","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6834","nonce":501143718},{"type":"v1","addr":"172.21.15.40:6835","nonce":501143718}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14535,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6837/2634476117","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6836","nonce":2634476117},{"type":"v1","addr":"172.21.15.40:6837","nonce":2634476117}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":27,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2023-07-14T19:15:57.650710+0000","modified":"2023-07-14T19:17:25.590847+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24539,"mds_1":24508,"mds_2":24506,"mds_3":24490,"mds_4":24482},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14586":{"gid":14586,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.40:6839/959937545","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6838","nonce":959937545},{"type":"v1","addr":"172.21.15.40:6839","nonce":959937545}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_14598":{"gid":14598,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.40:6841/1590486077","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6840","nonce":1590486077},{"type":"v1","addr":"172.21.15.40:6841","nonce":1590486077}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24433":{"gid":24433,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.191:6833/31509862","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6832","nonce":31509862},{"type":"v1","addr":"172.21.15.191:6833","nonce":31509862}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24452":{"gid":24452,"name":"k","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.130:6833/3677452688","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6832","nonce":3677452688},{"type":"v1","addr":"172.21.15.130:6833","nonce":3677452688}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24478":{"gid":24478,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.191:6835/3712674634","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":3712674634},{"type":"v1","addr":"172.21.15.191:6835","nonce":3712674634}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24482":{"gid":24482,"name":"e","rank":4,"incarnation":24,"state":"up:active","state_seq":8,"addr":"172.21.15.130:6835/3088490808","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6834","nonce":3088490808},{"type":"v1","addr":"172.21.15.130:6835","nonce":3088490808}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24490":{"gid":24490,"name":"c","rank":3,"incarnation":22,"state":"up:active","state_seq":7,"addr":"172.21.15.191:6837/2406892975","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6836","nonce":2406892975},{"type":"v1","addr":"172.21.15.191:6837","nonce":2406892975}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24506":{"gid":24506,"name":"h","rank":2,"incarnation":20,"state":"up:active","state_seq":6,"addr":"172.21.15.130:6837/2604019369","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6836","nonce":2604019369},{"type":"v1","addr":"172.21.15.130:6837","nonce":2604019369}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24508":{"gid":24508,"name":"i","rank":1,"incarnation":18,"state":"up:active","state_seq":5,"addr":"172.21.15.191:6839/1977507302","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6838","nonce":1977507302},{"type":"v1","addr":"172.21.15.191:6839","nonce":1977507302}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24539":{"gid":24539,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.130:6839/2145479535","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6838","nonce":2145479535},{"type":"v1","addr":"172.21.15.130:6839","nonce":2145479535}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-07-14T19:17:29.109 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:29.108+0000 7f7b70f4c700 1 -- 172.21.15.40:0/1445715125 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f7b580a12a0 msgr2=0x7f7b580a3750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:17:29.109 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:29.108+0000 7f7b70f4c700 1 --2- 172.21.15.40:0/1445715125 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f7b580a12a0 0x7f7b580a3750 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f7b54005d90 tx=0x7f7b54005d00 comp rx=0 tx=0).stop 2023-07-14T19:17:29.110 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:29.108+0000 7f7b70f4c700 1 -- 172.21.15.40:0/1445715125 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7b6c0feb00 msgr2=0x7f7b6c102a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:17:29.110 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:29.108+0000 7f7b70f4c700 1 --2- 172.21.15.40:0/1445715125 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7b6c0feb00 0x7f7b6c102a90 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f7b5c00e970 tx=0x7f7b5c00ed30 comp rx=0 tx=0).stop 2023-07-14T19:17:29.110 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:29.108+0000 7f7b70f4c700 1 -- 172.21.15.40:0/1445715125 shutdown_connections 2023-07-14T19:17:29.110 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:29.108+0000 7f7b70f4c700 1 --2- 172.21.15.40:0/1445715125 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f7b580a12a0 0x7f7b580a3750 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:29.111 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:29.108+0000 7f7b70f4c700 1 --2- 172.21.15.40:0/1445715125 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7b6c100680 0x7f7b6c101610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:29.111 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:29.108+0000 7f7b70f4c700 1 --2- 172.21.15.40:0/1445715125 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7b6c0feb00 0x7f7b6c102a90 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:29.111 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:29.108+0000 7f7b70f4c700 1 --2- 172.21.15.40:0/1445715125 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7b6c0ffcf0 0x7f7b6c1010d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:29.111 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:29.108+0000 7f7b70f4c700 1 -- 172.21.15.40:0/1445715125 >> 172.21.15.40:0/1445715125 conn(0x7f7b6c0fa100 msgr2=0x7f7b6c0677c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:17:29.111 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:29.108+0000 7f7b70f4c700 1 -- 172.21.15.40:0/1445715125 shutdown_connections 2023-07-14T19:17:29.112 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:29.108+0000 7f7b70f4c700 1 -- 172.21.15.40:0/1445715125 wait complete. 2023-07-14T19:17:29.112 INFO:teuthology.orchestra.run.smithi040.stderr:dumped fsmap epoch 27 2023-07-14T19:17:29.122 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-07-14T19:17:29.248 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:29.246+0000 7f6758f49700 1 -- 172.21.15.40:0/3446943486 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f67540feb00 msgr2=0x7f67540fef10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:17:29.248 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:29.246+0000 7f6758f49700 1 --2- 172.21.15.40:0/3446943486 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f67540feb00 0x7f67540fef10 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f6744009990 tx=0x7f6744009ca0 comp rx=0 tx=0).stop 2023-07-14T19:17:29.248 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:29.246+0000 7f6758f49700 1 -- 172.21.15.40:0/3446943486 shutdown_connections 2023-07-14T19:17:29.248 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:29.246+0000 7f6758f49700 1 --2- 172.21.15.40:0/3446943486 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6754100680 0x7f6754106f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:29.249 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:29.246+0000 7f6758f49700 1 --2- 172.21.15.40:0/3446943486 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f67540feb00 0x7f67540fef10 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:29.249 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:29.246+0000 7f6758f49700 1 --2- 172.21.15.40:0/3446943486 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f67540ffcf0 0x7f6754100140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:29.249 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:29.246+0000 7f6758f49700 1 -- 172.21.15.40:0/3446943486 >> 172.21.15.40:0/3446943486 conn(0x7f67540fa100 msgr2=0x7f67540fc550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:17:29.249 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:29.246+0000 7f6758f49700 1 -- 172.21.15.40:0/3446943486 shutdown_connections 2023-07-14T19:17:29.250 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:29.246+0000 7f6758f49700 1 -- 172.21.15.40:0/3446943486 wait complete. 2023-07-14T19:17:29.252 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:29.247+0000 7f6758f49700 1 Processor -- start 2023-07-14T19:17:29.252 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:29.247+0000 7f6758f49700 1 -- start start 2023-07-14T19:17:29.253 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:29.247+0000 7f6758f49700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f67540feb00 0x7f6754195920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:17:29.253 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:29.247+0000 7f6758f49700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f67540ffcf0 0x7f6754195e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:17:29.253 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:29.247+0000 7f6758f49700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6754100680 0x7f675419ced0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:17:29.253 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:29.247+0000 7f6758f49700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f675419d4a0 con 0x7f67540feb00 2023-07-14T19:17:29.253 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:29.247+0000 7f6758f49700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f675419d5e0 con 0x7f67540ffcf0 2023-07-14T19:17:29.254 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:29.247+0000 7f6758f49700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f675419d770 con 0x7f6754100680 2023-07-14T19:17:29.254 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:29.247+0000 7f675259c700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f67540feb00 0x7f6754195920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:17:29.254 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:29.247+0000 7f675259c700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f67540feb00 0x7f6754195920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:34524/0 (socket says 172.21.15.40:34524) 2023-07-14T19:17:29.255 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:29.247+0000 7f675259c700 1 -- 172.21.15.40:0/2705479666 learned_addr learned my addr 172.21.15.40:0/2705479666 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-07-14T19:17:29.255 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:29.247+0000 7f6751d9b700 1 --2- 172.21.15.40:0/2705479666 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f67540ffcf0 0x7f6754195e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:17:29.255 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:29.247+0000 7f6752d9d700 1 --2- 172.21.15.40:0/2705479666 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6754100680 0x7f675419ced0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:17:29.255 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:29.247+0000 7f675259c700 1 -- 172.21.15.40:0/2705479666 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6754100680 msgr2=0x7f675419ced0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:17:29.255 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:29.247+0000 7f675259c700 1 --2- 172.21.15.40:0/2705479666 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6754100680 0x7f675419ced0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:29.256 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:29.247+0000 7f675259c700 1 -- 172.21.15.40:0/2705479666 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f67540ffcf0 msgr2=0x7f6754195e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:17:29.256 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:29.247+0000 7f675259c700 1 --2- 172.21.15.40:0/2705479666 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f67540ffcf0 0x7f6754195e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:29.256 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:29.247+0000 7f675259c700 1 -- 172.21.15.40:0/2705479666 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67440095f0 con 0x7f67540feb00 2023-07-14T19:17:29.256 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:29.247+0000 7f6751d9b700 1 --2- 172.21.15.40:0/2705479666 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f67540ffcf0 0x7f6754195e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T19:17:29.256 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:29.247+0000 7f675259c700 1 --2- 172.21.15.40:0/2705479666 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f67540feb00 0x7f6754195920 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f674400b620 tx=0x7f6744005ef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:17:29.257 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:29.247+0000 7f67437fe700 1 -- 172.21.15.40:0/2705479666 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f674401d070 con 0x7f67540feb00 2023-07-14T19:17:29.257 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:29.248+0000 7f67437fe700 1 -- 172.21.15.40:0/2705479666 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f67440047e0 con 0x7f67540feb00 2023-07-14T19:17:29.257 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:29.248+0000 7f6758f49700 1 -- 172.21.15.40:0/2705479666 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f675419d9f0 con 0x7f67540feb00 2023-07-14T19:17:29.257 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:29.248+0000 7f67437fe700 1 -- 172.21.15.40:0/2705479666 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6744005560 con 0x7f67540feb00 2023-07-14T19:17:29.257 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:29.248+0000 7f6758f49700 1 -- 172.21.15.40:0/2705479666 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f675419de10 con 0x7f67540feb00 2023-07-14T19:17:29.258 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:29.248+0000 7f6758f49700 1 -- 172.21.15.40:0/2705479666 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f675410a2d0 con 0x7f67540feb00 2023-07-14T19:17:29.258 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:29.250+0000 7f67437fe700 1 -- 172.21.15.40:0/2705479666 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 19) v1 ==== 135766+0+0 (secure 0 0 0) 0x7f6744004950 con 0x7f67540feb00 2023-07-14T19:17:29.258 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:29.250+0000 7f67437fe700 1 --2- 172.21.15.40:0/2705479666 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f673c0a1220 0x7f673c0a36d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:17:29.258 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:29.250+0000 7f67437fe700 1 -- 172.21.15.40:0/2705479666 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 8269+0+0 (secure 0 0 0) 0x7f67440cdff0 con 0x7f67540feb00 2023-07-14T19:17:29.258 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:29.250+0000 7f6751d9b700 1 --2- 172.21.15.40:0/2705479666 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f673c0a1220 0x7f673c0a36d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:17:29.259 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:29.251+0000 7f6751d9b700 1 --2- 172.21.15.40:0/2705479666 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f673c0a1220 0x7f673c0a36d0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f6748009cb0 tx=0x7f6748009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:17:29.259 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:29.253+0000 7f67437fe700 1 -- 172.21.15.40:0/2705479666 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f6744097650 con 0x7f67540feb00 2023-07-14T19:17:29.456 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:29.454+0000 7f6758f49700 1 -- 172.21.15.40:0/2705479666 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f675419e3f0 con 0x7f67540feb00 2023-07-14T19:17:29.457 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:29.455+0000 7f67437fe700 1 -- 172.21.15.40:0/2705479666 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v83) v1 ==== 74+0+22647 (secure 0 0 0) 0x7f674409ac70 con 0x7f67540feb00 2023-07-14T19:17:29.458 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-07-14T19:17:29.459 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":83,"fsid":"adc52f04-2279-11ee-9b33-001a4aab830c","created":"2023-07-14T19:08:56.893730+0000","modified":"2023-07-14T19:16:52.763512+0000","last_up_change":"2023-07-14T19:15:16.632979+0000","last_in_change":"2023-07-14T19:14:58.961702+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-07-14T19:12:03.271333+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-07-14T19:15:56.073464+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-07-14T19:15:57.018726+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"0d79356a-95d2-49ad-857b-2092ae9f7e13","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":3599703125},{"type":"v1","addr":"172.21.15.40:6803","nonce":3599703125}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":3599703125},{"type":"v1","addr":"172.21.15.40:6805","nonce":3599703125}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":3599703125},{"type":"v1","addr":"172.21.15.40:6809","nonce":3599703125}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":3599703125},{"type":"v1","addr":"172.21.15.40:6807","nonce":3599703125}]},"public_addr":"172.21.15.40:6803/3599703125","cluster_addr":"172.21.15.40:6805/3599703125","heartbeat_back_addr":"172.21.15.40:6809/3599703125","heartbeat_front_addr":"172.21.15.40:6807/3599703125","state":["exists","up"]},{"osd":1,"uuid":"40f8a3d1-a5e4-473c-b9f2-7600ec22be0d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6800","nonce":2512760702},{"type":"v1","addr":"172.21.15.130:6801","nonce":2512760702}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6802","nonce":2512760702},{"type":"v1","addr":"172.21.15.130:6803","nonce":2512760702}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6806","nonce":2512760702},{"type":"v1","addr":"172.21.15.130:6807","nonce":2512760702}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6804","nonce":2512760702},{"type":"v1","addr":"172.21.15.130:6805","nonce":2512760702}]},"public_addr":"172.21.15.130:6801/2512760702","cluster_addr":"172.21.15.130:6803/2512760702","heartbeat_back_addr":"172.21.15.130:6807/2512760702","heartbeat_front_addr":"172.21.15.130:6805/2512760702","state":["exists","up"]},{"osd":2,"uuid":"5a157891-cb7a-4ea6-9239-f48a7c67c003","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6800","nonce":589569135},{"type":"v1","addr":"172.21.15.191:6801","nonce":589569135}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6802","nonce":589569135},{"type":"v1","addr":"172.21.15.191:6803","nonce":589569135}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6806","nonce":589569135},{"type":"v1","addr":"172.21.15.191:6807","nonce":589569135}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6804","nonce":589569135},{"type":"v1","addr":"172.21.15.191:6805","nonce":589569135}]},"public_addr":"172.21.15.191:6801/589569135","cluster_addr":"172.21.15.191:6803/589569135","heartbeat_back_addr":"172.21.15.191:6807/589569135","heartbeat_front_addr":"172.21.15.191:6805/589569135","state":["exists","up"]},{"osd":3,"uuid":"8020e7dc-ac40-487f-857a-8da40bccb459","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":996287860},{"type":"v1","addr":"172.21.15.40:6811","nonce":996287860}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6812","nonce":996287860},{"type":"v1","addr":"172.21.15.40:6813","nonce":996287860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6816","nonce":996287860},{"type":"v1","addr":"172.21.15.40:6817","nonce":996287860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":996287860},{"type":"v1","addr":"172.21.15.40:6815","nonce":996287860}]},"public_addr":"172.21.15.40:6811/996287860","cluster_addr":"172.21.15.40:6813/996287860","heartbeat_back_addr":"172.21.15.40:6817/996287860","heartbeat_front_addr":"172.21.15.40:6815/996287860","state":["exists","up"]},{"osd":4,"uuid":"c5501b35-f62d-482d-8d22-115c88f0298e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6808","nonce":3507448720},{"type":"v1","addr":"172.21.15.130:6809","nonce":3507448720}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6810","nonce":3507448720},{"type":"v1","addr":"172.21.15.130:6811","nonce":3507448720}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6814","nonce":3507448720},{"type":"v1","addr":"172.21.15.130:6815","nonce":3507448720}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6812","nonce":3507448720},{"type":"v1","addr":"172.21.15.130:6813","nonce":3507448720}]},"public_addr":"172.21.15.130:6809/3507448720","cluster_addr":"172.21.15.130:6811/3507448720","heartbeat_back_addr":"172.21.15.130:6815/3507448720","heartbeat_front_addr":"172.21.15.130:6813/3507448720","state":["exists","up"]},{"osd":5,"uuid":"7770b6d6-5874-493a-986a-84db482cee42","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6808","nonce":1781944663},{"type":"v1","addr":"172.21.15.191:6809","nonce":1781944663}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6810","nonce":1781944663},{"type":"v1","addr":"172.21.15.191:6811","nonce":1781944663}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6814","nonce":1781944663},{"type":"v1","addr":"172.21.15.191:6815","nonce":1781944663}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6812","nonce":1781944663},{"type":"v1","addr":"172.21.15.191:6813","nonce":1781944663}]},"public_addr":"172.21.15.191:6809/1781944663","cluster_addr":"172.21.15.191:6811/1781944663","heartbeat_back_addr":"172.21.15.191:6815/1781944663","heartbeat_front_addr":"172.21.15.191:6813/1781944663","state":["exists","up"]},{"osd":6,"uuid":"ef65ad00-dfee-4300-9961-dfc8573aab06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6818","nonce":693186773},{"type":"v1","addr":"172.21.15.40:6819","nonce":693186773}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6820","nonce":693186773},{"type":"v1","addr":"172.21.15.40:6821","nonce":693186773}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6824","nonce":693186773},{"type":"v1","addr":"172.21.15.40:6825","nonce":693186773}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6822","nonce":693186773},{"type":"v1","addr":"172.21.15.40:6823","nonce":693186773}]},"public_addr":"172.21.15.40:6819/693186773","cluster_addr":"172.21.15.40:6821/693186773","heartbeat_back_addr":"172.21.15.40:6825/693186773","heartbeat_front_addr":"172.21.15.40:6823/693186773","state":["exists","up"]},{"osd":7,"uuid":"1700fbfa-cbfc-4f76-97be-727a3b100740","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":50,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6816","nonce":2384019528},{"type":"v1","addr":"172.21.15.130:6817","nonce":2384019528}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6818","nonce":2384019528},{"type":"v1","addr":"172.21.15.130:6819","nonce":2384019528}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6822","nonce":2384019528},{"type":"v1","addr":"172.21.15.130:6823","nonce":2384019528}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6820","nonce":2384019528},{"type":"v1","addr":"172.21.15.130:6821","nonce":2384019528}]},"public_addr":"172.21.15.130:6817/2384019528","cluster_addr":"172.21.15.130:6819/2384019528","heartbeat_back_addr":"172.21.15.130:6823/2384019528","heartbeat_front_addr":"172.21.15.130:6821/2384019528","state":["exists","up"]},{"osd":8,"uuid":"b76d4033-c1e2-457b-8571-c8a49fe443f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":57,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6816","nonce":2386818417},{"type":"v1","addr":"172.21.15.191:6817","nonce":2386818417}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6818","nonce":2386818417},{"type":"v1","addr":"172.21.15.191:6819","nonce":2386818417}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6822","nonce":2386818417},{"type":"v1","addr":"172.21.15.191:6823","nonce":2386818417}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6820","nonce":2386818417},{"type":"v1","addr":"172.21.15.191:6821","nonce":2386818417}]},"public_addr":"172.21.15.191:6817/2386818417","cluster_addr":"172.21.15.191:6819/2386818417","heartbeat_back_addr":"172.21.15.191:6823/2386818417","heartbeat_front_addr":"172.21.15.191:6821/2386818417","state":["exists","up"]},{"osd":9,"uuid":"df31bf2f-e30e-41b6-b2b9-0e26a07a88bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":64,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6826","nonce":2228380263},{"type":"v1","addr":"172.21.15.40:6827","nonce":2228380263}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6828","nonce":2228380263},{"type":"v1","addr":"172.21.15.40:6829","nonce":2228380263}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":2228380263},{"type":"v1","addr":"172.21.15.40:6833","nonce":2228380263}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6830","nonce":2228380263},{"type":"v1","addr":"172.21.15.40:6831","nonce":2228380263}]},"public_addr":"172.21.15.40:6827/2228380263","cluster_addr":"172.21.15.40:6829/2228380263","heartbeat_back_addr":"172.21.15.40:6833/2228380263","heartbeat_front_addr":"172.21.15.40:6831/2228380263","state":["exists","up"]},{"osd":10,"uuid":"4a93d621-49ee-4ad7-9e49-750614faf912","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6824","nonce":2417067798},{"type":"v1","addr":"172.21.15.130:6825","nonce":2417067798}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6826","nonce":2417067798},{"type":"v1","addr":"172.21.15.130:6827","nonce":2417067798}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6830","nonce":2417067798},{"type":"v1","addr":"172.21.15.130:6831","nonce":2417067798}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6828","nonce":2417067798},{"type":"v1","addr":"172.21.15.130:6829","nonce":2417067798}]},"public_addr":"172.21.15.130:6825/2417067798","cluster_addr":"172.21.15.130:6827/2417067798","heartbeat_back_addr":"172.21.15.130:6831/2417067798","heartbeat_front_addr":"172.21.15.130:6829/2417067798","state":["exists","up"]},{"osd":11,"uuid":"f43a2a6f-82f3-42ac-b77c-4f1b34da6918","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":76,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6824","nonce":2886059446},{"type":"v1","addr":"172.21.15.191:6825","nonce":2886059446}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6826","nonce":2886059446},{"type":"v1","addr":"172.21.15.191:6827","nonce":2886059446}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6830","nonce":2886059446},{"type":"v1","addr":"172.21.15.191:6831","nonce":2886059446}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6828","nonce":2886059446},{"type":"v1","addr":"172.21.15.191:6829","nonce":2886059446}]},"public_addr":"172.21.15.191:6825/2886059446","cluster_addr":"172.21.15.191:6827/2886059446","heartbeat_back_addr":"172.21.15.191:6831/2886059446","heartbeat_front_addr":"172.21.15.191:6829/2886059446","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:11:18.320495+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:11:39.901199+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:12:00.869042+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:12:23.292742+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:12:44.587019+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:13:06.494893+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:13:28.378374+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:13:50.033491+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:14:10.937062+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:14:32.358498+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:14:53.666584+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:15:14.837566+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"2.b","mappings":[{"from":8,"to":11}]},{"pgid":"2.c","mappings":[{"from":2,"to":4}]},{"pgid":"2.1f","mappings":[{"from":8,"to":1}]},{"pgid":"3.e","mappings":[{"from":4,"to":7}]},{"pgid":"3.17","mappings":[{"from":4,"to":8}]},{"pgid":"3.1a","mappings":[{"from":5,"to":6}]},{"pgid":"3.20","mappings":[{"from":5,"to":9}]},{"pgid":"3.27","mappings":[{"from":3,"to":9}]},{"pgid":"3.2c","mappings":[{"from":4,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.40:6801/3698757264":"2023-07-15T19:09:50.732458+0000","172.21.15.40:6800/3698757264":"2023-07-15T19:09:50.732458+0000","172.21.15.40:6800/2569015667":"2023-07-15T19:09:24.441543+0000","172.21.15.40:0/3653961003":"2023-07-15T19:09:24.441543+0000","172.21.15.40:0/3568861680":"2023-07-15T19:09:24.441543+0000","172.21.15.40:0/3249773172":"2023-07-15T19:09:50.732458+0000","172.21.15.40:0/4084215480":"2023-07-15T19:09:24.441543+0000","172.21.15.40:0/1163941760":"2023-07-15T19:09:50.732458+0000","172.21.15.40:6801/2569015667":"2023-07-15T19:09:24.441543+0000","172.21.15.40:0/2326639960":"2023-07-15T19:09:50.732458+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-07-14T19:17:29.461 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:29.457+0000 7f6758f49700 1 -- 172.21.15.40:0/2705479666 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f673c0a1220 msgr2=0x7f673c0a36d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:17:29.461 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:29.457+0000 7f6758f49700 1 --2- 172.21.15.40:0/2705479666 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f673c0a1220 0x7f673c0a36d0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f6748009cb0 tx=0x7f6748009260 comp rx=0 tx=0).stop 2023-07-14T19:17:29.461 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:29.457+0000 7f6758f49700 1 -- 172.21.15.40:0/2705479666 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f67540feb00 msgr2=0x7f6754195920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:17:29.462 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:29.458+0000 7f6758f49700 1 --2- 172.21.15.40:0/2705479666 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f67540feb00 0x7f6754195920 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f674400b620 tx=0x7f6744005ef0 comp rx=0 tx=0).stop 2023-07-14T19:17:29.462 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:29.458+0000 7f6758f49700 1 -- 172.21.15.40:0/2705479666 shutdown_connections 2023-07-14T19:17:29.463 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:29.458+0000 7f6758f49700 1 --2- 172.21.15.40:0/2705479666 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f673c0a1220 0x7f673c0a36d0 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:29.463 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:29.458+0000 7f6758f49700 1 --2- 172.21.15.40:0/2705479666 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f6754100680 0x7f675419ced0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:29.464 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:29.458+0000 7f6758f49700 1 --2- 172.21.15.40:0/2705479666 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f67540feb00 0x7f6754195920 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:29.464 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:29.458+0000 7f6758f49700 1 --2- 172.21.15.40:0/2705479666 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f67540ffcf0 0x7f6754195e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:29.465 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:29.458+0000 7f6758f49700 1 -- 172.21.15.40:0/2705479666 >> 172.21.15.40:0/2705479666 conn(0x7f67540fa100 msgr2=0x7f6754067840 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:17:29.465 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:29.458+0000 7f6758f49700 1 -- 172.21.15.40:0/2705479666 shutdown_connections 2023-07-14T19:17:29.465 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:29.458+0000 7f6758f49700 1 -- 172.21.15.40:0/2705479666 wait complete. 2023-07-14T19:17:29.473 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2023-07-14T19:17:29.600 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:29.598+0000 7f0957438700 1 -- 172.21.15.40:0/1297058441 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f09501018c0 msgr2=0x7f0950108180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:17:29.600 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:29.598+0000 7f0957438700 1 --2- 172.21.15.40:0/1297058441 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f09501018c0 0x7f0950108180 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f094800b0a0 tx=0x7f094800b3b0 comp rx=0 tx=0).stop 2023-07-14T19:17:29.601 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:29.598+0000 7f0957438700 1 -- 172.21.15.40:0/1297058441 shutdown_connections 2023-07-14T19:17:29.601 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:29.598+0000 7f0957438700 1 --2- 172.21.15.40:0/1297058441 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f09500fdec0 0x7f09500fe2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:29.601 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:29.598+0000 7f0957438700 1 --2- 172.21.15.40:0/1297058441 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f09501018c0 0x7f0950108180 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:29.601 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:29.598+0000 7f0957438700 1 --2- 172.21.15.40:0/1297058441 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0950100f30 0x7f0950101380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:29.602 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:29.598+0000 7f0957438700 1 -- 172.21.15.40:0/1297058441 >> 172.21.15.40:0/1297058441 conn(0x7f09500f98a0 msgr2=0x7f09500fbcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:17:29.602 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:29.598+0000 7f0957438700 1 -- 172.21.15.40:0/1297058441 shutdown_connections 2023-07-14T19:17:29.602 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:29.598+0000 7f0957438700 1 -- 172.21.15.40:0/1297058441 wait complete. 2023-07-14T19:17:29.604 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:29.599+0000 7f0957438700 1 Processor -- start 2023-07-14T19:17:29.604 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:29.599+0000 7f0957438700 1 -- start start 2023-07-14T19:17:29.605 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:29.599+0000 7f0957438700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f09500fdec0 0x7f09501008a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:17:29.605 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:29.599+0000 7f0957438700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0950100f30 0x7f09500feee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:17:29.605 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:29.599+0000 7f0957438700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f09501018c0 0x7f09500ff420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:17:29.605 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:29.599+0000 7f0957438700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09500ffbc0 con 0x7f0950100f30 2023-07-14T19:17:29.606 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:29.599+0000 7f0957438700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09500ffd30 con 0x7f09501018c0 2023-07-14T19:17:29.606 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:29.599+0000 7f0957438700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f095019fb50 con 0x7f09500fdec0 2023-07-14T19:17:29.606 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:29.599+0000 7f09549d3700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0950100f30 0x7f09500feee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:17:29.606 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:29.599+0000 7f09549d3700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0950100f30 0x7f09500feee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:34540/0 (socket says 172.21.15.40:34540) 2023-07-14T19:17:29.606 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:29.599+0000 7f09549d3700 1 -- 172.21.15.40:0/3616277774 learned_addr learned my addr 172.21.15.40:0/3616277774 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-07-14T19:17:29.607 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:29.599+0000 7f09551d4700 1 --2- 172.21.15.40:0/3616277774 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f09500fdec0 0x7f09501008a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:17:29.607 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:29.599+0000 7f09559d5700 1 --2- 172.21.15.40:0/3616277774 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f09501018c0 0x7f09500ff420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:17:29.607 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:29.599+0000 7f09549d3700 1 -- 172.21.15.40:0/3616277774 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f09500fdec0 msgr2=0x7f09501008a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:17:29.607 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:29.599+0000 7f09549d3700 1 --2- 172.21.15.40:0/3616277774 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f09500fdec0 0x7f09501008a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:29.607 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:29.599+0000 7f09549d3700 1 -- 172.21.15.40:0/3616277774 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f09501018c0 msgr2=0x7f09500ff420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:17:29.608 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:29.599+0000 7f09549d3700 1 --2- 172.21.15.40:0/3616277774 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f09501018c0 0x7f09500ff420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:29.608 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:29.599+0000 7f09549d3700 1 -- 172.21.15.40:0/3616277774 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0948009d00 con 0x7f0950100f30 2023-07-14T19:17:29.608 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:29.599+0000 7f09559d5700 1 --2- 172.21.15.40:0/3616277774 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f09501018c0 0x7f09500ff420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-14T19:17:29.608 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:29.600+0000 7f09549d3700 1 --2- 172.21.15.40:0/3616277774 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0950100f30 0x7f09500feee0 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f094c00b530 tx=0x7f094c00b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:17:29.608 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:29.600+0000 7f09427fc700 1 -- 172.21.15.40:0/3616277774 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f094c011830 con 0x7f0950100f30 2023-07-14T19:17:29.609 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:29.600+0000 7f09427fc700 1 -- 172.21.15.40:0/3616277774 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f094c011e50 con 0x7f0950100f30 2023-07-14T19:17:29.609 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:29.600+0000 7f0957438700 1 -- 172.21.15.40:0/3616277774 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f095019fe30 con 0x7f0950100f30 2023-07-14T19:17:29.609 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:29.600+0000 7f09427fc700 1 -- 172.21.15.40:0/3616277774 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f094c00f2e0 con 0x7f0950100f30 2023-07-14T19:17:29.609 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:29.600+0000 7f0957438700 1 -- 172.21.15.40:0/3616277774 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f09501a02b0 con 0x7f0950100f30 2023-07-14T19:17:29.609 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:29.601+0000 7f0957438700 1 -- 172.21.15.40:0/3616277774 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f095010b470 con 0x7f0950100f30 2023-07-14T19:17:29.610 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:29.602+0000 7f09427fc700 1 -- 172.21.15.40:0/3616277774 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 19) v1 ==== 135766+0+0 (secure 0 0 0) 0x7f094c010580 con 0x7f0950100f30 2023-07-14T19:17:29.610 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:29.602+0000 7f09427fc700 1 --2- 172.21.15.40:0/3616277774 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f093c0a11e0 0x7f093c0a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:17:29.610 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:29.603+0000 7f09551d4700 1 --2- 172.21.15.40:0/3616277774 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f093c0a11e0 0x7f093c0a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:17:29.610 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:29.603+0000 7f09427fc700 1 -- 172.21.15.40:0/3616277774 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 8269+0+0 (secure 0 0 0) 0x7f094c0cc510 con 0x7f0950100f30 2023-07-14T19:17:29.610 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:29.603+0000 7f09551d4700 1 --2- 172.21.15.40:0/3616277774 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f093c0a11e0 0x7f093c0a3690 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f0944005fd0 tx=0x7f094400a560 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:17:29.611 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:29.607+0000 7f09427fc700 1 -- 172.21.15.40:0/3616277774 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f094c095b90 con 0x7f0950100f30 2023-07-14T19:17:29.853 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:29.852+0000 7f0957438700 1 -- 172.21.15.40:0/3616277774 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f095010b630 con 0x7f0950100f30 2023-07-14T19:17:29.854 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:17:29 smithi040 ceph-mon[96518]: from='client.? 172.21.15.40:0/2292415615' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T19:17:29.855 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:17:29 smithi040 ceph-mon[96518]: from='client.? 172.21.15.40:0/1445715125' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T19:17:29.855 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:17:29 smithi040 ceph-mon[96518]: from='client.? 172.21.15.40:0/2705479666' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T19:17:29.859 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:29.857+0000 7f09427fc700 1 -- 172.21.15.40:0/3616277774 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v28) v1 ==== 161+0+0 (secure 0 0 0) 0x7f094c0991b0 con 0x7f0950100f30 2023-07-14T19:17:29.861 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:29.859+0000 7f0957438700 1 -- 172.21.15.40:0/3616277774 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f093c0a11e0 msgr2=0x7f093c0a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:17:29.861 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:29.859+0000 7f0957438700 1 --2- 172.21.15.40:0/3616277774 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f093c0a11e0 0x7f093c0a3690 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f0944005fd0 tx=0x7f094400a560 comp rx=0 tx=0).stop 2023-07-14T19:17:29.862 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:29.859+0000 7f0957438700 1 -- 172.21.15.40:0/3616277774 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0950100f30 msgr2=0x7f09500feee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:17:29.862 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:29.859+0000 7f0957438700 1 --2- 172.21.15.40:0/3616277774 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0950100f30 0x7f09500feee0 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f094c00b530 tx=0x7f094c00b8f0 comp rx=0 tx=0).stop 2023-07-14T19:17:29.862 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:29.860+0000 7f0957438700 1 -- 172.21.15.40:0/3616277774 shutdown_connections 2023-07-14T19:17:29.862 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:29.860+0000 7f0957438700 1 --2- 172.21.15.40:0/3616277774 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f093c0a11e0 0x7f093c0a3690 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:29.862 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:29.860+0000 7f0957438700 1 --2- 172.21.15.40:0/3616277774 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f09500fdec0 0x7f09501008a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:29.863 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:29.860+0000 7f0957438700 1 --2- 172.21.15.40:0/3616277774 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f09501018c0 0x7f09500ff420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:29.863 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:29.860+0000 7f0957438700 1 --2- 172.21.15.40:0/3616277774 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0950100f30 0x7f09500feee0 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:29.863 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:29.860+0000 7f0957438700 1 -- 172.21.15.40:0/3616277774 >> 172.21.15.40:0/3616277774 conn(0x7f09500f98a0 msgr2=0x7f09500fbad0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:17:29.863 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:29.860+0000 7f0957438700 1 -- 172.21.15.40:0/3616277774 shutdown_connections 2023-07-14T19:17:29.863 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:29.860+0000 7f0957438700 1 -- 172.21.15.40:0/3616277774 wait complete. 2023-07-14T19:17:29.864 INFO:teuthology.orchestra.run.smithi040.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2023-07-14T19:17:29.873 INFO:journalctl@ceph.mds.b.smithi130.stdout:Jul 14 19:17:29 smithi130 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-mds-b[122843]: -7097> 2023-07-14T19:16:08.854+0000 7fc0c3b89700 -1 mds.pinger is_rank_lagging: rank=3 was never sent ping request. 2023-07-14T19:17:29.874 INFO:journalctl@ceph.mds.b.smithi130.stdout:Jul 14 19:17:29 smithi130 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-mds-b[122843]: -7095> 2023-07-14T19:16:08.854+0000 7fc0c3b89700 -1 mds.pinger is_rank_lagging: rank=4 was never sent ping request. 2023-07-14T19:17:29.874 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:17:29 smithi130 ceph-mon[97832]: from='client.? 172.21.15.40:0/2292415615' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T19:17:29.874 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:17:29 smithi130 ceph-mon[97832]: from='client.? 172.21.15.40:0/1445715125' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T19:17:29.875 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:17:29 smithi130 ceph-mon[97832]: from='client.? 172.21.15.40:0/2705479666' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T19:17:29.876 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2023-07-14T19:17:29.892 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:17:29 smithi191 ceph-mon[98311]: from='client.? 172.21.15.40:0/2292415615' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T19:17:29.893 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:17:29 smithi191 ceph-mon[98311]: from='client.? 172.21.15.40:0/1445715125' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T19:17:29.893 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:17:29 smithi191 ceph-mon[98311]: from='client.? 172.21.15.40:0/2705479666' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T19:17:30.008 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:30.006+0000 7f0a05fd7700 1 -- 172.21.15.40:0/3612279090 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0a00100690 msgr2=0x7f0a00106f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:17:30.009 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:30.006+0000 7f0a05fd7700 1 --2- 172.21.15.40:0/3612279090 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0a00100690 0x7f0a00106f50 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f09f800b0a0 tx=0x7f09f800b3b0 comp rx=0 tx=0).stop 2023-07-14T19:17:30.009 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:30.006+0000 7f0a05fd7700 1 -- 172.21.15.40:0/3612279090 shutdown_connections 2023-07-14T19:17:30.009 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:30.006+0000 7f0a05fd7700 1 --2- 172.21.15.40:0/3612279090 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0a00100690 0x7f0a00106f50 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:30.009 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:30.006+0000 7f0a05fd7700 1 --2- 172.21.15.40:0/3612279090 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0a000feb10 0x7f0a000fef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:30.010 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:30.006+0000 7f0a05fd7700 1 --2- 172.21.15.40:0/3612279090 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0a000ffd00 0x7f0a00100150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:30.010 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:30.006+0000 7f0a05fd7700 1 -- 172.21.15.40:0/3612279090 >> 172.21.15.40:0/3612279090 conn(0x7f0a000fa130 msgr2=0x7f0a000fc560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:17:30.010 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:30.007+0000 7f0a05fd7700 1 -- 172.21.15.40:0/3612279090 shutdown_connections 2023-07-14T19:17:30.010 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:30.007+0000 7f0a05fd7700 1 -- 172.21.15.40:0/3612279090 wait complete. 2023-07-14T19:17:30.010 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:30.007+0000 7f0a05fd7700 1 Processor -- start 2023-07-14T19:17:30.012 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:30.007+0000 7f0a05fd7700 1 -- start start 2023-07-14T19:17:30.013 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:30.007+0000 7f0a05fd7700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0a000feb10 0x7f0a00069360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:17:30.013 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:30.007+0000 7f0a05fd7700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0a000ffd00 0x7f0a0019a770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:17:30.013 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:30.007+0000 7f0a05fd7700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0a00100690 0x7f0a0019cdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:17:30.013 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:30.007+0000 7f0a05fd7700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a0019d590 con 0x7f0a000ffd00 2023-07-14T19:17:30.013 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:30.007+0000 7f0a05fd7700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a0019d700 con 0x7f0a00100690 2023-07-14T19:17:30.014 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:30.007+0000 7f0a05fd7700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a0019d8c0 con 0x7f0a000feb10 2023-07-14T19:17:30.014 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:30.008+0000 7f09feffd700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0a000ffd00 0x7f0a0019a770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:17:30.014 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:30.008+0000 7f09feffd700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0a000ffd00 0x7f0a0019a770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:34580/0 (socket says 172.21.15.40:34580) 2023-07-14T19:17:30.014 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:30.008+0000 7f09feffd700 1 -- 172.21.15.40:0/559386693 learned_addr learned my addr 172.21.15.40:0/559386693 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-07-14T19:17:30.014 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:30.008+0000 7f09fffff700 1 --2- 172.21.15.40:0/559386693 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0a00100690 0x7f0a0019cdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:17:30.015 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:30.008+0000 7f09ff7fe700 1 --2- 172.21.15.40:0/559386693 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0a000feb10 0x7f0a00069360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:17:30.015 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:30.008+0000 7f09feffd700 1 -- 172.21.15.40:0/559386693 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0a000feb10 msgr2=0x7f0a00069360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:17:30.015 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:30.008+0000 7f09feffd700 1 --2- 172.21.15.40:0/559386693 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0a000feb10 0x7f0a00069360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:30.015 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:30.008+0000 7f09feffd700 1 -- 172.21.15.40:0/559386693 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0a00100690 msgr2=0x7f0a0019cdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:17:30.015 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:30.008+0000 7f09feffd700 1 --2- 172.21.15.40:0/559386693 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0a00100690 0x7f0a0019cdc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:30.016 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:30.008+0000 7f09feffd700 1 -- 172.21.15.40:0/559386693 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09f8009d00 con 0x7f0a000ffd00 2023-07-14T19:17:30.016 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:30.008+0000 7f09feffd700 1 --2- 172.21.15.40:0/559386693 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0a000ffd00 0x7f0a0019a770 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f09f400d950 tx=0x7f09f400dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:17:30.016 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:30.008+0000 7f09fcff9700 1 -- 172.21.15.40:0/559386693 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09f400f890 con 0x7f0a000ffd00 2023-07-14T19:17:30.016 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:30.008+0000 7f0a05fd7700 1 -- 172.21.15.40:0/559386693 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0a0019dba0 con 0x7f0a000ffd00 2023-07-14T19:17:30.017 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:30.008+0000 7f09fcff9700 1 -- 172.21.15.40:0/559386693 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f09f4010460 con 0x7f0a000ffd00 2023-07-14T19:17:30.017 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:30.008+0000 7f09fcff9700 1 -- 172.21.15.40:0/559386693 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09f400e610 con 0x7f0a000ffd00 2023-07-14T19:17:30.017 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:30.008+0000 7f0a05fd7700 1 -- 172.21.15.40:0/559386693 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0a0019e0b0 con 0x7f0a000ffd00 2023-07-14T19:17:30.017 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:30.009+0000 7f0a05fd7700 1 -- 172.21.15.40:0/559386693 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0a0010a2f0 con 0x7f0a000ffd00 2023-07-14T19:17:30.017 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:30.010+0000 7f09fcff9700 1 -- 172.21.15.40:0/559386693 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 19) v1 ==== 135766+0+0 (secure 0 0 0) 0x7f09f40108d0 con 0x7f0a000ffd00 2023-07-14T19:17:30.018 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:30.011+0000 7f09fcff9700 1 --2- 172.21.15.40:0/559386693 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f09e80a11e0 0x7f09e80a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:17:30.018 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:30.011+0000 7f09fcff9700 1 -- 172.21.15.40:0/559386693 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 8699+0+0 (secure 0 0 0) 0x7f09f40cca80 con 0x7f0a000ffd00 2023-07-14T19:17:30.018 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:30.011+0000 7f09ff7fe700 1 --2- 172.21.15.40:0/559386693 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f09e80a11e0 0x7f09e80a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:17:30.018 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:30.011+0000 7f09ff7fe700 1 --2- 172.21.15.40:0/559386693 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f09e80a11e0 0x7f09e80a3690 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f0a000ffb60 tx=0x7f09f000b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:17:30.019 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:30.015+0000 7f09fcff9700 1 -- 172.21.15.40:0/559386693 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f09f4095e40 con 0x7f0a000ffd00 2023-07-14T19:17:30.221 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:17:29 smithi040 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-mon-a[96494]: 2023-07-14T19:17:29.852+0000 7fc74623f700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2023-07-14T19:17:30.221 INFO:journalctl@ceph.mds.a.smithi040.stdout:Jul 14 19:17:29 smithi040 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-mds-a[131223]: ignoring --setuser ceph since I am not root 2023-07-14T19:17:30.221 INFO:journalctl@ceph.mds.a.smithi040.stdout:Jul 14 19:17:29 smithi040 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-mds-a[131223]: ignoring --setgroup ceph since I am not root 2023-07-14T19:17:30.222 INFO:journalctl@ceph.mds.a.smithi040.stdout:Jul 14 19:17:29 smithi040 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-mds-a[131223]: starting mds.a at 2023-07-14T19:17:30.222 INFO:journalctl@ceph.mds.g.smithi040.stdout:Jul 14 19:17:29 smithi040 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-mds-g[130836]: ignoring --setuser ceph since I am not root 2023-07-14T19:17:30.222 INFO:journalctl@ceph.mds.g.smithi040.stdout:Jul 14 19:17:29 smithi040 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-mds-g[130836]: ignoring --setgroup ceph since I am not root 2023-07-14T19:17:30.222 INFO:journalctl@ceph.mds.g.smithi040.stdout:Jul 14 19:17:29 smithi040 ceph-mds[130860]: -- 172.21.15.40:0/93566005 <== mon.1 v2:172.21.15.191:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5638228951e0 con 0x56382286a800 2023-07-14T19:17:30.223 INFO:journalctl@ceph.mds.g.smithi040.stdout:Jul 14 19:17:29 smithi040 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-mds-g[130836]: starting mds.g at 2023-07-14T19:17:30.223 INFO:journalctl@ceph.mds.c.smithi191.stdout:Jul 14 19:17:29 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-mds-c[124204]: ignoring --setuser ceph since I am not root 2023-07-14T19:17:30.224 INFO:journalctl@ceph.mds.c.smithi191.stdout:Jul 14 19:17:29 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-mds-c[124204]: ignoring --setgroup ceph since I am not root 2023-07-14T19:17:30.224 INFO:journalctl@ceph.mds.c.smithi191.stdout:Jul 14 19:17:29 smithi191 ceph-mds[124228]: -- 172.21.15.191:0/1823626576 <== mon.2 v2:172.21.15.130:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x555b19cab1e0 con 0x555b19c80800 2023-07-14T19:17:30.224 INFO:journalctl@ceph.mds.c.smithi191.stdout:Jul 14 19:17:29 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-mds-c[124204]: starting mds.c at 2023-07-14T19:17:30.224 INFO:journalctl@ceph.mds.f.smithi191.stdout:Jul 14 19:17:29 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-mds-f[123863]: ignoring --setuser ceph since I am not root 2023-07-14T19:17:30.225 INFO:journalctl@ceph.mds.f.smithi191.stdout:Jul 14 19:17:29 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-mds-f[123863]: ignoring --setgroup ceph since I am not root 2023-07-14T19:17:30.225 INFO:journalctl@ceph.mds.f.smithi191.stdout:Jul 14 19:17:29 smithi191 ceph-mds[123887]: -- 172.21.15.191:0/3463948906 <== mon.2 v2:172.21.15.130:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55d785af71e0 con 0x55d785acd000 2023-07-14T19:17:30.225 INFO:journalctl@ceph.mds.f.smithi191.stdout:Jul 14 19:17:29 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-mds-f[123863]: starting mds.f at 2023-07-14T19:17:30.225 INFO:journalctl@ceph.mds.i.smithi191.stdout:Jul 14 19:17:29 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-mds-i[124544]: ignoring --setuser ceph since I am not root 2023-07-14T19:17:30.226 INFO:journalctl@ceph.mds.i.smithi191.stdout:Jul 14 19:17:29 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-mds-i[124544]: ignoring --setgroup ceph since I am not root 2023-07-14T19:17:30.226 INFO:journalctl@ceph.mds.i.smithi191.stdout:Jul 14 19:17:29 smithi191 ceph-mds[124568]: -- 172.21.15.191:0/2295563813 <== mon.1 v2:172.21.15.191:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5649aaacb1e0 con 0x5649aaaa1000 2023-07-14T19:17:30.226 INFO:journalctl@ceph.mds.i.smithi191.stdout:Jul 14 19:17:29 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-mds-i[124544]: starting mds.i at 2023-07-14T19:17:30.226 INFO:journalctl@ceph.mds.l.smithi191.stdout:Jul 14 19:17:29 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-mds-l[123521]: ignoring --setuser ceph since I am not root 2023-07-14T19:17:30.227 INFO:journalctl@ceph.mds.l.smithi191.stdout:Jul 14 19:17:29 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-mds-l[123521]: ignoring --setgroup ceph since I am not root 2023-07-14T19:17:30.227 INFO:journalctl@ceph.mds.l.smithi191.stdout:Jul 14 19:17:29 smithi191 ceph-mds[123546]: -- 172.21.15.191:0/1281526442 <== mon.1 v2:172.21.15.191:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x56496cfa91e0 con 0x56496cf7e800 2023-07-14T19:17:30.227 INFO:journalctl@ceph.mds.l.smithi191.stdout:Jul 14 19:17:29 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-mds-l[123521]: starting mds.l at 2023-07-14T19:17:30.242 INFO:journalctl@ceph.mds.b.smithi130.stdout:Jul 14 19:17:29 smithi130 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-mds-b[122843]: ignoring --setuser ceph since I am not root 2023-07-14T19:17:30.243 INFO:journalctl@ceph.mds.b.smithi130.stdout:Jul 14 19:17:29 smithi130 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-mds-b[122843]: ignoring --setgroup ceph since I am not root 2023-07-14T19:17:30.243 INFO:journalctl@ceph.mds.b.smithi130.stdout:Jul 14 19:17:29 smithi130 ceph-mds[122868]: -- 172.21.15.130:0/2403213131 <== mon.2 v2:172.21.15.130:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x564ee9a111e0 con 0x564ee99e6c00 2023-07-14T19:17:30.243 INFO:journalctl@ceph.mds.b.smithi130.stdout:Jul 14 19:17:29 smithi130 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-mds-b[122843]: starting mds.b at 2023-07-14T19:17:30.245 INFO:journalctl@ceph.mds.e.smithi130.stdout:Jul 14 19:17:29 smithi130 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-mds-e[122160]: ignoring --setuser ceph since I am not root 2023-07-14T19:17:30.246 INFO:journalctl@ceph.mds.e.smithi130.stdout:Jul 14 19:17:29 smithi130 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-mds-e[122160]: ignoring --setgroup ceph since I am not root 2023-07-14T19:17:30.246 INFO:journalctl@ceph.mds.e.smithi130.stdout:Jul 14 19:17:29 smithi130 ceph-mds[122184]: -- 172.21.15.130:0/900072171 <== mon.0 v2:172.21.15.40:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55653f08b1e0 con 0x55653f060800 2023-07-14T19:17:30.246 INFO:journalctl@ceph.mds.e.smithi130.stdout:Jul 14 19:17:29 smithi130 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-mds-e[122160]: starting mds.e at 2023-07-14T19:17:30.247 INFO:journalctl@ceph.mds.h.smithi130.stdout:Jul 14 19:17:29 smithi130 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-mds-h[122505]: ignoring --setuser ceph since I am not root 2023-07-14T19:17:30.247 INFO:journalctl@ceph.mds.h.smithi130.stdout:Jul 14 19:17:29 smithi130 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-mds-h[122505]: ignoring --setgroup ceph since I am not root 2023-07-14T19:17:30.247 INFO:journalctl@ceph.mds.h.smithi130.stdout:Jul 14 19:17:29 smithi130 ceph-mds[122529]: -- 172.21.15.130:0/163178220 <== mon.0 v2:172.21.15.40:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x556dac1ef1e0 con 0x556dac1c5000 2023-07-14T19:17:30.248 INFO:journalctl@ceph.mds.h.smithi130.stdout:Jul 14 19:17:29 smithi130 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-mds-h[122505]: starting mds.h at 2023-07-14T19:17:30.248 INFO:journalctl@ceph.mds.k.smithi130.stdout:Jul 14 19:17:29 smithi130 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-mds-k[121818]: ignoring --setuser ceph since I am not root 2023-07-14T19:17:30.248 INFO:journalctl@ceph.mds.k.smithi130.stdout:Jul 14 19:17:29 smithi130 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-mds-k[121818]: ignoring --setgroup ceph since I am not root 2023-07-14T19:17:30.248 INFO:journalctl@ceph.mds.k.smithi130.stdout:Jul 14 19:17:29 smithi130 ceph-mds[121842]: -- 172.21.15.130:0/271057820 <== mon.2 v2:172.21.15.130:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55f6291731e0 con 0x55f629148c00 2023-07-14T19:17:30.249 INFO:journalctl@ceph.mds.k.smithi130.stdout:Jul 14 19:17:29 smithi130 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-mds-k[121818]: starting mds.k at 2023-07-14T19:17:30.262 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:30.260+0000 7f0a05fd7700 1 -- 172.21.15.40:0/559386693 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f0a00105970 con 0x7f0a000ffd00 2023-07-14T19:17:30.862 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:30.860+0000 7f09fcff9700 1 -- 172.21.15.40:0/559386693 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v29) v1 ==== 104+0+0 (secure 0 0 0) 0x7f09f4099460 con 0x7f0a000ffd00 2023-07-14T19:17:30.869 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:30.865+0000 7f0a05fd7700 1 -- 172.21.15.40:0/559386693 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f09e80a11e0 msgr2=0x7f09e80a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:17:30.869 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:30.865+0000 7f0a05fd7700 1 --2- 172.21.15.40:0/559386693 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f09e80a11e0 0x7f09e80a3690 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f0a000ffb60 tx=0x7f09f000b410 comp rx=0 tx=0).stop 2023-07-14T19:17:30.870 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:30.865+0000 7f0a05fd7700 1 -- 172.21.15.40:0/559386693 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0a000ffd00 msgr2=0x7f0a0019a770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:17:30.870 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:30.865+0000 7f0a05fd7700 1 --2- 172.21.15.40:0/559386693 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0a000ffd00 0x7f0a0019a770 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f09f400d950 tx=0x7f09f400dc60 comp rx=0 tx=0).stop 2023-07-14T19:17:30.870 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:30.865+0000 7f0a05fd7700 1 -- 172.21.15.40:0/559386693 shutdown_connections 2023-07-14T19:17:30.870 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:30.865+0000 7f0a05fd7700 1 --2- 172.21.15.40:0/559386693 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7f09e80a11e0 0x7f09e80a3690 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:30.870 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:30.865+0000 7f0a05fd7700 1 --2- 172.21.15.40:0/559386693 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0a000feb10 0x7f0a00069360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:30.871 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:30.865+0000 7f0a05fd7700 1 --2- 172.21.15.40:0/559386693 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0a00100690 0x7f0a0019cdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:30.871 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:30.865+0000 7f0a05fd7700 1 --2- 172.21.15.40:0/559386693 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0a000ffd00 0x7f0a0019a770 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:30.871 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:30.865+0000 7f0a05fd7700 1 -- 172.21.15.40:0/559386693 >> 172.21.15.40:0/559386693 conn(0x7f0a000fa130 msgr2=0x7f0a000fc310 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:17:30.872 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:30.865+0000 7f0a05fd7700 1 -- 172.21.15.40:0/559386693 shutdown_connections 2023-07-14T19:17:30.872 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:30.865+0000 7f0a05fd7700 1 -- 172.21.15.40:0/559386693 wait complete. 2023-07-14T19:17:30.882 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:17:30 smithi040 ceph-mon[96518]: pgmap v302: 97 pgs: 97 active+clean; 641 KiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 852 B/s wr, 11 op/s 2023-07-14T19:17:30.883 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:17:30 smithi040 ceph-mon[96518]: from='client.? 172.21.15.40:0/3616277774' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2023-07-14T19:17:30.883 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:17:30 smithi040 ceph-mon[96518]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2023-07-14T19:17:30.883 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:17:30 smithi040 ceph-mon[96518]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2023-07-14T19:17:30.883 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:17:30 smithi040 ceph-mon[96518]: osdmap e84: 12 total, 12 up, 12 in 2023-07-14T19:17:30.884 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:17:30 smithi040 ceph-mon[96518]: from='client.? 172.21.15.40:0/3616277774' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2023-07-14T19:17:30.884 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:17:30 smithi040 ceph-mon[96518]: fsmap cephfs:0/5 2 up:standby, 5 failed 2023-07-14T19:17:30.884 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:17:30 smithi040 ceph-mon[96518]: from='client.? 172.21.15.40:0/559386693' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2023-07-14T19:17:30.885 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2023-07-14T19:17:30.972 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:17:30 smithi191 ceph-mon[98311]: pgmap v302: 97 pgs: 97 active+clean; 641 KiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 852 B/s wr, 11 op/s 2023-07-14T19:17:30.972 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:17:30 smithi191 ceph-mon[98311]: from='client.? 172.21.15.40:0/3616277774' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2023-07-14T19:17:30.972 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:17:30 smithi191 ceph-mon[98311]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2023-07-14T19:17:30.973 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:17:30 smithi191 ceph-mon[98311]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2023-07-14T19:17:30.973 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:17:30 smithi191 ceph-mon[98311]: osdmap e84: 12 total, 12 up, 12 in 2023-07-14T19:17:30.973 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:17:30 smithi191 ceph-mon[98311]: from='client.? 172.21.15.40:0/3616277774' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2023-07-14T19:17:30.973 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:17:30 smithi191 ceph-mon[98311]: fsmap cephfs:0/5 2 up:standby, 5 failed 2023-07-14T19:17:30.973 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:17:30 smithi191 ceph-mon[98311]: from='client.? 172.21.15.40:0/559386693' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2023-07-14T19:17:30.992 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:17:30 smithi130 ceph-mon[97832]: pgmap v302: 97 pgs: 97 active+clean; 641 KiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 852 B/s wr, 11 op/s 2023-07-14T19:17:30.992 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:17:30 smithi130 ceph-mon[97832]: from='client.? 172.21.15.40:0/3616277774' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2023-07-14T19:17:30.993 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:17:30 smithi130 ceph-mon[97832]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2023-07-14T19:17:30.993 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:17:30 smithi130 ceph-mon[97832]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2023-07-14T19:17:30.993 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:17:30 smithi130 ceph-mon[97832]: osdmap e84: 12 total, 12 up, 12 in 2023-07-14T19:17:30.993 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:17:30 smithi130 ceph-mon[97832]: from='client.? 172.21.15.40:0/3616277774' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2023-07-14T19:17:30.993 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:17:30 smithi130 ceph-mon[97832]: fsmap cephfs:0/5 2 up:standby, 5 failed 2023-07-14T19:17:30.994 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:17:30 smithi130 ceph-mon[97832]: from='client.? 172.21.15.40:0/559386693' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2023-07-14T19:17:31.024 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:31.020+0000 7ff2a51ef700 1 -- 172.21.15.40:0/3888219315 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff2a01006a0 msgr2=0x7ff2a0106f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:17:31.025 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:31.020+0000 7ff2a51ef700 1 --2- 172.21.15.40:0/3888219315 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff2a01006a0 0x7ff2a0106f60 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7ff29800b0a0 tx=0x7ff29800b3b0 comp rx=0 tx=0).stop 2023-07-14T19:17:31.025 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:31.020+0000 7ff2a51ef700 1 -- 172.21.15.40:0/3888219315 shutdown_connections 2023-07-14T19:17:31.025 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:31.020+0000 7ff2a51ef700 1 --2- 172.21.15.40:0/3888219315 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff2a01006a0 0x7ff2a0106f60 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:31.025 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:31.020+0000 7ff2a51ef700 1 --2- 172.21.15.40:0/3888219315 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff2a00feb20 0x7ff2a00fef30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:31.025 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:31.020+0000 7ff2a51ef700 1 --2- 172.21.15.40:0/3888219315 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff2a00ffd10 0x7ff2a0100160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:31.026 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:31.020+0000 7ff2a51ef700 1 -- 172.21.15.40:0/3888219315 >> 172.21.15.40:0/3888219315 conn(0x7ff2a00fa160 msgr2=0x7ff2a00fc570 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:17:31.026 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:31.020+0000 7ff2a51ef700 1 -- 172.21.15.40:0/3888219315 shutdown_connections 2023-07-14T19:17:31.026 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:31.021+0000 7ff2a51ef700 1 -- 172.21.15.40:0/3888219315 wait complete. 2023-07-14T19:17:31.026 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:31.021+0000 7ff2a51ef700 1 Processor -- start 2023-07-14T19:17:31.026 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:31.021+0000 7ff2a51ef700 1 -- start start 2023-07-14T19:17:31.027 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:31.022+0000 7ff2a51ef700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff2a00feb20 0x7ff2a0194920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:17:31.027 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:31.022+0000 7ff2a51ef700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff2a00ffd10 0x7ff2a0194e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:17:31.027 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:31.022+0000 7ff2a51ef700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff2a01006a0 0x7ff2a019bed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:17:31.027 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:31.022+0000 7ff2a51ef700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2a019c4a0 con 0x7ff2a00feb20 2023-07-14T19:17:31.027 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:31.022+0000 7ff2a51ef700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2a019c5e0 con 0x7ff2a01006a0 2023-07-14T19:17:31.028 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:31.022+0000 7ff2a51ef700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2a019c770 con 0x7ff2a00ffd10 2023-07-14T19:17:31.028 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:31.022+0000 7ff29ed9d700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff2a00feb20 0x7ff2a0194920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:17:31.028 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:31.022+0000 7ff29ed9d700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff2a00feb20 0x7ff2a0194920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:34596/0 (socket says 172.21.15.40:34596) 2023-07-14T19:17:31.028 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:31.022+0000 7ff29ed9d700 1 -- 172.21.15.40:0/543539069 learned_addr learned my addr 172.21.15.40:0/543539069 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-07-14T19:17:31.029 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:31.022+0000 7ff29f59e700 1 --2- 172.21.15.40:0/543539069 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff2a01006a0 0x7ff2a019bed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:17:31.029 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:31.022+0000 7ff29e59c700 1 --2- 172.21.15.40:0/543539069 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff2a00ffd10 0x7ff2a0194e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:17:31.029 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:31.022+0000 7ff29ed9d700 1 -- 172.21.15.40:0/543539069 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff2a00ffd10 msgr2=0x7ff2a0194e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:17:31.029 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:31.022+0000 7ff29ed9d700 1 --2- 172.21.15.40:0/543539069 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff2a00ffd10 0x7ff2a0194e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:31.029 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:31.022+0000 7ff29ed9d700 1 -- 172.21.15.40:0/543539069 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff2a01006a0 msgr2=0x7ff2a019bed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:17:31.030 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:31.022+0000 7ff29ed9d700 1 --2- 172.21.15.40:0/543539069 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff2a01006a0 0x7ff2a019bed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:31.030 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:31.022+0000 7ff29ed9d700 1 -- 172.21.15.40:0/543539069 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff298009d00 con 0x7ff2a00feb20 2023-07-14T19:17:31.030 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:31.022+0000 7ff29ed9d700 1 --2- 172.21.15.40:0/543539069 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff2a00feb20 0x7ff2a0194920 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7ff29000e970 tx=0x7ff29000ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:17:31.030 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:31.022+0000 7ff287fff700 1 -- 172.21.15.40:0/543539069 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff290009790 con 0x7ff2a00feb20 2023-07-14T19:17:31.030 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:31.022+0000 7ff287fff700 1 -- 172.21.15.40:0/543539069 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff2900044e0 con 0x7ff2a00feb20 2023-07-14T19:17:31.031 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:31.022+0000 7ff2a51ef700 1 -- 172.21.15.40:0/543539069 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff2a019ca50 con 0x7ff2a00feb20 2023-07-14T19:17:31.031 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:31.022+0000 7ff287fff700 1 -- 172.21.15.40:0/543539069 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff290003e30 con 0x7ff2a00feb20 2023-07-14T19:17:31.031 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:31.023+0000 7ff2a51ef700 1 -- 172.21.15.40:0/543539069 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff2a019cf60 con 0x7ff2a00feb20 2023-07-14T19:17:31.032 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:31.024+0000 7ff2a51ef700 1 -- 172.21.15.40:0/543539069 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff2a0106030 con 0x7ff2a00feb20 2023-07-14T19:17:31.032 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:31.025+0000 7ff287fff700 1 -- 172.21.15.40:0/543539069 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 19) v1 ==== 135766+0+0 (secure 0 0 0) 0x7ff290004000 con 0x7ff2a00feb20 2023-07-14T19:17:31.032 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:31.025+0000 7ff287fff700 1 --2- 172.21.15.40:0/543539069 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7ff2880a11e0 0x7ff2880a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:17:31.033 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:31.025+0000 7ff287fff700 1 -- 172.21.15.40:0/543539069 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 8699+0+0 (secure 0 0 0) 0x7ff290014070 con 0x7ff2a00feb20 2023-07-14T19:17:31.033 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:31.025+0000 7ff29e59c700 1 --2- 172.21.15.40:0/543539069 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7ff2880a11e0 0x7ff2880a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:17:31.033 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:31.026+0000 7ff29e59c700 1 --2- 172.21.15.40:0/543539069 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7ff2880a11e0 0x7ff2880a3690 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7ff294009bf0 tx=0x7ff294009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:17:31.033 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:31.029+0000 7ff287fff700 1 -- 172.21.15.40:0/543539069 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7ff2900963c0 con 0x7ff2a00feb20 2023-07-14T19:17:31.230 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:31.229+0000 7ff2a51ef700 1 -- 172.21.15.40:0/543539069 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7ff2a002c8e0 con 0x7ff2a00feb20 2023-07-14T19:17:31.862 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:31.861+0000 7ff287fff700 1 -- 172.21.15.40:0/543539069 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v85) v1 ==== 181+0+0 (secure 0 0 0) 0x7ff2900999e0 con 0x7ff2a00feb20 2023-07-14T19:17:31.867 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:31.865+0000 7ff2a51ef700 1 -- 172.21.15.40:0/543539069 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7ff2880a11e0 msgr2=0x7ff2880a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:17:31.867 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:31.865+0000 7ff2a51ef700 1 --2- 172.21.15.40:0/543539069 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7ff2880a11e0 0x7ff2880a3690 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7ff294009bf0 tx=0x7ff294009260 comp rx=0 tx=0).stop 2023-07-14T19:17:31.867 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:31.865+0000 7ff2a51ef700 1 -- 172.21.15.40:0/543539069 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff2a00feb20 msgr2=0x7ff2a0194920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:17:31.867 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:31.865+0000 7ff2a51ef700 1 --2- 172.21.15.40:0/543539069 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff2a00feb20 0x7ff2a0194920 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7ff29000e970 tx=0x7ff29000ed30 comp rx=0 tx=0).stop 2023-07-14T19:17:31.868 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:31.865+0000 7ff2a51ef700 1 -- 172.21.15.40:0/543539069 shutdown_connections 2023-07-14T19:17:31.868 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:31.865+0000 7ff2a51ef700 1 --2- 172.21.15.40:0/543539069 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7ff2880a11e0 0x7ff2880a3690 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:31.868 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:31.865+0000 7ff2a51ef700 1 --2- 172.21.15.40:0/543539069 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff2a01006a0 0x7ff2a019bed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:31.868 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:31.865+0000 7ff2a51ef700 1 --2- 172.21.15.40:0/543539069 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff2a00feb20 0x7ff2a0194920 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:31.869 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:31.865+0000 7ff2a51ef700 1 --2- 172.21.15.40:0/543539069 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7ff2a00ffd10 0x7ff2a0194e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:31.869 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:31.865+0000 7ff2a51ef700 1 -- 172.21.15.40:0/543539069 >> 172.21.15.40:0/543539069 conn(0x7ff2a00fa160 msgr2=0x7ff2a0103170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:17:31.869 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:31.865+0000 7ff2a51ef700 1 -- 172.21.15.40:0/543539069 shutdown_connections 2023-07-14T19:17:31.869 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:31.865+0000 7ff2a51ef700 1 -- 172.21.15.40:0/543539069 wait complete. 2023-07-14T19:17:31.870 INFO:teuthology.orchestra.run.smithi040.stderr:pool 'cephfs_metadata' removed 2023-07-14T19:17:31.881 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:17:31 smithi040 ceph-mon[96518]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2023-07-14T19:17:31.881 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:17:31 smithi040 ceph-mon[96518]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2023-07-14T19:17:31.881 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:17:31 smithi040 ceph-mon[96518]: Cluster is now healthy 2023-07-14T19:17:31.882 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:17:31 smithi040 ceph-mon[96518]: mds.? [v2:172.21.15.40:6838/1141726659,v1:172.21.15.40:6840/1141726659] up:boot 2023-07-14T19:17:31.882 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:17:31 smithi040 ceph-mon[96518]: mds.? [v2:172.21.15.130:6833/1975336000,v1:172.21.15.130:6835/1975336000] up:boot 2023-07-14T19:17:31.882 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:17:31 smithi040 ceph-mon[96518]: mds.? [v2:172.21.15.191:6836/4173485736,v1:172.21.15.191:6838/4173485736] up:boot 2023-07-14T19:17:31.882 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:17:31 smithi040 ceph-mon[96518]: mds.? [v2:172.21.15.130:6836/473173200,v1:172.21.15.130:6837/473173200] up:boot 2023-07-14T19:17:31.883 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:17:31 smithi040 ceph-mon[96518]: mds.? [v2:172.21.15.191:6837/3430935667,v1:172.21.15.191:6839/3430935667] up:boot 2023-07-14T19:17:31.883 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:17:31 smithi040 ceph-mon[96518]: mds.? [v2:172.21.15.40:6839/1971340788,v1:172.21.15.40:6841/1971340788] up:boot 2023-07-14T19:17:31.883 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:17:31 smithi040 ceph-mon[96518]: mds.? [v2:172.21.15.130:6832/1795530013,v1:172.21.15.130:6834/1795530013] up:boot 2023-07-14T19:17:31.883 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:17:31 smithi040 ceph-mon[96518]: mds.? [v2:172.21.15.191:6833/1562905355,v1:172.21.15.191:6835/1562905355] up:boot 2023-07-14T19:17:31.883 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:17:31 smithi040 ceph-mon[96518]: mds.? [v2:172.21.15.191:6832/2640256985,v1:172.21.15.191:6834/2640256985] up:boot 2023-07-14T19:17:31.884 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:17:31 smithi040 ceph-mon[96518]: mds.? [v2:172.21.15.130:6838/1232902755,v1:172.21.15.130:6839/1232902755] up:boot 2023-07-14T19:17:31.884 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:17:31 smithi040 ceph-mon[96518]: from='client.? 172.21.15.40:0/559386693' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2023-07-14T19:17:31.884 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:17:31 smithi040 ceph-mon[96518]: fsmap 12 up:standby 2023-07-14T19:17:31.884 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:17:31 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2023-07-14T19:17:31.884 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:17:31 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2023-07-14T19:17:31.885 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:17:31 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2023-07-14T19:17:31.885 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:17:31 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2023-07-14T19:17:31.885 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:17:31 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2023-07-14T19:17:31.885 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:17:31 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2023-07-14T19:17:31.886 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:17:31 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2023-07-14T19:17:31.886 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:17:31 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2023-07-14T19:17:31.886 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:17:31 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2023-07-14T19:17:31.886 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:17:31 smithi040 ceph-mon[96518]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2023-07-14T19:17:31.886 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:17:31 smithi040 ceph-mon[96518]: from='client.? 172.21.15.40:0/543539069' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2023-07-14T19:17:31.888 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2023-07-14T19:17:31.972 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:17:31 smithi191 ceph-mon[98311]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2023-07-14T19:17:31.972 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:17:31 smithi191 ceph-mon[98311]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2023-07-14T19:17:31.973 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:17:31 smithi191 ceph-mon[98311]: Cluster is now healthy 2023-07-14T19:17:31.973 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:17:31 smithi191 ceph-mon[98311]: mds.? [v2:172.21.15.40:6838/1141726659,v1:172.21.15.40:6840/1141726659] up:boot 2023-07-14T19:17:31.973 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:17:31 smithi191 ceph-mon[98311]: mds.? [v2:172.21.15.130:6833/1975336000,v1:172.21.15.130:6835/1975336000] up:boot 2023-07-14T19:17:31.973 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:17:31 smithi191 ceph-mon[98311]: mds.? [v2:172.21.15.191:6836/4173485736,v1:172.21.15.191:6838/4173485736] up:boot 2023-07-14T19:17:31.973 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:17:31 smithi191 ceph-mon[98311]: mds.? [v2:172.21.15.130:6836/473173200,v1:172.21.15.130:6837/473173200] up:boot 2023-07-14T19:17:31.974 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:17:31 smithi191 ceph-mon[98311]: mds.? [v2:172.21.15.191:6837/3430935667,v1:172.21.15.191:6839/3430935667] up:boot 2023-07-14T19:17:31.974 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:17:31 smithi191 ceph-mon[98311]: mds.? [v2:172.21.15.40:6839/1971340788,v1:172.21.15.40:6841/1971340788] up:boot 2023-07-14T19:17:31.974 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:17:31 smithi191 ceph-mon[98311]: mds.? [v2:172.21.15.130:6832/1795530013,v1:172.21.15.130:6834/1795530013] up:boot 2023-07-14T19:17:31.974 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:17:31 smithi191 ceph-mon[98311]: mds.? [v2:172.21.15.191:6833/1562905355,v1:172.21.15.191:6835/1562905355] up:boot 2023-07-14T19:17:31.974 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:17:31 smithi191 ceph-mon[98311]: mds.? [v2:172.21.15.191:6832/2640256985,v1:172.21.15.191:6834/2640256985] up:boot 2023-07-14T19:17:31.975 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:17:31 smithi191 ceph-mon[98311]: mds.? [v2:172.21.15.130:6838/1232902755,v1:172.21.15.130:6839/1232902755] up:boot 2023-07-14T19:17:31.975 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:17:31 smithi191 ceph-mon[98311]: from='client.? 172.21.15.40:0/559386693' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2023-07-14T19:17:31.975 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:17:31 smithi191 ceph-mon[98311]: fsmap 12 up:standby 2023-07-14T19:17:31.975 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:17:31 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2023-07-14T19:17:31.975 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:17:31 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2023-07-14T19:17:31.976 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:17:31 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2023-07-14T19:17:31.976 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:17:31 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2023-07-14T19:17:31.976 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:17:31 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2023-07-14T19:17:31.976 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:17:31 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2023-07-14T19:17:31.977 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:17:31 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2023-07-14T19:17:31.977 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:17:31 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2023-07-14T19:17:31.977 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:17:31 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2023-07-14T19:17:31.977 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:17:31 smithi191 ceph-mon[98311]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2023-07-14T19:17:31.977 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:17:31 smithi191 ceph-mon[98311]: from='client.? 172.21.15.40:0/543539069' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2023-07-14T19:17:31.992 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:17:31 smithi130 ceph-mon[97832]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2023-07-14T19:17:31.992 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:17:31 smithi130 ceph-mon[97832]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2023-07-14T19:17:31.993 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:17:31 smithi130 ceph-mon[97832]: Cluster is now healthy 2023-07-14T19:17:31.993 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:17:31 smithi130 ceph-mon[97832]: mds.? [v2:172.21.15.40:6838/1141726659,v1:172.21.15.40:6840/1141726659] up:boot 2023-07-14T19:17:31.993 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:17:31 smithi130 ceph-mon[97832]: mds.? [v2:172.21.15.130:6833/1975336000,v1:172.21.15.130:6835/1975336000] up:boot 2023-07-14T19:17:31.993 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:17:31 smithi130 ceph-mon[97832]: mds.? [v2:172.21.15.191:6836/4173485736,v1:172.21.15.191:6838/4173485736] up:boot 2023-07-14T19:17:31.994 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:17:31 smithi130 ceph-mon[97832]: mds.? [v2:172.21.15.130:6836/473173200,v1:172.21.15.130:6837/473173200] up:boot 2023-07-14T19:17:31.994 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:17:31 smithi130 ceph-mon[97832]: mds.? [v2:172.21.15.191:6837/3430935667,v1:172.21.15.191:6839/3430935667] up:boot 2023-07-14T19:17:31.994 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:17:31 smithi130 ceph-mon[97832]: mds.? [v2:172.21.15.40:6839/1971340788,v1:172.21.15.40:6841/1971340788] up:boot 2023-07-14T19:17:31.994 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:17:31 smithi130 ceph-mon[97832]: mds.? [v2:172.21.15.130:6832/1795530013,v1:172.21.15.130:6834/1795530013] up:boot 2023-07-14T19:17:31.994 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:17:31 smithi130 ceph-mon[97832]: mds.? [v2:172.21.15.191:6833/1562905355,v1:172.21.15.191:6835/1562905355] up:boot 2023-07-14T19:17:31.995 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:17:31 smithi130 ceph-mon[97832]: mds.? [v2:172.21.15.191:6832/2640256985,v1:172.21.15.191:6834/2640256985] up:boot 2023-07-14T19:17:31.995 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:17:31 smithi130 ceph-mon[97832]: mds.? [v2:172.21.15.130:6838/1232902755,v1:172.21.15.130:6839/1232902755] up:boot 2023-07-14T19:17:31.995 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:17:31 smithi130 ceph-mon[97832]: from='client.? 172.21.15.40:0/559386693' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2023-07-14T19:17:31.995 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:17:31 smithi130 ceph-mon[97832]: fsmap 12 up:standby 2023-07-14T19:17:31.995 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:17:31 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2023-07-14T19:17:31.996 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:17:31 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2023-07-14T19:17:31.996 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:17:31 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2023-07-14T19:17:31.996 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:17:31 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2023-07-14T19:17:31.996 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:17:31 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2023-07-14T19:17:31.997 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:17:31 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2023-07-14T19:17:31.997 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:17:31 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2023-07-14T19:17:31.997 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:17:31 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2023-07-14T19:17:31.997 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:17:31 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2023-07-14T19:17:31.997 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:17:31 smithi130 ceph-mon[97832]: from='mgr.14152 172.21.15.40:0/2647863621' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2023-07-14T19:17:31.998 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:17:31 smithi130 ceph-mon[97832]: from='client.? 172.21.15.40:0/543539069' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2023-07-14T19:17:32.034 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:32.029+0000 7fc1df46e700 1 -- 172.21.15.40:0/1212347665 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc1d80ffd00 msgr2=0x7fc1d8100150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:17:32.035 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:32.029+0000 7fc1df46e700 1 --2- 172.21.15.40:0/1212347665 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc1d80ffd00 0x7fc1d8100150 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7fc1d4009990 tx=0x7fc1d4009ca0 comp rx=0 tx=0).stop 2023-07-14T19:17:32.035 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:32.029+0000 7fc1df46e700 1 -- 172.21.15.40:0/1212347665 shutdown_connections 2023-07-14T19:17:32.035 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:32.029+0000 7fc1df46e700 1 --2- 172.21.15.40:0/1212347665 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc1d8100690 0x7fc1d8106f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:32.036 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:32.029+0000 7fc1df46e700 1 --2- 172.21.15.40:0/1212347665 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc1d80feb10 0x7fc1d80fef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:32.036 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:32.029+0000 7fc1df46e700 1 --2- 172.21.15.40:0/1212347665 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc1d80ffd00 0x7fc1d8100150 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:32.036 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:32.029+0000 7fc1df46e700 1 -- 172.21.15.40:0/1212347665 >> 172.21.15.40:0/1212347665 conn(0x7fc1d80fa130 msgr2=0x7fc1d80fc560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:17:32.036 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:32.029+0000 7fc1df46e700 1 -- 172.21.15.40:0/1212347665 shutdown_connections 2023-07-14T19:17:32.036 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:32.029+0000 7fc1df46e700 1 -- 172.21.15.40:0/1212347665 wait complete. 2023-07-14T19:17:32.037 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:32.030+0000 7fc1df46e700 1 Processor -- start 2023-07-14T19:17:32.037 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:32.030+0000 7fc1df46e700 1 -- start start 2023-07-14T19:17:32.037 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:32.030+0000 7fc1df46e700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc1d80feb10 0x7fc1d8194910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:17:32.037 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:32.030+0000 7fc1df46e700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc1d80ffd00 0x7fc1d8194e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:17:32.037 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:32.030+0000 7fc1df46e700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc1d8100690 0x7fc1d819bec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:17:32.038 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:32.030+0000 7fc1df46e700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1d819c490 con 0x7fc1d80ffd00 2023-07-14T19:17:32.038 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:32.030+0000 7fc1df46e700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1d819c5d0 con 0x7fc1d8100690 2023-07-14T19:17:32.038 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:32.030+0000 7fc1df46e700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1d819c760 con 0x7fc1d80feb10 2023-07-14T19:17:32.038 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:32.030+0000 7fc1dca09700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc1d80ffd00 0x7fc1d8194e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:17:32.038 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:32.030+0000 7fc1dca09700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc1d80ffd00 0x7fc1d8194e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:34612/0 (socket says 172.21.15.40:34612) 2023-07-14T19:17:32.039 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:32.030+0000 7fc1dca09700 1 -- 172.21.15.40:0/3153940490 learned_addr learned my addr 172.21.15.40:0/3153940490 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-07-14T19:17:32.039 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:32.030+0000 7fc1dd20a700 1 --2- 172.21.15.40:0/3153940490 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc1d80feb10 0x7fc1d8194910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:17:32.039 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:32.030+0000 7fc1dda0b700 1 --2- 172.21.15.40:0/3153940490 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc1d8100690 0x7fc1d819bec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:17:32.039 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:32.030+0000 7fc1dca09700 1 -- 172.21.15.40:0/3153940490 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc1d80feb10 msgr2=0x7fc1d8194910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:17:32.040 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:32.030+0000 7fc1dca09700 1 --2- 172.21.15.40:0/3153940490 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc1d80feb10 0x7fc1d8194910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:32.040 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:32.030+0000 7fc1dca09700 1 -- 172.21.15.40:0/3153940490 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc1d8100690 msgr2=0x7fc1d819bec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:17:32.040 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:32.030+0000 7fc1dca09700 1 --2- 172.21.15.40:0/3153940490 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc1d8100690 0x7fc1d819bec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:32.040 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:32.030+0000 7fc1dca09700 1 -- 172.21.15.40:0/3153940490 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1d40095f0 con 0x7fc1d80ffd00 2023-07-14T19:17:32.041 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:32.030+0000 7fc1dda0b700 1 --2- 172.21.15.40:0/3153940490 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc1d8100690 0x7fc1d819bec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-14T19:17:32.041 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:32.030+0000 7fc1dca09700 1 --2- 172.21.15.40:0/3153940490 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc1d80ffd00 0x7fc1d8194e50 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7fc1d4005910 tx=0x7fc1d4009f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:17:32.041 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:32.030+0000 7fc1ca7fc700 1 -- 172.21.15.40:0/3153940490 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1d401f070 con 0x7fc1d80ffd00 2023-07-14T19:17:32.041 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:32.031+0000 7fc1ca7fc700 1 -- 172.21.15.40:0/3153940490 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc1d401ad40 con 0x7fc1d80ffd00 2023-07-14T19:17:32.041 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:32.031+0000 7fc1df46e700 1 -- 172.21.15.40:0/3153940490 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc1d80677a0 con 0x7fc1d80ffd00 2023-07-14T19:17:32.042 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:32.031+0000 7fc1ca7fc700 1 -- 172.21.15.40:0/3153940490 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1d400f460 con 0x7fc1d80ffd00 2023-07-14T19:17:32.042 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:32.031+0000 7fc1df46e700 1 -- 172.21.15.40:0/3153940490 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc1d8067d20 con 0x7fc1d80ffd00 2023-07-14T19:17:32.042 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:32.032+0000 7fc1df46e700 1 -- 172.21.15.40:0/3153940490 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc1d80673c0 con 0x7fc1d80ffd00 2023-07-14T19:17:32.042 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:32.033+0000 7fc1ca7fc700 1 -- 172.21.15.40:0/3153940490 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 19) v1 ==== 135766+0+0 (secure 0 0 0) 0x7fc1d4023070 con 0x7fc1d80ffd00 2023-07-14T19:17:32.044 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:32.033+0000 7fc1ca7fc700 1 --2- 172.21.15.40:0/3153940490 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7fc1c40a1260 0x7fc1c40a3710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:17:32.044 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:32.033+0000 7fc1dd20a700 1 --2- 172.21.15.40:0/3153940490 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7fc1c40a1260 0x7fc1c40a3710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:17:32.045 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:32.033+0000 7fc1ca7fc700 1 -- 172.21.15.40:0/3153940490 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8145+0+0 (secure 0 0 0) 0x7fc1d40cf8c0 con 0x7fc1d80ffd00 2023-07-14T19:17:32.045 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:32.034+0000 7fc1dd20a700 1 --2- 172.21.15.40:0/3153940490 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7fc1c40a1260 0x7fc1c40a3710 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fc1cc005fd0 tx=0x7fc1cc005dd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:17:32.046 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:32.038+0000 7fc1ca7fc700 1 -- 172.21.15.40:0/3153940490 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fc1d4099040 con 0x7fc1d80ffd00 2023-07-14T19:17:32.236 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:32.235+0000 7fc1df46e700 1 -- 172.21.15.40:0/3153940490 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fc1d819c9e0 con 0x7fc1d80ffd00 2023-07-14T19:17:32.867 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:32.865+0000 7fc1ca7fc700 1 -- 172.21.15.40:0/3153940490 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v86) v1 ==== 169+0+0 (secure 0 0 0) 0x7fc1d409c660 con 0x7fc1d80ffd00 2023-07-14T19:17:32.871 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:32.869+0000 7fc1df46e700 1 -- 172.21.15.40:0/3153940490 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7fc1c40a1260 msgr2=0x7fc1c40a3710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:17:32.872 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:32.869+0000 7fc1df46e700 1 --2- 172.21.15.40:0/3153940490 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7fc1c40a1260 0x7fc1c40a3710 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fc1cc005fd0 tx=0x7fc1cc005dd0 comp rx=0 tx=0).stop 2023-07-14T19:17:32.872 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:32.870+0000 7fc1df46e700 1 -- 172.21.15.40:0/3153940490 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc1d80ffd00 msgr2=0x7fc1d8194e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:17:32.872 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:32.870+0000 7fc1df46e700 1 --2- 172.21.15.40:0/3153940490 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc1d80ffd00 0x7fc1d8194e50 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7fc1d4005910 tx=0x7fc1d4009f90 comp rx=0 tx=0).stop 2023-07-14T19:17:32.874 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:32.870+0000 7fc1df46e700 1 -- 172.21.15.40:0/3153940490 shutdown_connections 2023-07-14T19:17:32.874 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:32.870+0000 7fc1df46e700 1 --2- 172.21.15.40:0/3153940490 >> [v2:172.21.15.40:6800/1693027118,v1:172.21.15.40:6801/1693027118] conn(0x7fc1c40a1260 0x7fc1c40a3710 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:32.875 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:32.870+0000 7fc1df46e700 1 --2- 172.21.15.40:0/3153940490 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc1d80feb10 0x7fc1d8194910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:32.875 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:32.870+0000 7fc1df46e700 1 --2- 172.21.15.40:0/3153940490 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc1d8100690 0x7fc1d819bec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:32.875 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:32.870+0000 7fc1df46e700 1 --2- 172.21.15.40:0/3153940490 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc1d80ffd00 0x7fc1d8194e50 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:32.876 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:32.870+0000 7fc1df46e700 1 -- 172.21.15.40:0/3153940490 >> 172.21.15.40:0/3153940490 conn(0x7fc1d80fa130 msgr2=0x7fc1d81031e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:17:32.876 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:32.871+0000 7fc1df46e700 1 -- 172.21.15.40:0/3153940490 shutdown_connections 2023-07-14T19:17:32.877 INFO:teuthology.orchestra.run.smithi040.stderr:2023-07-14T19:17:32.871+0000 7fc1df46e700 1 -- 172.21.15.40:0/3153940490 wait complete. 2023-07-14T19:17:32.877 INFO:teuthology.orchestra.run.smithi040.stderr:pool 'cephfs_data' removed 2023-07-14T19:17:32.886 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:17:32 smithi040 ceph-mon[96518]: pgmap v304: 97 pgs: 97 active+clean; 645 KiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 2.0 KiB/s wr, 12 op/s 2023-07-14T19:17:32.887 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:17:32 smithi040 ceph-mon[96518]: from='client.? 172.21.15.40:0/543539069' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2023-07-14T19:17:32.887 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:17:32 smithi040 ceph-mon[96518]: osdmap e85: 12 total, 12 up, 12 in 2023-07-14T19:17:32.887 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:17:32 smithi040 ceph-mon[96518]: from='client.? 172.21.15.40:0/3153940490' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2023-07-14T19:17:32.888 DEBUG:teuthology.orchestra.run.smithi040:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2023-07-14T19:17:32.919 DEBUG:teuthology.orchestra.run.smithi130:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2023-07-14T19:17:32.953 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:17:32 smithi130 ceph-mon[97832]: pgmap v304: 97 pgs: 97 active+clean; 645 KiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 2.0 KiB/s wr, 12 op/s 2023-07-14T19:17:32.953 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:17:32 smithi130 ceph-mon[97832]: from='client.? 172.21.15.40:0/543539069' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2023-07-14T19:17:32.954 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:17:32 smithi130 ceph-mon[97832]: osdmap e85: 12 total, 12 up, 12 in 2023-07-14T19:17:32.954 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:17:32 smithi130 ceph-mon[97832]: from='client.? 172.21.15.40:0/3153940490' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2023-07-14T19:17:32.958 DEBUG:teuthology.orchestra.run.smithi191:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2023-07-14T19:17:32.972 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:17:32 smithi191 ceph-mon[98311]: pgmap v304: 97 pgs: 97 active+clean; 645 KiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 2.0 KiB/s wr, 12 op/s 2023-07-14T19:17:32.972 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:17:32 smithi191 ceph-mon[98311]: from='client.? 172.21.15.40:0/543539069' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2023-07-14T19:17:32.973 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:17:32 smithi191 ceph-mon[98311]: osdmap e85: 12 total, 12 up, 12 in 2023-07-14T19:17:32.973 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:17:32 smithi191 ceph-mon[98311]: from='client.? 172.21.15.40:0/3153940490' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2023-07-14T19:17:32.997 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2023-07-14T19:17:32.998 DEBUG:teuthology.orchestra.run.smithi040:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2023-07-14T19:17:33.019 DEBUG:teuthology.orchestra.run.smithi130:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2023-07-14T19:17:33.040 DEBUG:teuthology.orchestra.run.smithi191:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2023-07-14T19:17:33.060 INFO:tasks.cephadm:Stopping all daemons... 2023-07-14T19:17:33.061 INFO:tasks.cephadm.mon.a:Stopping mon.a... 2023-07-14T19:17:33.062 DEBUG:teuthology.orchestra.run.smithi040:> sudo systemctl stop ceph-adc52f04-2279-11ee-9b33-001a4aab830c@mon.a 2023-07-14T19:17:33.221 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:17:33 smithi040 systemd[1]: Stopping Ceph mon.a for adc52f04-2279-11ee-9b33-001a4aab830c... 2023-07-14T19:17:33.721 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:17:33 smithi040 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-mon-a[96494]: 2023-07-14T19:17:33.420+0000 7fc74ba4a700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2023-07-14T19:17:33.721 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:17:33 smithi040 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-mon-a[96494]: 2023-07-14T19:17:33.420+0000 7fc74ba4a700 -1 mon.a@0(leader) e3 *** Got Signal Terminated *** 2023-07-14T19:17:34.221 INFO:journalctl@ceph.mon.a.smithi040.stdout:Jul 14 19:17:33 smithi040 bash[145549]: ceph-adc52f04-2279-11ee-9b33-001a4aab830c-mon-a 2023-07-14T19:17:34.230 DEBUG:teuthology.orchestra.run.smithi040:> sudo pkill -f 'journalctl -f -n 0 -u ceph-adc52f04-2279-11ee-9b33-001a4aab830c@mon.a.service' 2023-07-14T19:17:34.283 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-14T19:17:34.284 INFO:tasks.cephadm.mon.a:Stopped mon.a 2023-07-14T19:17:34.284 INFO:tasks.cephadm.mon.c:Stopping mon.b... 2023-07-14T19:17:34.285 DEBUG:teuthology.orchestra.run.smithi130:> sudo systemctl stop ceph-adc52f04-2279-11ee-9b33-001a4aab830c@mon.b 2023-07-14T19:17:34.742 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:17:34 smithi130 systemd[1]: Stopping Ceph mon.b for adc52f04-2279-11ee-9b33-001a4aab830c... 2023-07-14T19:17:35.222 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:17:34 smithi130 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-mon-b[97808]: 2023-07-14T19:17:34.795+0000 7fbc53ae0700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2023-07-14T19:17:35.223 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:17:34 smithi130 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-mon-b[97808]: 2023-07-14T19:17:34.795+0000 7fbc53ae0700 -1 mon.b@2(peon) e3 *** Got Signal Terminated *** 2023-07-14T19:17:35.492 INFO:journalctl@ceph.mon.b.smithi130.stdout:Jul 14 19:17:35 smithi130 bash[124664]: ceph-adc52f04-2279-11ee-9b33-001a4aab830c-mon-b 2023-07-14T19:17:35.673 DEBUG:teuthology.orchestra.run.smithi130:> sudo pkill -f 'journalctl -f -n 0 -u ceph-adc52f04-2279-11ee-9b33-001a4aab830c@mon.b.service' 2023-07-14T19:17:35.736 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-14T19:17:35.737 INFO:tasks.cephadm.mon.c:Stopped mon.b 2023-07-14T19:17:35.737 INFO:tasks.cephadm.mon.c:Stopping mon.c... 2023-07-14T19:17:35.738 DEBUG:teuthology.orchestra.run.smithi191:> sudo systemctl stop ceph-adc52f04-2279-11ee-9b33-001a4aab830c@mon.c 2023-07-14T19:17:36.087 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:17:35 smithi191 systemd[1]: Stopping Ceph mon.c for adc52f04-2279-11ee-9b33-001a4aab830c... 2023-07-14T19:17:36.361 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:17:36 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-mon-c[98287]: 2023-07-14T19:17:36.085+0000 7fd2a1520700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mon -n mon.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2023-07-14T19:17:36.362 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:17:36 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-mon-c[98287]: 2023-07-14T19:17:36.085+0000 7fd2a1520700 -1 mon.c@1(peon) e3 *** Got Signal Terminated *** 2023-07-14T19:17:36.713 INFO:journalctl@ceph.mon.c.smithi191.stdout:Jul 14 19:17:36 smithi191 bash[126528]: ceph-adc52f04-2279-11ee-9b33-001a4aab830c-mon-c 2023-07-14T19:17:36.719 DEBUG:teuthology.orchestra.run.smithi191:> sudo pkill -f 'journalctl -f -n 0 -u ceph-adc52f04-2279-11ee-9b33-001a4aab830c@mon.c.service' 2023-07-14T19:17:36.777 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-14T19:17:36.778 INFO:tasks.cephadm.mon.c:Stopped mon.c 2023-07-14T19:17:36.778 INFO:tasks.cephadm.mgr.x:Stopping mgr.x... 2023-07-14T19:17:36.779 DEBUG:teuthology.orchestra.run.smithi040:> sudo systemctl stop ceph-adc52f04-2279-11ee-9b33-001a4aab830c@mgr.x 2023-07-14T19:17:37.142 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Jul 14 19:17:36 smithi040 systemd[1]: Stopping Ceph mgr.x for adc52f04-2279-11ee-9b33-001a4aab830c... 2023-07-14T19:17:37.457 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Jul 14 19:17:37 smithi040 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-mgr-x[96762]: 2023-07-14T19:17:37.139+0000 7f302574b700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mgr -n mgr.x -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-07-14T19:17:37.677 DEBUG:teuthology.orchestra.run.smithi040:> sudo pkill -f 'journalctl -f -n 0 -u ceph-adc52f04-2279-11ee-9b33-001a4aab830c@mgr.x.service' 2023-07-14T19:17:37.719 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-14T19:17:37.720 INFO:tasks.cephadm.mgr.x:Stopped mgr.x 2023-07-14T19:17:37.721 INFO:tasks.cephadm.mgr.y:Stopping mgr.y... 2023-07-14T19:17:37.721 DEBUG:teuthology.orchestra.run.smithi130:> sudo systemctl stop ceph-adc52f04-2279-11ee-9b33-001a4aab830c@mgr.y 2023-07-14T19:17:38.148 INFO:journalctl@ceph.mgr.y.smithi130.stdout:Jul 14 19:17:37 smithi130 systemd[1]: Stopping Ceph mgr.y for adc52f04-2279-11ee-9b33-001a4aab830c... 2023-07-14T19:17:38.433 INFO:journalctl@ceph.mgr.y.smithi130.stdout:Jul 14 19:17:38 smithi130 bash[124990]: ceph-adc52f04-2279-11ee-9b33-001a4aab830c-mgr-y 2023-07-14T19:17:38.618 INFO:journalctl@ceph.mgr.y.smithi130.stdout:Jul 14 19:17:38 smithi130 systemd[1]: ceph-adc52f04-2279-11ee-9b33-001a4aab830c@mgr.y.service: Main process exited, code=exited, status=143/n/a 2023-07-14T19:17:38.622 DEBUG:teuthology.orchestra.run.smithi130:> sudo pkill -f 'journalctl -f -n 0 -u ceph-adc52f04-2279-11ee-9b33-001a4aab830c@mgr.y.service' 2023-07-14T19:17:38.661 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-14T19:17:38.661 INFO:tasks.cephadm.mgr.y:Stopped mgr.y 2023-07-14T19:17:38.662 INFO:tasks.cephadm.mgr.z:Stopping mgr.z... 2023-07-14T19:17:38.662 DEBUG:teuthology.orchestra.run.smithi191:> sudo systemctl stop ceph-adc52f04-2279-11ee-9b33-001a4aab830c@mgr.z 2023-07-14T19:17:38.971 INFO:journalctl@ceph.mgr.z.smithi191.stdout:Jul 14 19:17:38 smithi191 systemd[1]: Stopping Ceph mgr.z for adc52f04-2279-11ee-9b33-001a4aab830c... 2023-07-14T19:17:39.721 INFO:journalctl@ceph.mgr.z.smithi191.stdout:Jul 14 19:17:39 smithi191 bash[126692]: ceph-adc52f04-2279-11ee-9b33-001a4aab830c-mgr-z 2023-07-14T19:17:39.722 INFO:journalctl@ceph.mgr.z.smithi191.stdout:Jul 14 19:17:39 smithi191 systemd[1]: ceph-adc52f04-2279-11ee-9b33-001a4aab830c@mgr.z.service: Main process exited, code=exited, status=143/n/a 2023-07-14T19:17:39.753 DEBUG:teuthology.orchestra.run.smithi191:> sudo pkill -f 'journalctl -f -n 0 -u ceph-adc52f04-2279-11ee-9b33-001a4aab830c@mgr.z.service' 2023-07-14T19:17:39.791 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-14T19:17:39.792 INFO:tasks.cephadm.mgr.z:Stopped mgr.z 2023-07-14T19:17:39.792 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2023-07-14T19:17:39.792 DEBUG:teuthology.orchestra.run.smithi040:> sudo systemctl stop ceph-adc52f04-2279-11ee-9b33-001a4aab830c@osd.0 2023-07-14T19:17:40.159 INFO:journalctl@ceph.osd.0.smithi040.stdout:Jul 14 19:17:39 smithi040 systemd[1]: Stopping Ceph osd.0 for adc52f04-2279-11ee-9b33-001a4aab830c... 2023-07-14T19:17:40.471 INFO:journalctl@ceph.osd.0.smithi040.stdout:Jul 14 19:17:40 smithi040 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-0[109451]: 2023-07-14T19:17:40.156+0000 7f7279f8d700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-07-14T19:17:40.471 INFO:journalctl@ceph.osd.0.smithi040.stdout:Jul 14 19:17:40 smithi040 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-0[109451]: 2023-07-14T19:17:40.156+0000 7f7279f8d700 -1 osd.0 86 *** Got signal Terminated *** 2023-07-14T19:17:40.471 INFO:journalctl@ceph.osd.0.smithi040.stdout:Jul 14 19:17:40 smithi040 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-0[109451]: 2023-07-14T19:17:40.156+0000 7f7279f8d700 -1 osd.0 86 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-07-14T19:17:45.930 INFO:journalctl@ceph.osd.0.smithi040.stdout:Jul 14 19:17:45 smithi040 bash[146204]: ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-0 2023-07-14T19:17:47.407 DEBUG:teuthology.orchestra.run.smithi040:> sudo pkill -f 'journalctl -f -n 0 -u ceph-adc52f04-2279-11ee-9b33-001a4aab830c@osd.0.service' 2023-07-14T19:17:47.450 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-14T19:17:47.450 INFO:tasks.cephadm.osd.0:Stopped osd.0 2023-07-14T19:17:47.451 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2023-07-14T19:17:47.451 DEBUG:teuthology.orchestra.run.smithi130:> sudo systemctl stop ceph-adc52f04-2279-11ee-9b33-001a4aab830c@osd.1 2023-07-14T19:17:47.847 INFO:journalctl@ceph.osd.1.smithi130.stdout:Jul 14 19:17:47 smithi130 systemd[1]: Stopping Ceph osd.1 for adc52f04-2279-11ee-9b33-001a4aab830c... 2023-07-14T19:17:48.242 INFO:journalctl@ceph.osd.1.smithi130.stdout:Jul 14 19:17:47 smithi130 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-1[102894]: 2023-07-14T19:17:47.844+0000 7f64d49b3700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-07-14T19:17:48.243 INFO:journalctl@ceph.osd.1.smithi130.stdout:Jul 14 19:17:47 smithi130 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-1[102894]: 2023-07-14T19:17:47.844+0000 7f64d49b3700 -1 osd.1 86 *** Got signal Terminated *** 2023-07-14T19:17:48.243 INFO:journalctl@ceph.osd.1.smithi130.stdout:Jul 14 19:17:47 smithi130 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-1[102894]: 2023-07-14T19:17:47.844+0000 7f64d49b3700 -1 osd.1 86 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-07-14T19:17:53.992 INFO:journalctl@ceph.osd.1.smithi130.stdout:Jul 14 19:17:53 smithi130 bash[125143]: ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-1 2023-07-14T19:17:54.893 DEBUG:teuthology.orchestra.run.smithi130:> sudo pkill -f 'journalctl -f -n 0 -u ceph-adc52f04-2279-11ee-9b33-001a4aab830c@osd.1.service' 2023-07-14T19:17:54.933 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-14T19:17:54.934 INFO:tasks.cephadm.osd.1:Stopped osd.1 2023-07-14T19:17:54.934 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2023-07-14T19:17:54.935 DEBUG:teuthology.orchestra.run.smithi191:> sudo systemctl stop ceph-adc52f04-2279-11ee-9b33-001a4aab830c@osd.2 2023-07-14T19:17:55.308 INFO:journalctl@ceph.osd.2.smithi191.stdout:Jul 14 19:17:54 smithi191 systemd[1]: Stopping Ceph osd.2 for adc52f04-2279-11ee-9b33-001a4aab830c... 2023-07-14T19:17:55.722 INFO:journalctl@ceph.osd.2.smithi191.stdout:Jul 14 19:17:55 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-2[104126]: 2023-07-14T19:17:55.306+0000 7f58e4fc1700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-07-14T19:17:55.722 INFO:journalctl@ceph.osd.2.smithi191.stdout:Jul 14 19:17:55 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-2[104126]: 2023-07-14T19:17:55.306+0000 7f58e4fc1700 -1 osd.2 86 *** Got signal Terminated *** 2023-07-14T19:17:55.722 INFO:journalctl@ceph.osd.2.smithi191.stdout:Jul 14 19:17:55 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-2[104126]: 2023-07-14T19:17:55.306+0000 7f58e4fc1700 -1 osd.2 86 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-07-14T19:18:01.471 INFO:journalctl@ceph.osd.2.smithi191.stdout:Jul 14 19:18:01 smithi191 bash[126836]: ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-2 2023-07-14T19:18:02.494 DEBUG:teuthology.orchestra.run.smithi191:> sudo pkill -f 'journalctl -f -n 0 -u ceph-adc52f04-2279-11ee-9b33-001a4aab830c@osd.2.service' 2023-07-14T19:18:02.535 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-14T19:18:02.536 INFO:tasks.cephadm.osd.2:Stopped osd.2 2023-07-14T19:18:02.536 INFO:tasks.cephadm.osd.3:Stopping osd.3... 2023-07-14T19:18:02.537 DEBUG:teuthology.orchestra.run.smithi040:> sudo systemctl stop ceph-adc52f04-2279-11ee-9b33-001a4aab830c@osd.3 2023-07-14T19:18:02.905 INFO:journalctl@ceph.osd.3.smithi040.stdout:Jul 14 19:18:02 smithi040 systemd[1]: Stopping Ceph osd.3 for adc52f04-2279-11ee-9b33-001a4aab830c... 2023-07-14T19:18:03.221 INFO:journalctl@ceph.osd.3.smithi040.stdout:Jul 14 19:18:02 smithi040 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-3[114874]: 2023-07-14T19:18:02.903+0000 7fd23f27f700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-07-14T19:18:03.221 INFO:journalctl@ceph.osd.3.smithi040.stdout:Jul 14 19:18:02 smithi040 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-3[114874]: 2023-07-14T19:18:02.903+0000 7fd23f27f700 -1 osd.3 86 *** Got signal Terminated *** 2023-07-14T19:18:03.221 INFO:journalctl@ceph.osd.3.smithi040.stdout:Jul 14 19:18:02 smithi040 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-3[114874]: 2023-07-14T19:18:02.903+0000 7fd23f27f700 -1 osd.3 86 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-07-14T19:18:07.128 INFO:journalctl@ceph.osd.5.smithi191.stdout:Jul 14 19:18:06 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-5[109616]: 2023-07-14T19:18:06.751+0000 7fa8a5556700 -1 osd.5 86 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2023-07-14T19:17:40.437352+0000 front 2023-07-14T19:17:40.437431+0000 (oldest deadline 2023-07-14T19:18:06.337579+0000) 2023-07-14T19:18:07.242 INFO:journalctl@ceph.osd.4.smithi130.stdout:Jul 14 19:18:07 smithi130 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-4[108332]: 2023-07-14T19:18:07.094+0000 7fae1d557700 -1 osd.4 86 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2023-07-14T19:17:44.520849+0000 front 2023-07-14T19:17:44.520793+0000 (oldest deadline 2023-07-14T19:18:06.220816+0000) 2023-07-14T19:18:07.243 INFO:journalctl@ceph.osd.10.smithi130.stdout:Jul 14 19:18:06 smithi130 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-10[119160]: 2023-07-14T19:18:06.935+0000 7fa1fdfb9700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2023-07-14T19:17:41.727090+0000 front 2023-07-14T19:17:41.727036+0000 (oldest deadline 2023-07-14T19:18:06.427174+0000) 2023-07-14T19:18:07.472 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jul 14 19:18:07 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-11[120628]: 2023-07-14T19:18:07.126+0000 7f5ef7fea700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2023-07-14T19:17:44.480647+0000 front 2023-07-14T19:17:44.480662+0000 (oldest deadline 2023-07-14T19:18:06.780727+0000) 2023-07-14T19:18:07.613 INFO:journalctl@ceph.osd.9.smithi040.stdout:Jul 14 19:18:07 smithi040 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-9[125889]: 2023-07-14T19:18:07.354+0000 7fa29d625700 -1 osd.9 86 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2023-07-14T19:17:42.530269+0000 front 2023-07-14T19:17:42.530319+0000 (oldest deadline 2023-07-14T19:18:06.531071+0000) 2023-07-14T19:18:07.971 INFO:journalctl@ceph.osd.3.smithi040.stdout:Jul 14 19:18:07 smithi040 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-3[114874]: 2023-07-14T19:18:07.612+0000 7fd239872700 -1 osd.3 86 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2023-07-14T19:17:44.992144+0000 front 2023-07-14T19:17:44.992107+0000 (oldest deadline 2023-07-14T19:18:07.292422+0000) 2023-07-14T19:18:07.971 INFO:journalctl@ceph.osd.5.smithi191.stdout:Jul 14 19:18:07 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-5[109616]: 2023-07-14T19:18:07.716+0000 7fa8a5556700 -1 osd.5 86 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2023-07-14T19:17:40.437352+0000 front 2023-07-14T19:17:40.437431+0000 (oldest deadline 2023-07-14T19:18:06.337579+0000) 2023-07-14T19:18:08.242 INFO:journalctl@ceph.osd.4.smithi130.stdout:Jul 14 19:18:08 smithi130 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-4[108332]: 2023-07-14T19:18:08.122+0000 7fae1d557700 -1 osd.4 86 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2023-07-14T19:17:44.520849+0000 front 2023-07-14T19:17:44.520793+0000 (oldest deadline 2023-07-14T19:18:06.220816+0000) 2023-07-14T19:18:08.243 INFO:journalctl@ceph.osd.10.smithi130.stdout:Jul 14 19:18:07 smithi130 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-10[119160]: 2023-07-14T19:18:07.912+0000 7fa1fdfb9700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2023-07-14T19:17:41.727090+0000 front 2023-07-14T19:17:41.727036+0000 (oldest deadline 2023-07-14T19:18:06.427174+0000) 2023-07-14T19:18:08.472 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jul 14 19:18:08 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-11[120628]: 2023-07-14T19:18:08.101+0000 7f5ef7fea700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2023-07-14T19:17:44.480647+0000 front 2023-07-14T19:17:44.480662+0000 (oldest deadline 2023-07-14T19:18:06.780727+0000) 2023-07-14T19:18:08.721 INFO:journalctl@ceph.osd.6.smithi040.stdout:Jul 14 19:18:08 smithi040 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-6[120375]: 2023-07-14T19:18:08.591+0000 7f5cfd6d6700 -1 osd.6 86 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2023-07-14T19:17:44.759194+0000 front 2023-07-14T19:17:44.759148+0000 (oldest deadline 2023-07-14T19:18:07.659321+0000) 2023-07-14T19:18:08.722 INFO:journalctl@ceph.osd.9.smithi040.stdout:Jul 14 19:18:08 smithi040 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-9[125889]: 2023-07-14T19:18:08.365+0000 7fa29d625700 -1 osd.9 86 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2023-07-14T19:17:42.530269+0000 front 2023-07-14T19:17:42.530319+0000 (oldest deadline 2023-07-14T19:18:06.531071+0000) 2023-07-14T19:18:08.971 INFO:journalctl@ceph.osd.5.smithi191.stdout:Jul 14 19:18:08 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-5[109616]: 2023-07-14T19:18:08.676+0000 7fa8a5556700 -1 osd.5 86 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2023-07-14T19:17:40.437352+0000 front 2023-07-14T19:17:40.437431+0000 (oldest deadline 2023-07-14T19:18:06.337579+0000) 2023-07-14T19:18:09.221 INFO:journalctl@ceph.osd.3.smithi040.stdout:Jul 14 19:18:08 smithi040 bash[146473]: ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-3 2023-07-14T19:18:09.242 INFO:journalctl@ceph.osd.4.smithi130.stdout:Jul 14 19:18:09 smithi130 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-4[108332]: 2023-07-14T19:18:09.089+0000 7fae1d557700 -1 osd.4 86 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2023-07-14T19:17:44.520849+0000 front 2023-07-14T19:17:44.520793+0000 (oldest deadline 2023-07-14T19:18:06.220816+0000) 2023-07-14T19:18:09.242 INFO:journalctl@ceph.osd.10.smithi130.stdout:Jul 14 19:18:08 smithi130 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-10[119160]: 2023-07-14T19:18:08.869+0000 7fa1fdfb9700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2023-07-14T19:17:41.727090+0000 front 2023-07-14T19:17:41.727036+0000 (oldest deadline 2023-07-14T19:18:06.427174+0000) 2023-07-14T19:18:09.471 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jul 14 19:18:09 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-11[120628]: 2023-07-14T19:18:09.113+0000 7f5ef7fea700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2023-07-14T19:17:44.480647+0000 front 2023-07-14T19:17:44.480662+0000 (oldest deadline 2023-07-14T19:18:06.780727+0000) 2023-07-14T19:18:09.721 INFO:journalctl@ceph.osd.6.smithi040.stdout:Jul 14 19:18:09 smithi040 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-6[120375]: 2023-07-14T19:18:09.603+0000 7f5cfd6d6700 -1 osd.6 86 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2023-07-14T19:17:44.759194+0000 front 2023-07-14T19:17:44.759148+0000 (oldest deadline 2023-07-14T19:18:07.659321+0000) 2023-07-14T19:18:09.721 INFO:journalctl@ceph.osd.9.smithi040.stdout:Jul 14 19:18:09 smithi040 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-9[125889]: 2023-07-14T19:18:09.369+0000 7fa29d625700 -1 osd.9 86 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2023-07-14T19:17:42.530269+0000 front 2023-07-14T19:17:42.530319+0000 (oldest deadline 2023-07-14T19:18:06.531071+0000) 2023-07-14T19:18:09.972 INFO:journalctl@ceph.osd.5.smithi191.stdout:Jul 14 19:18:09 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-5[109616]: 2023-07-14T19:18:09.654+0000 7fa8a5556700 -1 osd.5 86 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2023-07-14T19:17:40.437352+0000 front 2023-07-14T19:17:40.437431+0000 (oldest deadline 2023-07-14T19:18:06.337579+0000) 2023-07-14T19:18:10.113 INFO:journalctl@ceph.osd.10.smithi130.stdout:Jul 14 19:18:09 smithi130 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-10[119160]: 2023-07-14T19:18:09.819+0000 7fa1fdfb9700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2023-07-14T19:17:41.727090+0000 front 2023-07-14T19:17:41.727036+0000 (oldest deadline 2023-07-14T19:18:06.427174+0000) 2023-07-14T19:18:10.159 DEBUG:teuthology.orchestra.run.smithi040:> sudo pkill -f 'journalctl -f -n 0 -u ceph-adc52f04-2279-11ee-9b33-001a4aab830c@osd.3.service' 2023-07-14T19:18:10.215 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-14T19:18:10.215 INFO:tasks.cephadm.osd.3:Stopped osd.3 2023-07-14T19:18:10.216 INFO:tasks.cephadm.osd.4:Stopping osd.4... 2023-07-14T19:18:10.216 DEBUG:teuthology.orchestra.run.smithi130:> sudo systemctl stop ceph-adc52f04-2279-11ee-9b33-001a4aab830c@osd.4 2023-07-14T19:18:10.471 INFO:journalctl@ceph.osd.9.smithi040.stdout:Jul 14 19:18:10 smithi040 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-9[125889]: 2023-07-14T19:18:10.378+0000 7fa29d625700 -1 osd.9 86 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2023-07-14T19:17:42.530269+0000 front 2023-07-14T19:17:42.530319+0000 (oldest deadline 2023-07-14T19:18:06.531071+0000) 2023-07-14T19:18:10.471 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jul 14 19:18:10 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-11[120628]: 2023-07-14T19:18:10.100+0000 7f5ef7fea700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2023-07-14T19:17:44.480647+0000 front 2023-07-14T19:17:44.480662+0000 (oldest deadline 2023-07-14T19:18:06.780727+0000) 2023-07-14T19:18:10.492 INFO:journalctl@ceph.osd.4.smithi130.stdout:Jul 14 19:18:10 smithi130 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-4[108332]: 2023-07-14T19:18:10.112+0000 7fae1d557700 -1 osd.4 86 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2023-07-14T19:17:44.520849+0000 front 2023-07-14T19:17:44.520793+0000 (oldest deadline 2023-07-14T19:18:06.220816+0000) 2023-07-14T19:18:10.492 INFO:journalctl@ceph.osd.4.smithi130.stdout:Jul 14 19:18:10 smithi130 systemd[1]: Stopping Ceph osd.4 for adc52f04-2279-11ee-9b33-001a4aab830c... 2023-07-14T19:18:10.833 INFO:journalctl@ceph.osd.4.smithi130.stdout:Jul 14 19:18:10 smithi130 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-4[108332]: 2023-07-14T19:18:10.565+0000 7fae21f62700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-07-14T19:18:10.833 INFO:journalctl@ceph.osd.4.smithi130.stdout:Jul 14 19:18:10 smithi130 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-4[108332]: 2023-07-14T19:18:10.565+0000 7fae21f62700 -1 osd.4 86 *** Got signal Terminated *** 2023-07-14T19:18:10.834 INFO:journalctl@ceph.osd.4.smithi130.stdout:Jul 14 19:18:10 smithi130 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-4[108332]: 2023-07-14T19:18:10.565+0000 7fae21f62700 -1 osd.4 86 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-07-14T19:18:10.970 INFO:journalctl@ceph.osd.6.smithi040.stdout:Jul 14 19:18:10 smithi040 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-6[120375]: 2023-07-14T19:18:10.606+0000 7f5cfd6d6700 -1 osd.6 86 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2023-07-14T19:17:44.759194+0000 front 2023-07-14T19:17:44.759148+0000 (oldest deadline 2023-07-14T19:18:07.659321+0000) 2023-07-14T19:18:10.971 INFO:journalctl@ceph.osd.5.smithi191.stdout:Jul 14 19:18:10 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-5[109616]: 2023-07-14T19:18:10.617+0000 7fa8a5556700 -1 osd.5 86 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2023-07-14T19:17:40.437352+0000 front 2023-07-14T19:17:40.437431+0000 (oldest deadline 2023-07-14T19:18:06.337579+0000) 2023-07-14T19:18:11.242 INFO:journalctl@ceph.osd.4.smithi130.stdout:Jul 14 19:18:11 smithi130 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-4[108332]: 2023-07-14T19:18:11.077+0000 7fae1d557700 -1 osd.4 86 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2023-07-14T19:17:44.520849+0000 front 2023-07-14T19:17:44.520793+0000 (oldest deadline 2023-07-14T19:18:06.220816+0000) 2023-07-14T19:18:11.242 INFO:journalctl@ceph.osd.10.smithi130.stdout:Jul 14 19:18:10 smithi130 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-10[119160]: 2023-07-14T19:18:10.831+0000 7fa1fdfb9700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2023-07-14T19:17:41.727090+0000 front 2023-07-14T19:17:41.727036+0000 (oldest deadline 2023-07-14T19:18:06.427174+0000) 2023-07-14T19:18:11.471 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jul 14 19:18:11 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-11[120628]: 2023-07-14T19:18:11.112+0000 7f5ef7fea700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2023-07-14T19:17:44.480647+0000 front 2023-07-14T19:17:44.480662+0000 (oldest deadline 2023-07-14T19:18:06.780727+0000) 2023-07-14T19:18:11.720 INFO:journalctl@ceph.osd.6.smithi040.stdout:Jul 14 19:18:11 smithi040 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-6[120375]: 2023-07-14T19:18:11.602+0000 7f5cfd6d6700 -1 osd.6 86 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2023-07-14T19:17:44.759194+0000 front 2023-07-14T19:17:44.759148+0000 (oldest deadline 2023-07-14T19:18:07.659321+0000) 2023-07-14T19:18:11.721 INFO:journalctl@ceph.osd.9.smithi040.stdout:Jul 14 19:18:11 smithi040 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-9[125889]: 2023-07-14T19:18:11.364+0000 7fa29d625700 -1 osd.9 86 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2023-07-14T19:17:42.530269+0000 front 2023-07-14T19:17:42.530319+0000 (oldest deadline 2023-07-14T19:18:06.531071+0000) 2023-07-14T19:18:11.972 INFO:journalctl@ceph.osd.5.smithi191.stdout:Jul 14 19:18:11 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-5[109616]: 2023-07-14T19:18:11.585+0000 7fa8a5556700 -1 osd.5 86 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2023-07-14T19:17:40.437352+0000 front 2023-07-14T19:17:40.437431+0000 (oldest deadline 2023-07-14T19:18:06.337579+0000) 2023-07-14T19:18:12.069 INFO:journalctl@ceph.osd.10.smithi130.stdout:Jul 14 19:18:11 smithi130 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-10[119160]: 2023-07-14T19:18:11.790+0000 7fa1fdfb9700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2023-07-14T19:17:41.727090+0000 front 2023-07-14T19:17:41.727036+0000 (oldest deadline 2023-07-14T19:18:06.427174+0000) 2023-07-14T19:18:12.472 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jul 14 19:18:12 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-11[120628]: 2023-07-14T19:18:12.097+0000 7f5ef7fea700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2023-07-14T19:17:44.480647+0000 front 2023-07-14T19:17:44.480662+0000 (oldest deadline 2023-07-14T19:18:06.780727+0000) 2023-07-14T19:18:12.492 INFO:journalctl@ceph.osd.4.smithi130.stdout:Jul 14 19:18:12 smithi130 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-4[108332]: 2023-07-14T19:18:12.067+0000 7fae1d557700 -1 osd.4 86 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2023-07-14T19:17:44.520849+0000 front 2023-07-14T19:17:44.520793+0000 (oldest deadline 2023-07-14T19:18:06.220816+0000) 2023-07-14T19:18:12.721 INFO:journalctl@ceph.osd.6.smithi040.stdout:Jul 14 19:18:12 smithi040 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-6[120375]: 2023-07-14T19:18:12.641+0000 7f5cfd6d6700 -1 osd.6 86 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2023-07-14T19:17:44.759194+0000 front 2023-07-14T19:17:44.759148+0000 (oldest deadline 2023-07-14T19:18:07.659321+0000) 2023-07-14T19:18:12.721 INFO:journalctl@ceph.osd.9.smithi040.stdout:Jul 14 19:18:12 smithi040 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-9[125889]: 2023-07-14T19:18:12.405+0000 7fa29d625700 -1 osd.9 86 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2023-07-14T19:17:42.530269+0000 front 2023-07-14T19:17:42.530319+0000 (oldest deadline 2023-07-14T19:18:06.531071+0000) 2023-07-14T19:18:12.971 INFO:journalctl@ceph.osd.5.smithi191.stdout:Jul 14 19:18:12 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-5[109616]: 2023-07-14T19:18:12.600+0000 7fa8a5556700 -1 osd.5 86 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2023-07-14T19:17:40.437352+0000 front 2023-07-14T19:17:40.437431+0000 (oldest deadline 2023-07-14T19:18:06.337579+0000) 2023-07-14T19:18:13.083 INFO:journalctl@ceph.osd.10.smithi130.stdout:Jul 14 19:18:12 smithi130 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-10[119160]: 2023-07-14T19:18:12.764+0000 7fa1fdfb9700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2023-07-14T19:17:41.727090+0000 front 2023-07-14T19:17:41.727036+0000 (oldest deadline 2023-07-14T19:18:06.427174+0000) 2023-07-14T19:18:13.472 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jul 14 19:18:13 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-11[120628]: 2023-07-14T19:18:13.074+0000 7f5ef7fea700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2023-07-14T19:17:44.480647+0000 front 2023-07-14T19:17:44.480662+0000 (oldest deadline 2023-07-14T19:18:06.780727+0000) 2023-07-14T19:18:13.492 INFO:journalctl@ceph.osd.4.smithi130.stdout:Jul 14 19:18:13 smithi130 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-4[108332]: 2023-07-14T19:18:13.082+0000 7fae1d557700 -1 osd.4 86 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2023-07-14T19:17:44.520849+0000 front 2023-07-14T19:17:44.520793+0000 (oldest deadline 2023-07-14T19:18:06.220816+0000) 2023-07-14T19:18:13.685 INFO:journalctl@ceph.osd.9.smithi040.stdout:Jul 14 19:18:13 smithi040 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-9[125889]: 2023-07-14T19:18:13.398+0000 7fa29d625700 -1 osd.9 86 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2023-07-14T19:17:42.530269+0000 front 2023-07-14T19:17:42.530319+0000 (oldest deadline 2023-07-14T19:18:06.531071+0000) 2023-07-14T19:18:13.971 INFO:journalctl@ceph.osd.6.smithi040.stdout:Jul 14 19:18:13 smithi040 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-6[120375]: 2023-07-14T19:18:13.684+0000 7f5cfd6d6700 -1 osd.6 86 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2023-07-14T19:17:44.759194+0000 front 2023-07-14T19:17:44.759148+0000 (oldest deadline 2023-07-14T19:18:07.659321+0000) 2023-07-14T19:18:13.971 INFO:journalctl@ceph.osd.5.smithi191.stdout:Jul 14 19:18:13 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-5[109616]: 2023-07-14T19:18:13.559+0000 7fa8a5556700 -1 osd.5 86 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2023-07-14T19:17:40.437352+0000 front 2023-07-14T19:17:40.437431+0000 (oldest deadline 2023-07-14T19:18:06.337579+0000) 2023-07-14T19:18:14.073 INFO:journalctl@ceph.osd.10.smithi130.stdout:Jul 14 19:18:13 smithi130 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-10[119160]: 2023-07-14T19:18:13.765+0000 7fa1fdfb9700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2023-07-14T19:17:41.727090+0000 front 2023-07-14T19:17:41.727036+0000 (oldest deadline 2023-07-14T19:18:06.427174+0000) 2023-07-14T19:18:14.471 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jul 14 19:18:14 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-11[120628]: 2023-07-14T19:18:14.074+0000 7f5ef7fea700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2023-07-14T19:17:44.480647+0000 front 2023-07-14T19:17:44.480662+0000 (oldest deadline 2023-07-14T19:18:06.780727+0000) 2023-07-14T19:18:14.492 INFO:journalctl@ceph.osd.4.smithi130.stdout:Jul 14 19:18:14 smithi130 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-4[108332]: 2023-07-14T19:18:14.071+0000 7fae1d557700 -1 osd.4 86 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2023-07-14T19:17:44.520849+0000 front 2023-07-14T19:17:44.520793+0000 (oldest deadline 2023-07-14T19:18:06.220816+0000) 2023-07-14T19:18:14.721 INFO:journalctl@ceph.osd.6.smithi040.stdout:Jul 14 19:18:14 smithi040 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-6[120375]: 2023-07-14T19:18:14.673+0000 7f5cfd6d6700 -1 osd.6 86 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2023-07-14T19:17:44.759194+0000 front 2023-07-14T19:17:44.759148+0000 (oldest deadline 2023-07-14T19:18:07.659321+0000) 2023-07-14T19:18:14.722 INFO:journalctl@ceph.osd.9.smithi040.stdout:Jul 14 19:18:14 smithi040 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-9[125889]: 2023-07-14T19:18:14.445+0000 7fa29d625700 -1 osd.9 86 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2023-07-14T19:17:42.530269+0000 front 2023-07-14T19:17:42.530319+0000 (oldest deadline 2023-07-14T19:18:06.531071+0000) 2023-07-14T19:18:14.971 INFO:journalctl@ceph.osd.5.smithi191.stdout:Jul 14 19:18:14 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-5[109616]: 2023-07-14T19:18:14.597+0000 7fa8a5556700 -1 osd.5 86 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2023-07-14T19:17:40.437352+0000 front 2023-07-14T19:17:40.437431+0000 (oldest deadline 2023-07-14T19:18:06.337579+0000) 2023-07-14T19:18:14.972 INFO:journalctl@ceph.osd.5.smithi191.stdout:Jul 14 19:18:14 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-5[109616]: 2023-07-14T19:18:14.597+0000 7fa8a5556700 -1 osd.5 86 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-07-14T19:17:51.039013+0000 front 2023-07-14T19:17:51.038875+0000 (oldest deadline 2023-07-14T19:18:13.938865+0000) 2023-07-14T19:18:14.992 INFO:journalctl@ceph.osd.10.smithi130.stdout:Jul 14 19:18:14 smithi130 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-10[119160]: 2023-07-14T19:18:14.737+0000 7fa1fdfb9700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2023-07-14T19:17:41.727090+0000 front 2023-07-14T19:17:41.727036+0000 (oldest deadline 2023-07-14T19:18:06.427174+0000) 2023-07-14T19:18:14.992 INFO:journalctl@ceph.osd.10.smithi130.stdout:Jul 14 19:18:14 smithi130 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-10[119160]: 2023-07-14T19:18:14.737+0000 7fa1fdfb9700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-07-14T19:17:52.828799+0000 front 2023-07-14T19:17:52.828840+0000 (oldest deadline 2023-07-14T19:18:14.528908+0000) 2023-07-14T19:18:15.452 INFO:journalctl@ceph.osd.4.smithi130.stdout:Jul 14 19:18:15 smithi130 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-4[108332]: 2023-07-14T19:18:15.041+0000 7fae1d557700 -1 osd.4 86 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2023-07-14T19:17:44.520849+0000 front 2023-07-14T19:17:44.520793+0000 (oldest deadline 2023-07-14T19:18:06.220816+0000) 2023-07-14T19:18:15.471 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jul 14 19:18:15 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-11[120628]: 2023-07-14T19:18:15.054+0000 7f5ef7fea700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2023-07-14T19:17:44.480647+0000 front 2023-07-14T19:17:44.480662+0000 (oldest deadline 2023-07-14T19:18:06.780727+0000) 2023-07-14T19:18:15.472 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jul 14 19:18:15 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-11[120628]: 2023-07-14T19:18:15.054+0000 7f5ef7fea700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-07-14T19:17:50.782602+0000 front 2023-07-14T19:17:50.782583+0000 (oldest deadline 2023-07-14T19:18:14.282678+0000) 2023-07-14T19:18:15.711 INFO:journalctl@ceph.osd.7.smithi130.stdout:Jul 14 19:18:15 smithi130 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-7[113739]: 2023-07-14T19:18:15.451+0000 7f621462a700 -1 osd.7 86 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-07-14T19:17:48.674688+0000 front 2023-07-14T19:17:48.674657+0000 (oldest deadline 2023-07-14T19:18:14.574846+0000) 2023-07-14T19:18:15.720 INFO:journalctl@ceph.osd.6.smithi040.stdout:Jul 14 19:18:15 smithi040 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-6[120375]: 2023-07-14T19:18:15.634+0000 7f5cfd6d6700 -1 osd.6 86 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2023-07-14T19:17:44.759194+0000 front 2023-07-14T19:17:44.759148+0000 (oldest deadline 2023-07-14T19:18:07.659321+0000) 2023-07-14T19:18:15.721 INFO:journalctl@ceph.osd.6.smithi040.stdout:Jul 14 19:18:15 smithi040 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-6[120375]: 2023-07-14T19:18:15.634+0000 7f5cfd6d6700 -1 osd.6 86 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-07-14T19:17:51.160890+0000 front 2023-07-14T19:17:51.160697+0000 (oldest deadline 2023-07-14T19:18:15.260756+0000) 2023-07-14T19:18:15.721 INFO:journalctl@ceph.osd.9.smithi040.stdout:Jul 14 19:18:15 smithi040 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-9[125889]: 2023-07-14T19:18:15.400+0000 7fa29d625700 -1 osd.9 86 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2023-07-14T19:17:42.530269+0000 front 2023-07-14T19:17:42.530319+0000 (oldest deadline 2023-07-14T19:18:06.531071+0000) 2023-07-14T19:18:15.972 INFO:journalctl@ceph.osd.5.smithi191.stdout:Jul 14 19:18:15 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-5[109616]: 2023-07-14T19:18:15.550+0000 7fa8a5556700 -1 osd.5 86 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2023-07-14T19:17:40.437352+0000 front 2023-07-14T19:17:40.437431+0000 (oldest deadline 2023-07-14T19:18:06.337579+0000) 2023-07-14T19:18:15.972 INFO:journalctl@ceph.osd.5.smithi191.stdout:Jul 14 19:18:15 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-5[109616]: 2023-07-14T19:18:15.550+0000 7fa8a5556700 -1 osd.5 86 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-07-14T19:17:51.039013+0000 front 2023-07-14T19:17:51.038875+0000 (oldest deadline 2023-07-14T19:18:13.938865+0000) 2023-07-14T19:18:15.992 INFO:journalctl@ceph.osd.10.smithi130.stdout:Jul 14 19:18:15 smithi130 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-10[119160]: 2023-07-14T19:18:15.710+0000 7fa1fdfb9700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2023-07-14T19:17:41.727090+0000 front 2023-07-14T19:17:41.727036+0000 (oldest deadline 2023-07-14T19:18:06.427174+0000) 2023-07-14T19:18:15.993 INFO:journalctl@ceph.osd.10.smithi130.stdout:Jul 14 19:18:15 smithi130 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-10[119160]: 2023-07-14T19:18:15.710+0000 7fa1fdfb9700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-07-14T19:17:52.828799+0000 front 2023-07-14T19:17:52.828840+0000 (oldest deadline 2023-07-14T19:18:14.528908+0000) 2023-07-14T19:18:16.472 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jul 14 19:18:16 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-11[120628]: 2023-07-14T19:18:16.059+0000 7f5ef7fea700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2023-07-14T19:17:44.480647+0000 front 2023-07-14T19:17:44.480662+0000 (oldest deadline 2023-07-14T19:18:06.780727+0000) 2023-07-14T19:18:16.472 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jul 14 19:18:16 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-11[120628]: 2023-07-14T19:18:16.059+0000 7f5ef7fea700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-07-14T19:17:50.782602+0000 front 2023-07-14T19:17:50.782583+0000 (oldest deadline 2023-07-14T19:18:14.282678+0000) 2023-07-14T19:18:16.486 INFO:journalctl@ceph.osd.4.smithi130.stdout:Jul 14 19:18:16 smithi130 bash[125410]: ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-4 2023-07-14T19:18:16.721 INFO:journalctl@ceph.osd.6.smithi040.stdout:Jul 14 19:18:16 smithi040 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-6[120375]: 2023-07-14T19:18:16.622+0000 7f5cfd6d6700 -1 osd.6 86 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2023-07-14T19:17:44.759194+0000 front 2023-07-14T19:17:44.759148+0000 (oldest deadline 2023-07-14T19:18:07.659321+0000) 2023-07-14T19:18:16.721 INFO:journalctl@ceph.osd.6.smithi040.stdout:Jul 14 19:18:16 smithi040 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-6[120375]: 2023-07-14T19:18:16.622+0000 7f5cfd6d6700 -1 osd.6 86 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-07-14T19:17:51.160890+0000 front 2023-07-14T19:17:51.160697+0000 (oldest deadline 2023-07-14T19:18:15.260756+0000) 2023-07-14T19:18:16.722 INFO:journalctl@ceph.osd.9.smithi040.stdout:Jul 14 19:18:16 smithi040 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-9[125889]: 2023-07-14T19:18:16.428+0000 7fa29d625700 -1 osd.9 86 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2023-07-14T19:17:42.530269+0000 front 2023-07-14T19:17:42.530319+0000 (oldest deadline 2023-07-14T19:18:06.531071+0000) 2023-07-14T19:18:16.742 INFO:journalctl@ceph.osd.7.smithi130.stdout:Jul 14 19:18:16 smithi130 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-7[113739]: 2023-07-14T19:18:16.484+0000 7f621462a700 -1 osd.7 86 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-07-14T19:17:48.674688+0000 front 2023-07-14T19:17:48.674657+0000 (oldest deadline 2023-07-14T19:18:14.574846+0000) 2023-07-14T19:18:16.742 INFO:journalctl@ceph.osd.10.smithi130.stdout:Jul 14 19:18:16 smithi130 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-10[119160]: 2023-07-14T19:18:16.723+0000 7fa1fdfb9700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2023-07-14T19:17:41.727090+0000 front 2023-07-14T19:17:41.727036+0000 (oldest deadline 2023-07-14T19:18:06.427174+0000) 2023-07-14T19:18:16.743 INFO:journalctl@ceph.osd.10.smithi130.stdout:Jul 14 19:18:16 smithi130 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-10[119160]: 2023-07-14T19:18:16.723+0000 7fa1fdfb9700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-07-14T19:17:52.828799+0000 front 2023-07-14T19:17:52.828840+0000 (oldest deadline 2023-07-14T19:18:14.528908+0000) 2023-07-14T19:18:16.972 INFO:journalctl@ceph.osd.5.smithi191.stdout:Jul 14 19:18:16 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-5[109616]: 2023-07-14T19:18:16.504+0000 7fa8a5556700 -1 osd.5 86 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2023-07-14T19:17:40.437352+0000 front 2023-07-14T19:17:40.437431+0000 (oldest deadline 2023-07-14T19:18:06.337579+0000) 2023-07-14T19:18:16.972 INFO:journalctl@ceph.osd.5.smithi191.stdout:Jul 14 19:18:16 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-5[109616]: 2023-07-14T19:18:16.504+0000 7fa8a5556700 -1 osd.5 86 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-07-14T19:17:51.039013+0000 front 2023-07-14T19:17:51.038875+0000 (oldest deadline 2023-07-14T19:18:13.938865+0000) 2023-07-14T19:18:17.415 DEBUG:teuthology.orchestra.run.smithi130:> sudo pkill -f 'journalctl -f -n 0 -u ceph-adc52f04-2279-11ee-9b33-001a4aab830c@osd.4.service' 2023-07-14T19:18:17.456 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-14T19:18:17.456 INFO:tasks.cephadm.osd.4:Stopped osd.4 2023-07-14T19:18:17.457 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2023-07-14T19:18:17.457 DEBUG:teuthology.orchestra.run.smithi191:> sudo systemctl stop ceph-adc52f04-2279-11ee-9b33-001a4aab830c@osd.5 2023-07-14T19:18:17.471 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jul 14 19:18:17 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-11[120628]: 2023-07-14T19:18:17.095+0000 7f5ef7fea700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2023-07-14T19:17:44.480647+0000 front 2023-07-14T19:17:44.480662+0000 (oldest deadline 2023-07-14T19:18:06.780727+0000) 2023-07-14T19:18:17.472 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jul 14 19:18:17 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-11[120628]: 2023-07-14T19:18:17.095+0000 7f5ef7fea700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-07-14T19:17:50.782602+0000 front 2023-07-14T19:17:50.782583+0000 (oldest deadline 2023-07-14T19:18:14.282678+0000) 2023-07-14T19:18:17.651 INFO:journalctl@ceph.osd.9.smithi040.stdout:Jul 14 19:18:17 smithi040 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-9[125889]: 2023-07-14T19:18:17.393+0000 7fa29d625700 -1 osd.9 86 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2023-07-14T19:17:42.530269+0000 front 2023-07-14T19:17:42.530319+0000 (oldest deadline 2023-07-14T19:18:06.531071+0000) 2023-07-14T19:18:17.651 INFO:journalctl@ceph.osd.9.smithi040.stdout:Jul 14 19:18:17 smithi040 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-9[125889]: 2023-07-14T19:18:17.393+0000 7fa29d625700 -1 osd.9 86 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-07-14T19:17:50.632219+0000 front 2023-07-14T19:17:50.632182+0000 (oldest deadline 2023-07-14T19:18:16.532211+0000) 2023-07-14T19:18:17.742 INFO:journalctl@ceph.osd.7.smithi130.stdout:Jul 14 19:18:17 smithi130 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-7[113739]: 2023-07-14T19:18:17.514+0000 7f621462a700 -1 osd.7 86 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-07-14T19:17:48.674688+0000 front 2023-07-14T19:17:48.674657+0000 (oldest deadline 2023-07-14T19:18:14.574846+0000) 2023-07-14T19:18:17.812 INFO:journalctl@ceph.osd.5.smithi191.stdout:Jul 14 19:18:17 smithi191 systemd[1]: Stopping Ceph osd.5 for adc52f04-2279-11ee-9b33-001a4aab830c... 2023-07-14T19:18:17.812 INFO:journalctl@ceph.osd.5.smithi191.stdout:Jul 14 19:18:17 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-5[109616]: 2023-07-14T19:18:17.526+0000 7fa8a5556700 -1 osd.5 86 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2023-07-14T19:17:40.437352+0000 front 2023-07-14T19:17:40.437431+0000 (oldest deadline 2023-07-14T19:18:06.337579+0000) 2023-07-14T19:18:17.812 INFO:journalctl@ceph.osd.5.smithi191.stdout:Jul 14 19:18:17 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-5[109616]: 2023-07-14T19:18:17.526+0000 7fa8a5556700 -1 osd.5 86 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-07-14T19:17:51.039013+0000 front 2023-07-14T19:17:51.038875+0000 (oldest deadline 2023-07-14T19:18:13.938865+0000) 2023-07-14T19:18:17.813 INFO:journalctl@ceph.osd.5.smithi191.stdout:Jul 14 19:18:17 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-5[109616]: 2023-07-14T19:18:17.810+0000 7fa8a9f61700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-07-14T19:18:17.970 INFO:journalctl@ceph.osd.6.smithi040.stdout:Jul 14 19:18:17 smithi040 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-6[120375]: 2023-07-14T19:18:17.650+0000 7f5cfd6d6700 -1 osd.6 86 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2023-07-14T19:17:44.759194+0000 front 2023-07-14T19:17:44.759148+0000 (oldest deadline 2023-07-14T19:18:07.659321+0000) 2023-07-14T19:18:17.971 INFO:journalctl@ceph.osd.6.smithi040.stdout:Jul 14 19:18:17 smithi040 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-6[120375]: 2023-07-14T19:18:17.650+0000 7f5cfd6d6700 -1 osd.6 86 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-07-14T19:17:51.160890+0000 front 2023-07-14T19:17:51.160697+0000 (oldest deadline 2023-07-14T19:18:15.260756+0000) 2023-07-14T19:18:18.221 INFO:journalctl@ceph.osd.5.smithi191.stdout:Jul 14 19:18:17 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-5[109616]: 2023-07-14T19:18:17.810+0000 7fa8a9f61700 -1 osd.5 86 *** Got signal Terminated *** 2023-07-14T19:18:18.222 INFO:journalctl@ceph.osd.5.smithi191.stdout:Jul 14 19:18:17 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-5[109616]: 2023-07-14T19:18:17.810+0000 7fa8a9f61700 -1 osd.5 86 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-07-14T19:18:18.222 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jul 14 19:18:18 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-11[120628]: 2023-07-14T19:18:18.050+0000 7f5ef7fea700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2023-07-14T19:17:44.480647+0000 front 2023-07-14T19:17:44.480662+0000 (oldest deadline 2023-07-14T19:18:06.780727+0000) 2023-07-14T19:18:18.222 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jul 14 19:18:18 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-11[120628]: 2023-07-14T19:18:18.050+0000 7f5ef7fea700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-07-14T19:17:50.782602+0000 front 2023-07-14T19:17:50.782583+0000 (oldest deadline 2023-07-14T19:18:14.282678+0000) 2023-07-14T19:18:18.242 INFO:journalctl@ceph.osd.10.smithi130.stdout:Jul 14 19:18:17 smithi130 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-10[119160]: 2023-07-14T19:18:17.770+0000 7fa1fdfb9700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2023-07-14T19:17:41.727090+0000 front 2023-07-14T19:17:41.727036+0000 (oldest deadline 2023-07-14T19:18:06.427174+0000) 2023-07-14T19:18:18.242 INFO:journalctl@ceph.osd.10.smithi130.stdout:Jul 14 19:18:17 smithi130 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-10[119160]: 2023-07-14T19:18:17.770+0000 7fa1fdfb9700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-07-14T19:17:52.828799+0000 front 2023-07-14T19:17:52.828840+0000 (oldest deadline 2023-07-14T19:18:14.528908+0000) 2023-07-14T19:18:18.675 INFO:journalctl@ceph.osd.9.smithi040.stdout:Jul 14 19:18:18 smithi040 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-9[125889]: 2023-07-14T19:18:18.419+0000 7fa29d625700 -1 osd.9 86 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2023-07-14T19:17:42.530269+0000 front 2023-07-14T19:17:42.530319+0000 (oldest deadline 2023-07-14T19:18:06.531071+0000) 2023-07-14T19:18:18.676 INFO:journalctl@ceph.osd.9.smithi040.stdout:Jul 14 19:18:18 smithi040 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-9[125889]: 2023-07-14T19:18:18.419+0000 7fa29d625700 -1 osd.9 86 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-07-14T19:17:50.632219+0000 front 2023-07-14T19:17:50.632182+0000 (oldest deadline 2023-07-14T19:18:16.532211+0000) 2023-07-14T19:18:18.935 INFO:journalctl@ceph.osd.5.smithi191.stdout:Jul 14 19:18:18 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-5[109616]: 2023-07-14T19:18:18.530+0000 7fa8a5556700 -1 osd.5 86 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2023-07-14T19:17:40.437352+0000 front 2023-07-14T19:17:40.437431+0000 (oldest deadline 2023-07-14T19:18:06.337579+0000) 2023-07-14T19:18:18.936 INFO:journalctl@ceph.osd.5.smithi191.stdout:Jul 14 19:18:18 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-5[109616]: 2023-07-14T19:18:18.530+0000 7fa8a5556700 -1 osd.5 86 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-07-14T19:17:51.039013+0000 front 2023-07-14T19:17:51.038875+0000 (oldest deadline 2023-07-14T19:18:13.938865+0000) 2023-07-14T19:18:18.970 INFO:journalctl@ceph.osd.6.smithi040.stdout:Jul 14 19:18:18 smithi040 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-6[120375]: 2023-07-14T19:18:18.675+0000 7f5cfd6d6700 -1 osd.6 86 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2023-07-14T19:17:44.759194+0000 front 2023-07-14T19:17:44.759148+0000 (oldest deadline 2023-07-14T19:18:07.659321+0000) 2023-07-14T19:18:18.971 INFO:journalctl@ceph.osd.6.smithi040.stdout:Jul 14 19:18:18 smithi040 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-6[120375]: 2023-07-14T19:18:18.675+0000 7f5cfd6d6700 -1 osd.6 86 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-07-14T19:17:51.160890+0000 front 2023-07-14T19:17:51.160697+0000 (oldest deadline 2023-07-14T19:18:15.260756+0000) 2023-07-14T19:18:18.992 INFO:journalctl@ceph.osd.7.smithi130.stdout:Jul 14 19:18:18 smithi130 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-7[113739]: 2023-07-14T19:18:18.516+0000 7f621462a700 -1 osd.7 86 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-07-14T19:17:48.674688+0000 front 2023-07-14T19:17:48.674657+0000 (oldest deadline 2023-07-14T19:18:14.574846+0000) 2023-07-14T19:18:18.992 INFO:journalctl@ceph.osd.10.smithi130.stdout:Jul 14 19:18:18 smithi130 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-10[119160]: 2023-07-14T19:18:18.757+0000 7fa1fdfb9700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2023-07-14T19:17:41.727090+0000 front 2023-07-14T19:17:41.727036+0000 (oldest deadline 2023-07-14T19:18:06.427174+0000) 2023-07-14T19:18:18.993 INFO:journalctl@ceph.osd.10.smithi130.stdout:Jul 14 19:18:18 smithi130 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-10[119160]: 2023-07-14T19:18:18.757+0000 7fa1fdfb9700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-07-14T19:17:52.828799+0000 front 2023-07-14T19:17:52.828840+0000 (oldest deadline 2023-07-14T19:18:14.528908+0000) 2023-07-14T19:18:19.222 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jul 14 19:18:19 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-11[120628]: 2023-07-14T19:18:19.071+0000 7f5ef7fea700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2023-07-14T19:17:44.480647+0000 front 2023-07-14T19:17:44.480662+0000 (oldest deadline 2023-07-14T19:18:06.780727+0000) 2023-07-14T19:18:19.222 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jul 14 19:18:19 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-11[120628]: 2023-07-14T19:18:19.071+0000 7f5ef7fea700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-07-14T19:17:50.782602+0000 front 2023-07-14T19:17:50.782583+0000 (oldest deadline 2023-07-14T19:18:14.282678+0000) 2023-07-14T19:18:19.223 INFO:journalctl@ceph.osd.8.smithi191.stdout:Jul 14 19:18:18 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-8[115063]: 2023-07-14T19:18:18.933+0000 7f42fdd64700 -1 osd.8 86 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-07-14T19:17:52.163739+0000 front 2023-07-14T19:17:52.163632+0000 (oldest deadline 2023-07-14T19:18:17.964264+0000) 2023-07-14T19:18:19.720 INFO:journalctl@ceph.osd.6.smithi040.stdout:Jul 14 19:18:19 smithi040 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-6[120375]: 2023-07-14T19:18:19.702+0000 7f5cfd6d6700 -1 osd.6 86 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2023-07-14T19:17:44.759194+0000 front 2023-07-14T19:17:44.759148+0000 (oldest deadline 2023-07-14T19:18:07.659321+0000) 2023-07-14T19:18:19.721 INFO:journalctl@ceph.osd.6.smithi040.stdout:Jul 14 19:18:19 smithi040 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-6[120375]: 2023-07-14T19:18:19.702+0000 7f5cfd6d6700 -1 osd.6 86 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-07-14T19:17:51.160890+0000 front 2023-07-14T19:17:51.160697+0000 (oldest deadline 2023-07-14T19:18:15.260756+0000) 2023-07-14T19:18:19.721 INFO:journalctl@ceph.osd.9.smithi040.stdout:Jul 14 19:18:19 smithi040 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-9[125889]: 2023-07-14T19:18:19.456+0000 7fa29d625700 -1 osd.9 86 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2023-07-14T19:17:42.530269+0000 front 2023-07-14T19:17:42.530319+0000 (oldest deadline 2023-07-14T19:18:06.531071+0000) 2023-07-14T19:18:19.721 INFO:journalctl@ceph.osd.9.smithi040.stdout:Jul 14 19:18:19 smithi040 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-9[125889]: 2023-07-14T19:18:19.456+0000 7fa29d625700 -1 osd.9 86 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-07-14T19:17:50.632219+0000 front 2023-07-14T19:17:50.632182+0000 (oldest deadline 2023-07-14T19:18:16.532211+0000) 2023-07-14T19:18:19.757 INFO:journalctl@ceph.osd.7.smithi130.stdout:Jul 14 19:18:19 smithi130 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-7[113739]: 2023-07-14T19:18:19.495+0000 7f621462a700 -1 osd.7 86 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-07-14T19:17:48.674688+0000 front 2023-07-14T19:17:48.674657+0000 (oldest deadline 2023-07-14T19:18:14.574846+0000) 2023-07-14T19:18:19.906 INFO:journalctl@ceph.osd.5.smithi191.stdout:Jul 14 19:18:19 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-5[109616]: 2023-07-14T19:18:19.549+0000 7fa8a5556700 -1 osd.5 86 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2023-07-14T19:17:40.437352+0000 front 2023-07-14T19:17:40.437431+0000 (oldest deadline 2023-07-14T19:18:06.337579+0000) 2023-07-14T19:18:19.906 INFO:journalctl@ceph.osd.5.smithi191.stdout:Jul 14 19:18:19 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-5[109616]: 2023-07-14T19:18:19.549+0000 7fa8a5556700 -1 osd.5 86 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-07-14T19:17:51.039013+0000 front 2023-07-14T19:17:51.038875+0000 (oldest deadline 2023-07-14T19:18:13.938865+0000) 2023-07-14T19:18:20.222 INFO:journalctl@ceph.osd.8.smithi191.stdout:Jul 14 19:18:19 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-8[115063]: 2023-07-14T19:18:19.904+0000 7f42fdd64700 -1 osd.8 86 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-07-14T19:17:52.163739+0000 front 2023-07-14T19:17:52.163632+0000 (oldest deadline 2023-07-14T19:18:17.964264+0000) 2023-07-14T19:18:20.222 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jul 14 19:18:20 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-11[120628]: 2023-07-14T19:18:20.070+0000 7f5ef7fea700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2023-07-14T19:17:44.480647+0000 front 2023-07-14T19:17:44.480662+0000 (oldest deadline 2023-07-14T19:18:06.780727+0000) 2023-07-14T19:18:20.222 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jul 14 19:18:20 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-11[120628]: 2023-07-14T19:18:20.070+0000 7f5ef7fea700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-07-14T19:17:50.782602+0000 front 2023-07-14T19:17:50.782583+0000 (oldest deadline 2023-07-14T19:18:14.282678+0000) 2023-07-14T19:18:20.242 INFO:journalctl@ceph.osd.10.smithi130.stdout:Jul 14 19:18:19 smithi130 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-10[119160]: 2023-07-14T19:18:19.756+0000 7fa1fdfb9700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2023-07-14T19:17:41.727090+0000 front 2023-07-14T19:17:41.727036+0000 (oldest deadline 2023-07-14T19:18:06.427174+0000) 2023-07-14T19:18:20.243 INFO:journalctl@ceph.osd.10.smithi130.stdout:Jul 14 19:18:19 smithi130 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-10[119160]: 2023-07-14T19:18:19.756+0000 7fa1fdfb9700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-07-14T19:17:52.828799+0000 front 2023-07-14T19:17:52.828840+0000 (oldest deadline 2023-07-14T19:18:14.528908+0000) 2023-07-14T19:18:20.714 INFO:journalctl@ceph.osd.7.smithi130.stdout:Jul 14 19:18:20 smithi130 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-7[113739]: 2023-07-14T19:18:20.457+0000 7f621462a700 -1 osd.7 86 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-07-14T19:17:48.674688+0000 front 2023-07-14T19:17:48.674657+0000 (oldest deadline 2023-07-14T19:18:14.574846+0000) 2023-07-14T19:18:20.904 INFO:journalctl@ceph.osd.5.smithi191.stdout:Jul 14 19:18:20 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-5[109616]: 2023-07-14T19:18:20.552+0000 7fa8a5556700 -1 osd.5 86 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2023-07-14T19:17:40.437352+0000 front 2023-07-14T19:17:40.437431+0000 (oldest deadline 2023-07-14T19:18:06.337579+0000) 2023-07-14T19:18:20.905 INFO:journalctl@ceph.osd.5.smithi191.stdout:Jul 14 19:18:20 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-5[109616]: 2023-07-14T19:18:20.552+0000 7fa8a5556700 -1 osd.5 86 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-07-14T19:17:51.039013+0000 front 2023-07-14T19:17:51.038875+0000 (oldest deadline 2023-07-14T19:18:13.938865+0000) 2023-07-14T19:18:20.971 INFO:journalctl@ceph.osd.6.smithi040.stdout:Jul 14 19:18:20 smithi040 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-6[120375]: 2023-07-14T19:18:20.666+0000 7f5cfd6d6700 -1 osd.6 86 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2023-07-14T19:17:44.759194+0000 front 2023-07-14T19:17:44.759148+0000 (oldest deadline 2023-07-14T19:18:07.659321+0000) 2023-07-14T19:18:20.971 INFO:journalctl@ceph.osd.6.smithi040.stdout:Jul 14 19:18:20 smithi040 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-6[120375]: 2023-07-14T19:18:20.666+0000 7f5cfd6d6700 -1 osd.6 86 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-07-14T19:17:51.160890+0000 front 2023-07-14T19:17:51.160697+0000 (oldest deadline 2023-07-14T19:18:15.260756+0000) 2023-07-14T19:18:20.971 INFO:journalctl@ceph.osd.9.smithi040.stdout:Jul 14 19:18:20 smithi040 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-9[125889]: 2023-07-14T19:18:20.501+0000 7fa29d625700 -1 osd.9 86 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2023-07-14T19:17:42.530269+0000 front 2023-07-14T19:17:42.530319+0000 (oldest deadline 2023-07-14T19:18:06.531071+0000) 2023-07-14T19:18:20.972 INFO:journalctl@ceph.osd.9.smithi040.stdout:Jul 14 19:18:20 smithi040 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-9[125889]: 2023-07-14T19:18:20.501+0000 7fa29d625700 -1 osd.9 86 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-07-14T19:17:50.632219+0000 front 2023-07-14T19:17:50.632182+0000 (oldest deadline 2023-07-14T19:18:16.532211+0000) 2023-07-14T19:18:20.992 INFO:journalctl@ceph.osd.10.smithi130.stdout:Jul 14 19:18:20 smithi130 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-10[119160]: 2023-07-14T19:18:20.713+0000 7fa1fdfb9700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2023-07-14T19:17:41.727090+0000 front 2023-07-14T19:17:41.727036+0000 (oldest deadline 2023-07-14T19:18:06.427174+0000) 2023-07-14T19:18:20.992 INFO:journalctl@ceph.osd.10.smithi130.stdout:Jul 14 19:18:20 smithi130 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-10[119160]: 2023-07-14T19:18:20.713+0000 7fa1fdfb9700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-07-14T19:17:52.828799+0000 front 2023-07-14T19:17:52.828840+0000 (oldest deadline 2023-07-14T19:18:14.528908+0000) 2023-07-14T19:18:21.221 INFO:journalctl@ceph.osd.8.smithi191.stdout:Jul 14 19:18:20 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-8[115063]: 2023-07-14T19:18:20.903+0000 7f42fdd64700 -1 osd.8 86 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-07-14T19:17:52.163739+0000 front 2023-07-14T19:17:52.163632+0000 (oldest deadline 2023-07-14T19:18:17.964264+0000) 2023-07-14T19:18:21.222 INFO:journalctl@ceph.osd.8.smithi191.stdout:Jul 14 19:18:20 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-8[115063]: 2023-07-14T19:18:20.903+0000 7f42fdd64700 -1 osd.8 86 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2023-07-14T19:17:57.964682+0000 front 2023-07-14T19:17:57.964797+0000 (oldest deadline 2023-07-14T19:18:20.864905+0000) 2023-07-14T19:18:21.222 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jul 14 19:18:21 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-11[120628]: 2023-07-14T19:18:21.044+0000 7f5ef7fea700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2023-07-14T19:17:44.480647+0000 front 2023-07-14T19:17:44.480662+0000 (oldest deadline 2023-07-14T19:18:06.780727+0000) 2023-07-14T19:18:21.222 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jul 14 19:18:21 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-11[120628]: 2023-07-14T19:18:21.044+0000 7f5ef7fea700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-07-14T19:17:50.782602+0000 front 2023-07-14T19:17:50.782583+0000 (oldest deadline 2023-07-14T19:18:14.282678+0000) 2023-07-14T19:18:21.721 INFO:journalctl@ceph.osd.6.smithi040.stdout:Jul 14 19:18:21 smithi040 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-6[120375]: 2023-07-14T19:18:21.684+0000 7f5cfd6d6700 -1 osd.6 86 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2023-07-14T19:17:44.759194+0000 front 2023-07-14T19:17:44.759148+0000 (oldest deadline 2023-07-14T19:18:07.659321+0000) 2023-07-14T19:18:21.721 INFO:journalctl@ceph.osd.6.smithi040.stdout:Jul 14 19:18:21 smithi040 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-6[120375]: 2023-07-14T19:18:21.684+0000 7f5cfd6d6700 -1 osd.6 86 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-07-14T19:17:51.160890+0000 front 2023-07-14T19:17:51.160697+0000 (oldest deadline 2023-07-14T19:18:15.260756+0000) 2023-07-14T19:18:21.721 INFO:journalctl@ceph.osd.6.smithi040.stdout:Jul 14 19:18:21 smithi040 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-6[120375]: 2023-07-14T19:18:21.684+0000 7f5cfd6d6700 -1 osd.6 86 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2023-07-14T19:17:55.762030+0000 front 2023-07-14T19:17:55.762203+0000 (oldest deadline 2023-07-14T19:18:21.062020+0000) 2023-07-14T19:18:21.722 INFO:journalctl@ceph.osd.9.smithi040.stdout:Jul 14 19:18:21 smithi040 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-9[125889]: 2023-07-14T19:18:21.469+0000 7fa29d625700 -1 osd.9 86 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2023-07-14T19:17:42.530269+0000 front 2023-07-14T19:17:42.530319+0000 (oldest deadline 2023-07-14T19:18:06.531071+0000) 2023-07-14T19:18:21.722 INFO:journalctl@ceph.osd.9.smithi040.stdout:Jul 14 19:18:21 smithi040 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-9[125889]: 2023-07-14T19:18:21.469+0000 7fa29d625700 -1 osd.9 86 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-07-14T19:17:50.632219+0000 front 2023-07-14T19:17:50.632182+0000 (oldest deadline 2023-07-14T19:18:16.532211+0000) 2023-07-14T19:18:21.926 INFO:journalctl@ceph.osd.5.smithi191.stdout:Jul 14 19:18:21 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-5[109616]: 2023-07-14T19:18:21.588+0000 7fa8a5556700 -1 osd.5 86 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2023-07-14T19:17:40.437352+0000 front 2023-07-14T19:17:40.437431+0000 (oldest deadline 2023-07-14T19:18:06.337579+0000) 2023-07-14T19:18:21.927 INFO:journalctl@ceph.osd.5.smithi191.stdout:Jul 14 19:18:21 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-5[109616]: 2023-07-14T19:18:21.588+0000 7fa8a5556700 -1 osd.5 86 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-07-14T19:17:51.039013+0000 front 2023-07-14T19:17:51.038875+0000 (oldest deadline 2023-07-14T19:18:13.938865+0000) 2023-07-14T19:18:21.992 INFO:journalctl@ceph.osd.7.smithi130.stdout:Jul 14 19:18:21 smithi130 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-7[113739]: 2023-07-14T19:18:21.499+0000 7f621462a700 -1 osd.7 86 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-07-14T19:17:48.674688+0000 front 2023-07-14T19:17:48.674657+0000 (oldest deadline 2023-07-14T19:18:14.574846+0000) 2023-07-14T19:18:21.992 INFO:journalctl@ceph.osd.10.smithi130.stdout:Jul 14 19:18:21 smithi130 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-10[119160]: 2023-07-14T19:18:21.719+0000 7fa1fdfb9700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2023-07-14T19:17:41.727090+0000 front 2023-07-14T19:17:41.727036+0000 (oldest deadline 2023-07-14T19:18:06.427174+0000) 2023-07-14T19:18:21.993 INFO:journalctl@ceph.osd.10.smithi130.stdout:Jul 14 19:18:21 smithi130 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-10[119160]: 2023-07-14T19:18:21.719+0000 7fa1fdfb9700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-07-14T19:17:52.828799+0000 front 2023-07-14T19:17:52.828840+0000 (oldest deadline 2023-07-14T19:18:14.528908+0000) 2023-07-14T19:18:22.222 INFO:journalctl@ceph.osd.8.smithi191.stdout:Jul 14 19:18:21 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-8[115063]: 2023-07-14T19:18:21.925+0000 7f42fdd64700 -1 osd.8 86 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-07-14T19:17:52.163739+0000 front 2023-07-14T19:17:52.163632+0000 (oldest deadline 2023-07-14T19:18:17.964264+0000) 2023-07-14T19:18:22.222 INFO:journalctl@ceph.osd.8.smithi191.stdout:Jul 14 19:18:21 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-8[115063]: 2023-07-14T19:18:21.925+0000 7f42fdd64700 -1 osd.8 86 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2023-07-14T19:17:57.964682+0000 front 2023-07-14T19:17:57.964797+0000 (oldest deadline 2023-07-14T19:18:20.864905+0000) 2023-07-14T19:18:22.222 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jul 14 19:18:22 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-11[120628]: 2023-07-14T19:18:22.094+0000 7f5ef7fea700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2023-07-14T19:17:44.480647+0000 front 2023-07-14T19:17:44.480662+0000 (oldest deadline 2023-07-14T19:18:06.780727+0000) 2023-07-14T19:18:22.223 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jul 14 19:18:22 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-11[120628]: 2023-07-14T19:18:22.094+0000 7f5ef7fea700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-07-14T19:17:50.782602+0000 front 2023-07-14T19:17:50.782583+0000 (oldest deadline 2023-07-14T19:18:14.282678+0000) 2023-07-14T19:18:22.961 INFO:journalctl@ceph.osd.5.smithi191.stdout:Jul 14 19:18:22 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-5[109616]: 2023-07-14T19:18:22.569+0000 7fa8a5556700 -1 osd.5 86 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2023-07-14T19:17:40.437352+0000 front 2023-07-14T19:17:40.437431+0000 (oldest deadline 2023-07-14T19:18:06.337579+0000) 2023-07-14T19:18:22.962 INFO:journalctl@ceph.osd.5.smithi191.stdout:Jul 14 19:18:22 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-5[109616]: 2023-07-14T19:18:22.569+0000 7fa8a5556700 -1 osd.5 86 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-07-14T19:17:51.039013+0000 front 2023-07-14T19:17:51.038875+0000 (oldest deadline 2023-07-14T19:18:13.938865+0000) 2023-07-14T19:18:22.970 INFO:journalctl@ceph.osd.6.smithi040.stdout:Jul 14 19:18:22 smithi040 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-6[120375]: 2023-07-14T19:18:22.718+0000 7f5cfd6d6700 -1 osd.6 86 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2023-07-14T19:17:44.759194+0000 front 2023-07-14T19:17:44.759148+0000 (oldest deadline 2023-07-14T19:18:07.659321+0000) 2023-07-14T19:18:22.971 INFO:journalctl@ceph.osd.6.smithi040.stdout:Jul 14 19:18:22 smithi040 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-6[120375]: 2023-07-14T19:18:22.718+0000 7f5cfd6d6700 -1 osd.6 86 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-07-14T19:17:51.160890+0000 front 2023-07-14T19:17:51.160697+0000 (oldest deadline 2023-07-14T19:18:15.260756+0000) 2023-07-14T19:18:22.971 INFO:journalctl@ceph.osd.6.smithi040.stdout:Jul 14 19:18:22 smithi040 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-6[120375]: 2023-07-14T19:18:22.718+0000 7f5cfd6d6700 -1 osd.6 86 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2023-07-14T19:17:55.762030+0000 front 2023-07-14T19:17:55.762203+0000 (oldest deadline 2023-07-14T19:18:21.062020+0000) 2023-07-14T19:18:22.971 INFO:journalctl@ceph.osd.9.smithi040.stdout:Jul 14 19:18:22 smithi040 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-9[125889]: 2023-07-14T19:18:22.507+0000 7fa29d625700 -1 osd.9 86 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2023-07-14T19:17:42.530269+0000 front 2023-07-14T19:17:42.530319+0000 (oldest deadline 2023-07-14T19:18:06.531071+0000) 2023-07-14T19:18:22.972 INFO:journalctl@ceph.osd.9.smithi040.stdout:Jul 14 19:18:22 smithi040 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-9[125889]: 2023-07-14T19:18:22.507+0000 7fa29d625700 -1 osd.9 86 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-07-14T19:17:50.632219+0000 front 2023-07-14T19:17:50.632182+0000 (oldest deadline 2023-07-14T19:18:16.532211+0000) 2023-07-14T19:18:22.972 INFO:journalctl@ceph.osd.9.smithi040.stdout:Jul 14 19:18:22 smithi040 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-9[125889]: 2023-07-14T19:18:22.507+0000 7fa29d625700 -1 osd.9 86 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2023-07-14T19:17:57.033248+0000 front 2023-07-14T19:17:57.033262+0000 (oldest deadline 2023-07-14T19:18:22.333292+0000) 2023-07-14T19:18:22.992 INFO:journalctl@ceph.osd.7.smithi130.stdout:Jul 14 19:18:22 smithi130 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-7[113739]: 2023-07-14T19:18:22.541+0000 7f621462a700 -1 osd.7 86 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-07-14T19:17:48.674688+0000 front 2023-07-14T19:17:48.674657+0000 (oldest deadline 2023-07-14T19:18:14.574846+0000) 2023-07-14T19:18:22.992 INFO:journalctl@ceph.osd.10.smithi130.stdout:Jul 14 19:18:22 smithi130 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-10[119160]: 2023-07-14T19:18:22.699+0000 7fa1fdfb9700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2023-07-14T19:17:41.727090+0000 front 2023-07-14T19:17:41.727036+0000 (oldest deadline 2023-07-14T19:18:06.427174+0000) 2023-07-14T19:18:22.992 INFO:journalctl@ceph.osd.10.smithi130.stdout:Jul 14 19:18:22 smithi130 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-10[119160]: 2023-07-14T19:18:22.699+0000 7fa1fdfb9700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-07-14T19:17:52.828799+0000 front 2023-07-14T19:17:52.828840+0000 (oldest deadline 2023-07-14T19:18:14.528908+0000) 2023-07-14T19:18:22.993 INFO:journalctl@ceph.osd.10.smithi130.stdout:Jul 14 19:18:22 smithi130 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-10[119160]: 2023-07-14T19:18:22.699+0000 7fa1fdfb9700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2023-07-14T19:17:56.730545+0000 front 2023-07-14T19:17:56.730590+0000 (oldest deadline 2023-07-14T19:18:22.630599+0000) 2023-07-14T19:18:23.222 INFO:journalctl@ceph.osd.8.smithi191.stdout:Jul 14 19:18:22 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-8[115063]: 2023-07-14T19:18:22.960+0000 7f42fdd64700 -1 osd.8 86 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-07-14T19:17:52.163739+0000 front 2023-07-14T19:17:52.163632+0000 (oldest deadline 2023-07-14T19:18:17.964264+0000) 2023-07-14T19:18:23.222 INFO:journalctl@ceph.osd.8.smithi191.stdout:Jul 14 19:18:22 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-8[115063]: 2023-07-14T19:18:22.960+0000 7f42fdd64700 -1 osd.8 86 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2023-07-14T19:17:57.964682+0000 front 2023-07-14T19:17:57.964797+0000 (oldest deadline 2023-07-14T19:18:20.864905+0000) 2023-07-14T19:18:23.222 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jul 14 19:18:23 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-11[120628]: 2023-07-14T19:18:23.088+0000 7f5ef7fea700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2023-07-14T19:17:44.480647+0000 front 2023-07-14T19:17:44.480662+0000 (oldest deadline 2023-07-14T19:18:06.780727+0000) 2023-07-14T19:18:23.223 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jul 14 19:18:23 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-11[120628]: 2023-07-14T19:18:23.088+0000 7f5ef7fea700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-07-14T19:17:50.782602+0000 front 2023-07-14T19:17:50.782583+0000 (oldest deadline 2023-07-14T19:18:14.282678+0000) 2023-07-14T19:18:23.593 INFO:journalctl@ceph.osd.5.smithi191.stdout:Jul 14 19:18:23 smithi191 bash[127117]: ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-5 2023-07-14T19:18:23.971 INFO:journalctl@ceph.osd.6.smithi040.stdout:Jul 14 19:18:23 smithi040 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-6[120375]: 2023-07-14T19:18:23.768+0000 7f5cfd6d6700 -1 osd.6 86 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2023-07-14T19:17:44.759194+0000 front 2023-07-14T19:17:44.759148+0000 (oldest deadline 2023-07-14T19:18:07.659321+0000) 2023-07-14T19:18:23.971 INFO:journalctl@ceph.osd.6.smithi040.stdout:Jul 14 19:18:23 smithi040 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-6[120375]: 2023-07-14T19:18:23.768+0000 7f5cfd6d6700 -1 osd.6 86 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-07-14T19:17:51.160890+0000 front 2023-07-14T19:17:51.160697+0000 (oldest deadline 2023-07-14T19:18:15.260756+0000) 2023-07-14T19:18:23.971 INFO:journalctl@ceph.osd.6.smithi040.stdout:Jul 14 19:18:23 smithi040 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-6[120375]: 2023-07-14T19:18:23.768+0000 7f5cfd6d6700 -1 osd.6 86 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2023-07-14T19:17:55.762030+0000 front 2023-07-14T19:17:55.762203+0000 (oldest deadline 2023-07-14T19:18:21.062020+0000) 2023-07-14T19:18:23.972 INFO:journalctl@ceph.osd.9.smithi040.stdout:Jul 14 19:18:23 smithi040 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-9[125889]: 2023-07-14T19:18:23.556+0000 7fa29d625700 -1 osd.9 86 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2023-07-14T19:17:42.530269+0000 front 2023-07-14T19:17:42.530319+0000 (oldest deadline 2023-07-14T19:18:06.531071+0000) 2023-07-14T19:18:23.972 INFO:journalctl@ceph.osd.9.smithi040.stdout:Jul 14 19:18:23 smithi040 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-9[125889]: 2023-07-14T19:18:23.556+0000 7fa29d625700 -1 osd.9 86 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-07-14T19:17:50.632219+0000 front 2023-07-14T19:17:50.632182+0000 (oldest deadline 2023-07-14T19:18:16.532211+0000) 2023-07-14T19:18:23.972 INFO:journalctl@ceph.osd.9.smithi040.stdout:Jul 14 19:18:23 smithi040 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-9[125889]: 2023-07-14T19:18:23.556+0000 7fa29d625700 -1 osd.9 86 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2023-07-14T19:17:57.033248+0000 front 2023-07-14T19:17:57.033262+0000 (oldest deadline 2023-07-14T19:18:22.333292+0000) 2023-07-14T19:18:23.992 INFO:journalctl@ceph.osd.7.smithi130.stdout:Jul 14 19:18:23 smithi130 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-7[113739]: 2023-07-14T19:18:23.502+0000 7f621462a700 -1 osd.7 86 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-07-14T19:17:48.674688+0000 front 2023-07-14T19:17:48.674657+0000 (oldest deadline 2023-07-14T19:18:14.574846+0000) 2023-07-14T19:18:23.992 INFO:journalctl@ceph.osd.7.smithi130.stdout:Jul 14 19:18:23 smithi130 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-7[113739]: 2023-07-14T19:18:23.502+0000 7f621462a700 -1 osd.7 86 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2023-07-14T19:17:57.475991+0000 front 2023-07-14T19:17:57.475868+0000 (oldest deadline 2023-07-14T19:18:23.376038+0000) 2023-07-14T19:18:23.992 INFO:journalctl@ceph.osd.10.smithi130.stdout:Jul 14 19:18:23 smithi130 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-10[119160]: 2023-07-14T19:18:23.693+0000 7fa1fdfb9700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2023-07-14T19:17:41.727090+0000 front 2023-07-14T19:17:41.727036+0000 (oldest deadline 2023-07-14T19:18:06.427174+0000) 2023-07-14T19:18:23.993 INFO:journalctl@ceph.osd.10.smithi130.stdout:Jul 14 19:18:23 smithi130 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-10[119160]: 2023-07-14T19:18:23.693+0000 7fa1fdfb9700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-07-14T19:17:52.828799+0000 front 2023-07-14T19:17:52.828840+0000 (oldest deadline 2023-07-14T19:18:14.528908+0000) 2023-07-14T19:18:23.993 INFO:journalctl@ceph.osd.10.smithi130.stdout:Jul 14 19:18:23 smithi130 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-10[119160]: 2023-07-14T19:18:23.693+0000 7fa1fdfb9700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2023-07-14T19:17:56.730545+0000 front 2023-07-14T19:17:56.730590+0000 (oldest deadline 2023-07-14T19:18:22.630599+0000) 2023-07-14T19:18:24.472 INFO:journalctl@ceph.osd.8.smithi191.stdout:Jul 14 19:18:23 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-8[115063]: 2023-07-14T19:18:23.989+0000 7f42fdd64700 -1 osd.8 86 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-07-14T19:17:52.163739+0000 front 2023-07-14T19:17:52.163632+0000 (oldest deadline 2023-07-14T19:18:17.964264+0000) 2023-07-14T19:18:24.472 INFO:journalctl@ceph.osd.8.smithi191.stdout:Jul 14 19:18:23 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-8[115063]: 2023-07-14T19:18:23.989+0000 7f42fdd64700 -1 osd.8 86 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2023-07-14T19:17:57.964682+0000 front 2023-07-14T19:17:57.964797+0000 (oldest deadline 2023-07-14T19:18:20.864905+0000) 2023-07-14T19:18:24.473 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jul 14 19:18:24 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-11[120628]: 2023-07-14T19:18:24.069+0000 7f5ef7fea700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2023-07-14T19:17:44.480647+0000 front 2023-07-14T19:17:44.480662+0000 (oldest deadline 2023-07-14T19:18:06.780727+0000) 2023-07-14T19:18:24.473 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jul 14 19:18:24 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-11[120628]: 2023-07-14T19:18:24.069+0000 7f5ef7fea700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-07-14T19:17:50.782602+0000 front 2023-07-14T19:17:50.782583+0000 (oldest deadline 2023-07-14T19:18:14.282678+0000) 2023-07-14T19:18:24.742 INFO:journalctl@ceph.osd.7.smithi130.stdout:Jul 14 19:18:24 smithi130 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-7[113739]: 2023-07-14T19:18:24.496+0000 7f621462a700 -1 osd.7 86 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-07-14T19:17:48.674688+0000 front 2023-07-14T19:17:48.674657+0000 (oldest deadline 2023-07-14T19:18:14.574846+0000) 2023-07-14T19:18:24.742 INFO:journalctl@ceph.osd.7.smithi130.stdout:Jul 14 19:18:24 smithi130 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-7[113739]: 2023-07-14T19:18:24.496+0000 7f621462a700 -1 osd.7 86 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2023-07-14T19:17:57.475991+0000 front 2023-07-14T19:17:57.475868+0000 (oldest deadline 2023-07-14T19:18:23.376038+0000) 2023-07-14T19:18:24.742 INFO:journalctl@ceph.osd.10.smithi130.stdout:Jul 14 19:18:24 smithi130 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-10[119160]: 2023-07-14T19:18:24.676+0000 7fa1fdfb9700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2023-07-14T19:17:41.727090+0000 front 2023-07-14T19:17:41.727036+0000 (oldest deadline 2023-07-14T19:18:06.427174+0000) 2023-07-14T19:18:24.743 INFO:journalctl@ceph.osd.10.smithi130.stdout:Jul 14 19:18:24 smithi130 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-10[119160]: 2023-07-14T19:18:24.676+0000 7fa1fdfb9700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-07-14T19:17:52.828799+0000 front 2023-07-14T19:17:52.828840+0000 (oldest deadline 2023-07-14T19:18:14.528908+0000) 2023-07-14T19:18:24.743 INFO:journalctl@ceph.osd.10.smithi130.stdout:Jul 14 19:18:24 smithi130 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-10[119160]: 2023-07-14T19:18:24.676+0000 7fa1fdfb9700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2023-07-14T19:17:56.730545+0000 front 2023-07-14T19:17:56.730590+0000 (oldest deadline 2023-07-14T19:18:22.630599+0000) 2023-07-14T19:18:24.956 DEBUG:teuthology.orchestra.run.smithi191:> sudo pkill -f 'journalctl -f -n 0 -u ceph-adc52f04-2279-11ee-9b33-001a4aab830c@osd.5.service' 2023-07-14T19:18:24.971 INFO:journalctl@ceph.osd.6.smithi040.stdout:Jul 14 19:18:24 smithi040 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-6[120375]: 2023-07-14T19:18:24.782+0000 7f5cfd6d6700 -1 osd.6 86 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2023-07-14T19:17:44.759194+0000 front 2023-07-14T19:17:44.759148+0000 (oldest deadline 2023-07-14T19:18:07.659321+0000) 2023-07-14T19:18:24.971 INFO:journalctl@ceph.osd.6.smithi040.stdout:Jul 14 19:18:24 smithi040 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-6[120375]: 2023-07-14T19:18:24.782+0000 7f5cfd6d6700 -1 osd.6 86 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-07-14T19:17:51.160890+0000 front 2023-07-14T19:17:51.160697+0000 (oldest deadline 2023-07-14T19:18:15.260756+0000) 2023-07-14T19:18:24.971 INFO:journalctl@ceph.osd.6.smithi040.stdout:Jul 14 19:18:24 smithi040 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-6[120375]: 2023-07-14T19:18:24.782+0000 7f5cfd6d6700 -1 osd.6 86 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2023-07-14T19:17:55.762030+0000 front 2023-07-14T19:17:55.762203+0000 (oldest deadline 2023-07-14T19:18:21.062020+0000) 2023-07-14T19:18:24.972 INFO:journalctl@ceph.osd.9.smithi040.stdout:Jul 14 19:18:24 smithi040 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-9[125889]: 2023-07-14T19:18:24.556+0000 7fa29d625700 -1 osd.9 86 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2023-07-14T19:17:42.530269+0000 front 2023-07-14T19:17:42.530319+0000 (oldest deadline 2023-07-14T19:18:06.531071+0000) 2023-07-14T19:18:24.972 INFO:journalctl@ceph.osd.9.smithi040.stdout:Jul 14 19:18:24 smithi040 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-9[125889]: 2023-07-14T19:18:24.556+0000 7fa29d625700 -1 osd.9 86 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-07-14T19:17:50.632219+0000 front 2023-07-14T19:17:50.632182+0000 (oldest deadline 2023-07-14T19:18:16.532211+0000) 2023-07-14T19:18:24.972 INFO:journalctl@ceph.osd.9.smithi040.stdout:Jul 14 19:18:24 smithi040 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-9[125889]: 2023-07-14T19:18:24.556+0000 7fa29d625700 -1 osd.9 86 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2023-07-14T19:17:57.033248+0000 front 2023-07-14T19:17:57.033262+0000 (oldest deadline 2023-07-14T19:18:22.333292+0000) 2023-07-14T19:18:25.036 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-14T19:18:25.037 INFO:tasks.cephadm.osd.5:Stopped osd.5 2023-07-14T19:18:25.037 INFO:tasks.cephadm.osd.6:Stopping osd.6... 2023-07-14T19:18:25.038 DEBUG:teuthology.orchestra.run.smithi040:> sudo systemctl stop ceph-adc52f04-2279-11ee-9b33-001a4aab830c@osd.6 2023-07-14T19:18:25.221 INFO:journalctl@ceph.osd.8.smithi191.stdout:Jul 14 19:18:24 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-8[115063]: 2023-07-14T19:18:24.992+0000 7f42fdd64700 -1 osd.8 86 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-07-14T19:17:52.163739+0000 front 2023-07-14T19:17:52.163632+0000 (oldest deadline 2023-07-14T19:18:17.964264+0000) 2023-07-14T19:18:25.222 INFO:journalctl@ceph.osd.8.smithi191.stdout:Jul 14 19:18:24 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-8[115063]: 2023-07-14T19:18:24.992+0000 7f42fdd64700 -1 osd.8 86 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2023-07-14T19:17:57.964682+0000 front 2023-07-14T19:17:57.964797+0000 (oldest deadline 2023-07-14T19:18:20.864905+0000) 2023-07-14T19:18:25.222 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jul 14 19:18:25 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-11[120628]: 2023-07-14T19:18:25.036+0000 7f5ef7fea700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2023-07-14T19:17:44.480647+0000 front 2023-07-14T19:17:44.480662+0000 (oldest deadline 2023-07-14T19:18:06.780727+0000) 2023-07-14T19:18:25.223 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jul 14 19:18:25 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-11[120628]: 2023-07-14T19:18:25.036+0000 7f5ef7fea700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-07-14T19:17:50.782602+0000 front 2023-07-14T19:17:50.782583+0000 (oldest deadline 2023-07-14T19:18:14.282678+0000) 2023-07-14T19:18:25.410 INFO:journalctl@ceph.osd.6.smithi040.stdout:Jul 14 19:18:25 smithi040 systemd[1]: Stopping Ceph osd.6 for adc52f04-2279-11ee-9b33-001a4aab830c... 2023-07-14T19:18:25.410 INFO:journalctl@ceph.osd.6.smithi040.stdout:Jul 14 19:18:25 smithi040 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-6[120375]: 2023-07-14T19:18:25.409+0000 7f5d020e1700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-07-14T19:18:25.721 INFO:journalctl@ceph.osd.6.smithi040.stdout:Jul 14 19:18:25 smithi040 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-6[120375]: 2023-07-14T19:18:25.409+0000 7f5d020e1700 -1 osd.6 86 *** Got signal Terminated *** 2023-07-14T19:18:25.721 INFO:journalctl@ceph.osd.6.smithi040.stdout:Jul 14 19:18:25 smithi040 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-6[120375]: 2023-07-14T19:18:25.409+0000 7f5d020e1700 -1 osd.6 86 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-07-14T19:18:25.721 INFO:journalctl@ceph.osd.9.smithi040.stdout:Jul 14 19:18:25 smithi040 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-9[125889]: 2023-07-14T19:18:25.572+0000 7fa29d625700 -1 osd.9 86 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2023-07-14T19:17:42.530269+0000 front 2023-07-14T19:17:42.530319+0000 (oldest deadline 2023-07-14T19:18:06.531071+0000) 2023-07-14T19:18:25.722 INFO:journalctl@ceph.osd.9.smithi040.stdout:Jul 14 19:18:25 smithi040 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-9[125889]: 2023-07-14T19:18:25.572+0000 7fa29d625700 -1 osd.9 86 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-07-14T19:17:50.632219+0000 front 2023-07-14T19:17:50.632182+0000 (oldest deadline 2023-07-14T19:18:16.532211+0000) 2023-07-14T19:18:25.722 INFO:journalctl@ceph.osd.9.smithi040.stdout:Jul 14 19:18:25 smithi040 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-9[125889]: 2023-07-14T19:18:25.572+0000 7fa29d625700 -1 osd.9 86 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2023-07-14T19:17:57.033248+0000 front 2023-07-14T19:17:57.033262+0000 (oldest deadline 2023-07-14T19:18:22.333292+0000) 2023-07-14T19:18:25.992 INFO:journalctl@ceph.osd.7.smithi130.stdout:Jul 14 19:18:25 smithi130 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-7[113739]: 2023-07-14T19:18:25.507+0000 7f621462a700 -1 osd.7 86 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-07-14T19:17:48.674688+0000 front 2023-07-14T19:17:48.674657+0000 (oldest deadline 2023-07-14T19:18:14.574846+0000) 2023-07-14T19:18:25.992 INFO:journalctl@ceph.osd.7.smithi130.stdout:Jul 14 19:18:25 smithi130 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-7[113739]: 2023-07-14T19:18:25.507+0000 7f621462a700 -1 osd.7 86 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2023-07-14T19:17:57.475991+0000 front 2023-07-14T19:17:57.475868+0000 (oldest deadline 2023-07-14T19:18:23.376038+0000) 2023-07-14T19:18:25.993 INFO:journalctl@ceph.osd.10.smithi130.stdout:Jul 14 19:18:25 smithi130 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-10[119160]: 2023-07-14T19:18:25.686+0000 7fa1fdfb9700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2023-07-14T19:17:41.727090+0000 front 2023-07-14T19:17:41.727036+0000 (oldest deadline 2023-07-14T19:18:06.427174+0000) 2023-07-14T19:18:25.993 INFO:journalctl@ceph.osd.10.smithi130.stdout:Jul 14 19:18:25 smithi130 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-10[119160]: 2023-07-14T19:18:25.686+0000 7fa1fdfb9700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-07-14T19:17:52.828799+0000 front 2023-07-14T19:17:52.828840+0000 (oldest deadline 2023-07-14T19:18:14.528908+0000) 2023-07-14T19:18:25.993 INFO:journalctl@ceph.osd.10.smithi130.stdout:Jul 14 19:18:25 smithi130 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-10[119160]: 2023-07-14T19:18:25.686+0000 7fa1fdfb9700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2023-07-14T19:17:56.730545+0000 front 2023-07-14T19:17:56.730590+0000 (oldest deadline 2023-07-14T19:18:22.630599+0000) 2023-07-14T19:18:26.221 INFO:journalctl@ceph.osd.6.smithi040.stdout:Jul 14 19:18:25 smithi040 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-6[120375]: 2023-07-14T19:18:25.750+0000 7f5cfd6d6700 -1 osd.6 86 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2023-07-14T19:17:44.759194+0000 front 2023-07-14T19:17:44.759148+0000 (oldest deadline 2023-07-14T19:18:07.659321+0000) 2023-07-14T19:18:26.221 INFO:journalctl@ceph.osd.6.smithi040.stdout:Jul 14 19:18:25 smithi040 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-6[120375]: 2023-07-14T19:18:25.750+0000 7f5cfd6d6700 -1 osd.6 86 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-07-14T19:17:51.160890+0000 front 2023-07-14T19:17:51.160697+0000 (oldest deadline 2023-07-14T19:18:15.260756+0000) 2023-07-14T19:18:26.221 INFO:journalctl@ceph.osd.6.smithi040.stdout:Jul 14 19:18:25 smithi040 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-6[120375]: 2023-07-14T19:18:25.750+0000 7f5cfd6d6700 -1 osd.6 86 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2023-07-14T19:17:55.762030+0000 front 2023-07-14T19:17:55.762203+0000 (oldest deadline 2023-07-14T19:18:21.062020+0000) 2023-07-14T19:18:26.472 INFO:journalctl@ceph.osd.8.smithi191.stdout:Jul 14 19:18:26 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-8[115063]: 2023-07-14T19:18:26.000+0000 7f42fdd64700 -1 osd.8 86 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-07-14T19:17:52.163739+0000 front 2023-07-14T19:17:52.163632+0000 (oldest deadline 2023-07-14T19:18:17.964264+0000) 2023-07-14T19:18:26.472 INFO:journalctl@ceph.osd.8.smithi191.stdout:Jul 14 19:18:26 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-8[115063]: 2023-07-14T19:18:26.000+0000 7f42fdd64700 -1 osd.8 86 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2023-07-14T19:17:57.964682+0000 front 2023-07-14T19:17:57.964797+0000 (oldest deadline 2023-07-14T19:18:20.864905+0000) 2023-07-14T19:18:26.472 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jul 14 19:18:26 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-11[120628]: 2023-07-14T19:18:26.027+0000 7f5ef7fea700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2023-07-14T19:17:44.480647+0000 front 2023-07-14T19:17:44.480662+0000 (oldest deadline 2023-07-14T19:18:06.780727+0000) 2023-07-14T19:18:26.473 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jul 14 19:18:26 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-11[120628]: 2023-07-14T19:18:26.027+0000 7f5ef7fea700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-07-14T19:17:50.782602+0000 front 2023-07-14T19:17:50.782583+0000 (oldest deadline 2023-07-14T19:18:14.282678+0000) 2023-07-14T19:18:26.473 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jul 14 19:18:26 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-11[120628]: 2023-07-14T19:18:26.027+0000 7f5ef7fea700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2023-07-14T19:17:58.983820+0000 front 2023-07-14T19:17:58.983785+0000 (oldest deadline 2023-07-14T19:18:25.984643+0000) 2023-07-14T19:18:26.970 INFO:journalctl@ceph.osd.6.smithi040.stdout:Jul 14 19:18:26 smithi040 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-6[120375]: 2023-07-14T19:18:26.718+0000 7f5cfd6d6700 -1 osd.6 86 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2023-07-14T19:17:44.759194+0000 front 2023-07-14T19:17:44.759148+0000 (oldest deadline 2023-07-14T19:18:07.659321+0000) 2023-07-14T19:18:26.971 INFO:journalctl@ceph.osd.6.smithi040.stdout:Jul 14 19:18:26 smithi040 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-6[120375]: 2023-07-14T19:18:26.718+0000 7f5cfd6d6700 -1 osd.6 86 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-07-14T19:17:51.160890+0000 front 2023-07-14T19:17:51.160697+0000 (oldest deadline 2023-07-14T19:18:15.260756+0000) 2023-07-14T19:18:26.971 INFO:journalctl@ceph.osd.6.smithi040.stdout:Jul 14 19:18:26 smithi040 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-6[120375]: 2023-07-14T19:18:26.718+0000 7f5cfd6d6700 -1 osd.6 86 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2023-07-14T19:17:55.762030+0000 front 2023-07-14T19:17:55.762203+0000 (oldest deadline 2023-07-14T19:18:21.062020+0000) 2023-07-14T19:18:26.972 INFO:journalctl@ceph.osd.9.smithi040.stdout:Jul 14 19:18:26 smithi040 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-9[125889]: 2023-07-14T19:18:26.559+0000 7fa29d625700 -1 osd.9 86 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2023-07-14T19:17:42.530269+0000 front 2023-07-14T19:17:42.530319+0000 (oldest deadline 2023-07-14T19:18:06.531071+0000) 2023-07-14T19:18:26.972 INFO:journalctl@ceph.osd.9.smithi040.stdout:Jul 14 19:18:26 smithi040 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-9[125889]: 2023-07-14T19:18:26.559+0000 7fa29d625700 -1 osd.9 86 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-07-14T19:17:50.632219+0000 front 2023-07-14T19:17:50.632182+0000 (oldest deadline 2023-07-14T19:18:16.532211+0000) 2023-07-14T19:18:26.972 INFO:journalctl@ceph.osd.9.smithi040.stdout:Jul 14 19:18:26 smithi040 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-9[125889]: 2023-07-14T19:18:26.559+0000 7fa29d625700 -1 osd.9 86 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2023-07-14T19:17:57.033248+0000 front 2023-07-14T19:17:57.033262+0000 (oldest deadline 2023-07-14T19:18:22.333292+0000) 2023-07-14T19:18:26.992 INFO:journalctl@ceph.osd.7.smithi130.stdout:Jul 14 19:18:26 smithi130 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-7[113739]: 2023-07-14T19:18:26.508+0000 7f621462a700 -1 osd.7 86 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-07-14T19:17:48.674688+0000 front 2023-07-14T19:17:48.674657+0000 (oldest deadline 2023-07-14T19:18:14.574846+0000) 2023-07-14T19:18:26.992 INFO:journalctl@ceph.osd.7.smithi130.stdout:Jul 14 19:18:26 smithi130 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-7[113739]: 2023-07-14T19:18:26.508+0000 7f621462a700 -1 osd.7 86 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2023-07-14T19:17:57.475991+0000 front 2023-07-14T19:17:57.475868+0000 (oldest deadline 2023-07-14T19:18:23.376038+0000) 2023-07-14T19:18:26.992 INFO:journalctl@ceph.osd.10.smithi130.stdout:Jul 14 19:18:26 smithi130 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-10[119160]: 2023-07-14T19:18:26.681+0000 7fa1fdfb9700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2023-07-14T19:17:41.727090+0000 front 2023-07-14T19:17:41.727036+0000 (oldest deadline 2023-07-14T19:18:06.427174+0000) 2023-07-14T19:18:26.993 INFO:journalctl@ceph.osd.10.smithi130.stdout:Jul 14 19:18:26 smithi130 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-10[119160]: 2023-07-14T19:18:26.681+0000 7fa1fdfb9700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-07-14T19:17:52.828799+0000 front 2023-07-14T19:17:52.828840+0000 (oldest deadline 2023-07-14T19:18:14.528908+0000) 2023-07-14T19:18:26.993 INFO:journalctl@ceph.osd.10.smithi130.stdout:Jul 14 19:18:26 smithi130 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-10[119160]: 2023-07-14T19:18:26.681+0000 7fa1fdfb9700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2023-07-14T19:17:56.730545+0000 front 2023-07-14T19:17:56.730590+0000 (oldest deadline 2023-07-14T19:18:22.630599+0000) 2023-07-14T19:18:27.472 INFO:journalctl@ceph.osd.8.smithi191.stdout:Jul 14 19:18:27 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-8[115063]: 2023-07-14T19:18:27.005+0000 7f42fdd64700 -1 osd.8 86 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-07-14T19:17:52.163739+0000 front 2023-07-14T19:17:52.163632+0000 (oldest deadline 2023-07-14T19:18:17.964264+0000) 2023-07-14T19:18:27.472 INFO:journalctl@ceph.osd.8.smithi191.stdout:Jul 14 19:18:27 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-8[115063]: 2023-07-14T19:18:27.005+0000 7f42fdd64700 -1 osd.8 86 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2023-07-14T19:17:57.964682+0000 front 2023-07-14T19:17:57.964797+0000 (oldest deadline 2023-07-14T19:18:20.864905+0000) 2023-07-14T19:18:27.473 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jul 14 19:18:27 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-11[120628]: 2023-07-14T19:18:27.050+0000 7f5ef7fea700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2023-07-14T19:17:44.480647+0000 front 2023-07-14T19:17:44.480662+0000 (oldest deadline 2023-07-14T19:18:06.780727+0000) 2023-07-14T19:18:27.473 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jul 14 19:18:27 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-11[120628]: 2023-07-14T19:18:27.050+0000 7f5ef7fea700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-07-14T19:17:50.782602+0000 front 2023-07-14T19:17:50.782583+0000 (oldest deadline 2023-07-14T19:18:14.282678+0000) 2023-07-14T19:18:27.473 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jul 14 19:18:27 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-11[120628]: 2023-07-14T19:18:27.050+0000 7f5ef7fea700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2023-07-14T19:17:58.983820+0000 front 2023-07-14T19:17:58.983785+0000 (oldest deadline 2023-07-14T19:18:25.984643+0000) 2023-07-14T19:18:27.971 INFO:journalctl@ceph.osd.6.smithi040.stdout:Jul 14 19:18:27 smithi040 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-6[120375]: 2023-07-14T19:18:27.693+0000 7f5cfd6d6700 -1 osd.6 86 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2023-07-14T19:17:44.759194+0000 front 2023-07-14T19:17:44.759148+0000 (oldest deadline 2023-07-14T19:18:07.659321+0000) 2023-07-14T19:18:27.971 INFO:journalctl@ceph.osd.6.smithi040.stdout:Jul 14 19:18:27 smithi040 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-6[120375]: 2023-07-14T19:18:27.693+0000 7f5cfd6d6700 -1 osd.6 86 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-07-14T19:17:51.160890+0000 front 2023-07-14T19:17:51.160697+0000 (oldest deadline 2023-07-14T19:18:15.260756+0000) 2023-07-14T19:18:27.971 INFO:journalctl@ceph.osd.6.smithi040.stdout:Jul 14 19:18:27 smithi040 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-6[120375]: 2023-07-14T19:18:27.693+0000 7f5cfd6d6700 -1 osd.6 86 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2023-07-14T19:17:55.762030+0000 front 2023-07-14T19:17:55.762203+0000 (oldest deadline 2023-07-14T19:18:21.062020+0000) 2023-07-14T19:18:27.972 INFO:journalctl@ceph.osd.9.smithi040.stdout:Jul 14 19:18:27 smithi040 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-9[125889]: 2023-07-14T19:18:27.597+0000 7fa29d625700 -1 osd.9 86 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2023-07-14T19:17:42.530269+0000 front 2023-07-14T19:17:42.530319+0000 (oldest deadline 2023-07-14T19:18:06.531071+0000) 2023-07-14T19:18:27.972 INFO:journalctl@ceph.osd.9.smithi040.stdout:Jul 14 19:18:27 smithi040 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-9[125889]: 2023-07-14T19:18:27.597+0000 7fa29d625700 -1 osd.9 86 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-07-14T19:17:50.632219+0000 front 2023-07-14T19:17:50.632182+0000 (oldest deadline 2023-07-14T19:18:16.532211+0000) 2023-07-14T19:18:27.972 INFO:journalctl@ceph.osd.9.smithi040.stdout:Jul 14 19:18:27 smithi040 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-9[125889]: 2023-07-14T19:18:27.597+0000 7fa29d625700 -1 osd.9 86 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2023-07-14T19:17:57.033248+0000 front 2023-07-14T19:17:57.033262+0000 (oldest deadline 2023-07-14T19:18:22.333292+0000) 2023-07-14T19:18:27.992 INFO:journalctl@ceph.osd.7.smithi130.stdout:Jul 14 19:18:27 smithi130 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-7[113739]: 2023-07-14T19:18:27.534+0000 7f621462a700 -1 osd.7 86 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-07-14T19:17:48.674688+0000 front 2023-07-14T19:17:48.674657+0000 (oldest deadline 2023-07-14T19:18:14.574846+0000) 2023-07-14T19:18:27.992 INFO:journalctl@ceph.osd.7.smithi130.stdout:Jul 14 19:18:27 smithi130 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-7[113739]: 2023-07-14T19:18:27.534+0000 7f621462a700 -1 osd.7 86 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2023-07-14T19:17:57.475991+0000 front 2023-07-14T19:17:57.475868+0000 (oldest deadline 2023-07-14T19:18:23.376038+0000) 2023-07-14T19:18:27.992 INFO:journalctl@ceph.osd.10.smithi130.stdout:Jul 14 19:18:27 smithi130 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-10[119160]: 2023-07-14T19:18:27.645+0000 7fa1fdfb9700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2023-07-14T19:17:41.727090+0000 front 2023-07-14T19:17:41.727036+0000 (oldest deadline 2023-07-14T19:18:06.427174+0000) 2023-07-14T19:18:27.993 INFO:journalctl@ceph.osd.10.smithi130.stdout:Jul 14 19:18:27 smithi130 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-10[119160]: 2023-07-14T19:18:27.645+0000 7fa1fdfb9700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-07-14T19:17:52.828799+0000 front 2023-07-14T19:17:52.828840+0000 (oldest deadline 2023-07-14T19:18:14.528908+0000) 2023-07-14T19:18:27.993 INFO:journalctl@ceph.osd.10.smithi130.stdout:Jul 14 19:18:27 smithi130 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-10[119160]: 2023-07-14T19:18:27.645+0000 7fa1fdfb9700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2023-07-14T19:17:56.730545+0000 front 2023-07-14T19:17:56.730590+0000 (oldest deadline 2023-07-14T19:18:22.630599+0000) 2023-07-14T19:18:28.472 INFO:journalctl@ceph.osd.8.smithi191.stdout:Jul 14 19:18:28 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-8[115063]: 2023-07-14T19:18:28.046+0000 7f42fdd64700 -1 osd.8 86 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-07-14T19:17:52.163739+0000 front 2023-07-14T19:17:52.163632+0000 (oldest deadline 2023-07-14T19:18:17.964264+0000) 2023-07-14T19:18:28.472 INFO:journalctl@ceph.osd.8.smithi191.stdout:Jul 14 19:18:28 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-8[115063]: 2023-07-14T19:18:28.046+0000 7f42fdd64700 -1 osd.8 86 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2023-07-14T19:17:57.964682+0000 front 2023-07-14T19:17:57.964797+0000 (oldest deadline 2023-07-14T19:18:20.864905+0000) 2023-07-14T19:18:28.472 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jul 14 19:18:28 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-11[120628]: 2023-07-14T19:18:28.054+0000 7f5ef7fea700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2023-07-14T19:17:44.480647+0000 front 2023-07-14T19:17:44.480662+0000 (oldest deadline 2023-07-14T19:18:06.780727+0000) 2023-07-14T19:18:28.473 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jul 14 19:18:28 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-11[120628]: 2023-07-14T19:18:28.054+0000 7f5ef7fea700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-07-14T19:17:50.782602+0000 front 2023-07-14T19:17:50.782583+0000 (oldest deadline 2023-07-14T19:18:14.282678+0000) 2023-07-14T19:18:28.473 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jul 14 19:18:28 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-11[120628]: 2023-07-14T19:18:28.054+0000 7f5ef7fea700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2023-07-14T19:17:58.983820+0000 front 2023-07-14T19:17:58.983785+0000 (oldest deadline 2023-07-14T19:18:25.984643+0000) 2023-07-14T19:18:28.971 INFO:journalctl@ceph.osd.6.smithi040.stdout:Jul 14 19:18:28 smithi040 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-6[120375]: 2023-07-14T19:18:28.713+0000 7f5cfd6d6700 -1 osd.6 86 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2023-07-14T19:17:44.759194+0000 front 2023-07-14T19:17:44.759148+0000 (oldest deadline 2023-07-14T19:18:07.659321+0000) 2023-07-14T19:18:28.971 INFO:journalctl@ceph.osd.6.smithi040.stdout:Jul 14 19:18:28 smithi040 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-6[120375]: 2023-07-14T19:18:28.713+0000 7f5cfd6d6700 -1 osd.6 86 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-07-14T19:17:51.160890+0000 front 2023-07-14T19:17:51.160697+0000 (oldest deadline 2023-07-14T19:18:15.260756+0000) 2023-07-14T19:18:28.971 INFO:journalctl@ceph.osd.6.smithi040.stdout:Jul 14 19:18:28 smithi040 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-6[120375]: 2023-07-14T19:18:28.713+0000 7f5cfd6d6700 -1 osd.6 86 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2023-07-14T19:17:55.762030+0000 front 2023-07-14T19:17:55.762203+0000 (oldest deadline 2023-07-14T19:18:21.062020+0000) 2023-07-14T19:18:28.972 INFO:journalctl@ceph.osd.9.smithi040.stdout:Jul 14 19:18:28 smithi040 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-9[125889]: 2023-07-14T19:18:28.550+0000 7fa29d625700 -1 osd.9 86 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2023-07-14T19:17:42.530269+0000 front 2023-07-14T19:17:42.530319+0000 (oldest deadline 2023-07-14T19:18:06.531071+0000) 2023-07-14T19:18:28.972 INFO:journalctl@ceph.osd.9.smithi040.stdout:Jul 14 19:18:28 smithi040 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-9[125889]: 2023-07-14T19:18:28.550+0000 7fa29d625700 -1 osd.9 86 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-07-14T19:17:50.632219+0000 front 2023-07-14T19:17:50.632182+0000 (oldest deadline 2023-07-14T19:18:16.532211+0000) 2023-07-14T19:18:28.972 INFO:journalctl@ceph.osd.9.smithi040.stdout:Jul 14 19:18:28 smithi040 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-9[125889]: 2023-07-14T19:18:28.550+0000 7fa29d625700 -1 osd.9 86 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2023-07-14T19:17:57.033248+0000 front 2023-07-14T19:17:57.033262+0000 (oldest deadline 2023-07-14T19:18:22.333292+0000) 2023-07-14T19:18:28.992 INFO:journalctl@ceph.osd.7.smithi130.stdout:Jul 14 19:18:28 smithi130 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-7[113739]: 2023-07-14T19:18:28.574+0000 7f621462a700 -1 osd.7 86 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-07-14T19:17:48.674688+0000 front 2023-07-14T19:17:48.674657+0000 (oldest deadline 2023-07-14T19:18:14.574846+0000) 2023-07-14T19:18:28.992 INFO:journalctl@ceph.osd.7.smithi130.stdout:Jul 14 19:18:28 smithi130 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-7[113739]: 2023-07-14T19:18:28.574+0000 7f621462a700 -1 osd.7 86 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2023-07-14T19:17:57.475991+0000 front 2023-07-14T19:17:57.475868+0000 (oldest deadline 2023-07-14T19:18:23.376038+0000) 2023-07-14T19:18:28.993 INFO:journalctl@ceph.osd.10.smithi130.stdout:Jul 14 19:18:28 smithi130 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-10[119160]: 2023-07-14T19:18:28.668+0000 7fa1fdfb9700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2023-07-14T19:17:41.727090+0000 front 2023-07-14T19:17:41.727036+0000 (oldest deadline 2023-07-14T19:18:06.427174+0000) 2023-07-14T19:18:28.993 INFO:journalctl@ceph.osd.10.smithi130.stdout:Jul 14 19:18:28 smithi130 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-10[119160]: 2023-07-14T19:18:28.668+0000 7fa1fdfb9700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-07-14T19:17:52.828799+0000 front 2023-07-14T19:17:52.828840+0000 (oldest deadline 2023-07-14T19:18:14.528908+0000) 2023-07-14T19:18:28.993 INFO:journalctl@ceph.osd.10.smithi130.stdout:Jul 14 19:18:28 smithi130 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-10[119160]: 2023-07-14T19:18:28.668+0000 7fa1fdfb9700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2023-07-14T19:17:56.730545+0000 front 2023-07-14T19:17:56.730590+0000 (oldest deadline 2023-07-14T19:18:22.630599+0000) 2023-07-14T19:18:29.472 INFO:journalctl@ceph.osd.8.smithi191.stdout:Jul 14 19:18:29 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-8[115063]: 2023-07-14T19:18:29.031+0000 7f42fdd64700 -1 osd.8 86 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-07-14T19:17:52.163739+0000 front 2023-07-14T19:17:52.163632+0000 (oldest deadline 2023-07-14T19:18:17.964264+0000) 2023-07-14T19:18:29.473 INFO:journalctl@ceph.osd.8.smithi191.stdout:Jul 14 19:18:29 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-8[115063]: 2023-07-14T19:18:29.031+0000 7f42fdd64700 -1 osd.8 86 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2023-07-14T19:17:57.964682+0000 front 2023-07-14T19:17:57.964797+0000 (oldest deadline 2023-07-14T19:18:20.864905+0000) 2023-07-14T19:18:29.473 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jul 14 19:18:29 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-11[120628]: 2023-07-14T19:18:29.013+0000 7f5ef7fea700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2023-07-14T19:17:44.480647+0000 front 2023-07-14T19:17:44.480662+0000 (oldest deadline 2023-07-14T19:18:06.780727+0000) 2023-07-14T19:18:29.474 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jul 14 19:18:29 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-11[120628]: 2023-07-14T19:18:29.013+0000 7f5ef7fea700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-07-14T19:17:50.782602+0000 front 2023-07-14T19:17:50.782583+0000 (oldest deadline 2023-07-14T19:18:14.282678+0000) 2023-07-14T19:18:29.474 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jul 14 19:18:29 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-11[120628]: 2023-07-14T19:18:29.013+0000 7f5ef7fea700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2023-07-14T19:17:58.983820+0000 front 2023-07-14T19:17:58.983785+0000 (oldest deadline 2023-07-14T19:18:25.984643+0000) 2023-07-14T19:18:29.971 INFO:journalctl@ceph.osd.6.smithi040.stdout:Jul 14 19:18:29 smithi040 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-6[120375]: 2023-07-14T19:18:29.692+0000 7f5cfd6d6700 -1 osd.6 86 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2023-07-14T19:17:44.759194+0000 front 2023-07-14T19:17:44.759148+0000 (oldest deadline 2023-07-14T19:18:07.659321+0000) 2023-07-14T19:18:29.972 INFO:journalctl@ceph.osd.6.smithi040.stdout:Jul 14 19:18:29 smithi040 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-6[120375]: 2023-07-14T19:18:29.692+0000 7f5cfd6d6700 -1 osd.6 86 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-07-14T19:17:51.160890+0000 front 2023-07-14T19:17:51.160697+0000 (oldest deadline 2023-07-14T19:18:15.260756+0000) 2023-07-14T19:18:29.972 INFO:journalctl@ceph.osd.6.smithi040.stdout:Jul 14 19:18:29 smithi040 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-6[120375]: 2023-07-14T19:18:29.692+0000 7f5cfd6d6700 -1 osd.6 86 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2023-07-14T19:17:55.762030+0000 front 2023-07-14T19:17:55.762203+0000 (oldest deadline 2023-07-14T19:18:21.062020+0000) 2023-07-14T19:18:29.973 INFO:journalctl@ceph.osd.6.smithi040.stdout:Jul 14 19:18:29 smithi040 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-6[120375]: 2023-07-14T19:18:29.692+0000 7f5cfd6d6700 -1 osd.6 86 heartbeat_check: no reply from 172.21.15.40:6814 osd.3 since back 2023-07-14T19:18:05.564068+0000 front 2023-07-14T19:18:05.564033+0000 (oldest deadline 2023-07-14T19:18:29.664166+0000) 2023-07-14T19:18:29.973 INFO:journalctl@ceph.osd.9.smithi040.stdout:Jul 14 19:18:29 smithi040 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-9[125889]: 2023-07-14T19:18:29.516+0000 7fa29d625700 -1 osd.9 86 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2023-07-14T19:17:42.530269+0000 front 2023-07-14T19:17:42.530319+0000 (oldest deadline 2023-07-14T19:18:06.531071+0000) 2023-07-14T19:18:29.973 INFO:journalctl@ceph.osd.9.smithi040.stdout:Jul 14 19:18:29 smithi040 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-9[125889]: 2023-07-14T19:18:29.516+0000 7fa29d625700 -1 osd.9 86 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-07-14T19:17:50.632219+0000 front 2023-07-14T19:17:50.632182+0000 (oldest deadline 2023-07-14T19:18:16.532211+0000) 2023-07-14T19:18:29.974 INFO:journalctl@ceph.osd.9.smithi040.stdout:Jul 14 19:18:29 smithi040 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-9[125889]: 2023-07-14T19:18:29.516+0000 7fa29d625700 -1 osd.9 86 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2023-07-14T19:17:57.033248+0000 front 2023-07-14T19:17:57.033262+0000 (oldest deadline 2023-07-14T19:18:22.333292+0000) 2023-07-14T19:18:29.992 INFO:journalctl@ceph.osd.7.smithi130.stdout:Jul 14 19:18:29 smithi130 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-7[113739]: 2023-07-14T19:18:29.582+0000 7f621462a700 -1 osd.7 86 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-07-14T19:17:48.674688+0000 front 2023-07-14T19:17:48.674657+0000 (oldest deadline 2023-07-14T19:18:14.574846+0000) 2023-07-14T19:18:29.992 INFO:journalctl@ceph.osd.7.smithi130.stdout:Jul 14 19:18:29 smithi130 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-7[113739]: 2023-07-14T19:18:29.582+0000 7f621462a700 -1 osd.7 86 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2023-07-14T19:17:57.475991+0000 front 2023-07-14T19:17:57.475868+0000 (oldest deadline 2023-07-14T19:18:23.376038+0000) 2023-07-14T19:18:29.993 INFO:journalctl@ceph.osd.7.smithi130.stdout:Jul 14 19:18:29 smithi130 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-7[113739]: 2023-07-14T19:18:29.582+0000 7f621462a700 -1 osd.7 86 heartbeat_check: no reply from 172.21.15.40:6814 osd.3 since back 2023-07-14T19:18:03.376649+0000 front 2023-07-14T19:18:03.376725+0000 (oldest deadline 2023-07-14T19:18:28.676747+0000) 2023-07-14T19:18:29.993 INFO:journalctl@ceph.osd.10.smithi130.stdout:Jul 14 19:18:29 smithi130 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-10[119160]: 2023-07-14T19:18:29.664+0000 7fa1fdfb9700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2023-07-14T19:17:41.727090+0000 front 2023-07-14T19:17:41.727036+0000 (oldest deadline 2023-07-14T19:18:06.427174+0000) 2023-07-14T19:18:29.993 INFO:journalctl@ceph.osd.10.smithi130.stdout:Jul 14 19:18:29 smithi130 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-10[119160]: 2023-07-14T19:18:29.664+0000 7fa1fdfb9700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-07-14T19:17:52.828799+0000 front 2023-07-14T19:17:52.828840+0000 (oldest deadline 2023-07-14T19:18:14.528908+0000) 2023-07-14T19:18:29.993 INFO:journalctl@ceph.osd.10.smithi130.stdout:Jul 14 19:18:29 smithi130 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-10[119160]: 2023-07-14T19:18:29.664+0000 7fa1fdfb9700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2023-07-14T19:17:56.730545+0000 front 2023-07-14T19:17:56.730590+0000 (oldest deadline 2023-07-14T19:18:22.630599+0000) 2023-07-14T19:18:30.222 INFO:journalctl@ceph.osd.8.smithi191.stdout:Jul 14 19:18:30 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-8[115063]: 2023-07-14T19:18:30.043+0000 7f42fdd64700 -1 osd.8 86 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-07-14T19:17:52.163739+0000 front 2023-07-14T19:17:52.163632+0000 (oldest deadline 2023-07-14T19:18:17.964264+0000) 2023-07-14T19:18:30.222 INFO:journalctl@ceph.osd.8.smithi191.stdout:Jul 14 19:18:30 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-8[115063]: 2023-07-14T19:18:30.043+0000 7f42fdd64700 -1 osd.8 86 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2023-07-14T19:17:57.964682+0000 front 2023-07-14T19:17:57.964797+0000 (oldest deadline 2023-07-14T19:18:20.864905+0000) 2023-07-14T19:18:30.222 INFO:journalctl@ceph.osd.8.smithi191.stdout:Jul 14 19:18:30 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-8[115063]: 2023-07-14T19:18:30.043+0000 7f42fdd64700 -1 osd.8 86 heartbeat_check: no reply from 172.21.15.40:6814 osd.3 since back 2023-07-14T19:18:03.765725+0000 front 2023-07-14T19:18:03.765750+0000 (oldest deadline 2023-07-14T19:18:29.665790+0000) 2023-07-14T19:18:30.223 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jul 14 19:18:29 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-11[120628]: 2023-07-14T19:18:29.969+0000 7f5ef7fea700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2023-07-14T19:17:44.480647+0000 front 2023-07-14T19:17:44.480662+0000 (oldest deadline 2023-07-14T19:18:06.780727+0000) 2023-07-14T19:18:30.223 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jul 14 19:18:29 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-11[120628]: 2023-07-14T19:18:29.969+0000 7f5ef7fea700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-07-14T19:17:50.782602+0000 front 2023-07-14T19:17:50.782583+0000 (oldest deadline 2023-07-14T19:18:14.282678+0000) 2023-07-14T19:18:30.223 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jul 14 19:18:29 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-11[120628]: 2023-07-14T19:18:29.969+0000 7f5ef7fea700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2023-07-14T19:17:58.983820+0000 front 2023-07-14T19:17:58.983785+0000 (oldest deadline 2023-07-14T19:18:25.984643+0000) 2023-07-14T19:18:30.924 INFO:journalctl@ceph.osd.9.smithi040.stdout:Jul 14 19:18:30 smithi040 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-9[125889]: 2023-07-14T19:18:30.531+0000 7fa29d625700 -1 osd.9 86 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2023-07-14T19:17:42.530269+0000 front 2023-07-14T19:17:42.530319+0000 (oldest deadline 2023-07-14T19:18:06.531071+0000) 2023-07-14T19:18:30.925 INFO:journalctl@ceph.osd.9.smithi040.stdout:Jul 14 19:18:30 smithi040 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-9[125889]: 2023-07-14T19:18:30.531+0000 7fa29d625700 -1 osd.9 86 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-07-14T19:17:50.632219+0000 front 2023-07-14T19:17:50.632182+0000 (oldest deadline 2023-07-14T19:18:16.532211+0000) 2023-07-14T19:18:30.925 INFO:journalctl@ceph.osd.9.smithi040.stdout:Jul 14 19:18:30 smithi040 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-9[125889]: 2023-07-14T19:18:30.531+0000 7fa29d625700 -1 osd.9 86 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2023-07-14T19:17:57.033248+0000 front 2023-07-14T19:17:57.033262+0000 (oldest deadline 2023-07-14T19:18:22.333292+0000) 2023-07-14T19:18:30.925 INFO:journalctl@ceph.osd.9.smithi040.stdout:Jul 14 19:18:30 smithi040 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-9[125889]: 2023-07-14T19:18:30.531+0000 7fa29d625700 -1 osd.9 86 heartbeat_check: no reply from 172.21.15.40:6814 osd.3 since back 2023-07-14T19:18:07.535132+0000 front 2023-07-14T19:18:07.535107+0000 (oldest deadline 2023-07-14T19:18:30.435355+0000) 2023-07-14T19:18:30.992 INFO:journalctl@ceph.osd.7.smithi130.stdout:Jul 14 19:18:30 smithi130 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-7[113739]: 2023-07-14T19:18:30.621+0000 7f621462a700 -1 osd.7 86 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-07-14T19:17:48.674688+0000 front 2023-07-14T19:17:48.674657+0000 (oldest deadline 2023-07-14T19:18:14.574846+0000) 2023-07-14T19:18:30.992 INFO:journalctl@ceph.osd.7.smithi130.stdout:Jul 14 19:18:30 smithi130 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-7[113739]: 2023-07-14T19:18:30.621+0000 7f621462a700 -1 osd.7 86 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2023-07-14T19:17:57.475991+0000 front 2023-07-14T19:17:57.475868+0000 (oldest deadline 2023-07-14T19:18:23.376038+0000) 2023-07-14T19:18:30.993 INFO:journalctl@ceph.osd.7.smithi130.stdout:Jul 14 19:18:30 smithi130 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-7[113739]: 2023-07-14T19:18:30.621+0000 7f621462a700 -1 osd.7 86 heartbeat_check: no reply from 172.21.15.40:6814 osd.3 since back 2023-07-14T19:18:03.376649+0000 front 2023-07-14T19:18:03.376725+0000 (oldest deadline 2023-07-14T19:18:28.676747+0000) 2023-07-14T19:18:30.993 INFO:journalctl@ceph.osd.10.smithi130.stdout:Jul 14 19:18:30 smithi130 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-10[119160]: 2023-07-14T19:18:30.624+0000 7fa1fdfb9700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2023-07-14T19:17:41.727090+0000 front 2023-07-14T19:17:41.727036+0000 (oldest deadline 2023-07-14T19:18:06.427174+0000) 2023-07-14T19:18:30.993 INFO:journalctl@ceph.osd.10.smithi130.stdout:Jul 14 19:18:30 smithi130 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-10[119160]: 2023-07-14T19:18:30.624+0000 7fa1fdfb9700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-07-14T19:17:52.828799+0000 front 2023-07-14T19:17:52.828840+0000 (oldest deadline 2023-07-14T19:18:14.528908+0000) 2023-07-14T19:18:30.994 INFO:journalctl@ceph.osd.10.smithi130.stdout:Jul 14 19:18:30 smithi130 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-10[119160]: 2023-07-14T19:18:30.624+0000 7fa1fdfb9700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2023-07-14T19:17:56.730545+0000 front 2023-07-14T19:17:56.730590+0000 (oldest deadline 2023-07-14T19:18:22.630599+0000) 2023-07-14T19:18:31.470 INFO:journalctl@ceph.osd.6.smithi040.stdout:Jul 14 19:18:31 smithi040 bash[146740]: ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-6 2023-07-14T19:18:31.471 INFO:journalctl@ceph.osd.8.smithi191.stdout:Jul 14 19:18:31 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-8[115063]: 2023-07-14T19:18:31.026+0000 7f42fdd64700 -1 osd.8 86 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-07-14T19:17:52.163739+0000 front 2023-07-14T19:17:52.163632+0000 (oldest deadline 2023-07-14T19:18:17.964264+0000) 2023-07-14T19:18:31.472 INFO:journalctl@ceph.osd.8.smithi191.stdout:Jul 14 19:18:31 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-8[115063]: 2023-07-14T19:18:31.026+0000 7f42fdd64700 -1 osd.8 86 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2023-07-14T19:17:57.964682+0000 front 2023-07-14T19:17:57.964797+0000 (oldest deadline 2023-07-14T19:18:20.864905+0000) 2023-07-14T19:18:31.472 INFO:journalctl@ceph.osd.8.smithi191.stdout:Jul 14 19:18:31 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-8[115063]: 2023-07-14T19:18:31.026+0000 7f42fdd64700 -1 osd.8 86 heartbeat_check: no reply from 172.21.15.40:6814 osd.3 since back 2023-07-14T19:18:03.765725+0000 front 2023-07-14T19:18:03.765750+0000 (oldest deadline 2023-07-14T19:18:29.665790+0000) 2023-07-14T19:18:31.472 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jul 14 19:18:30 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-11[120628]: 2023-07-14T19:18:30.983+0000 7f5ef7fea700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2023-07-14T19:17:44.480647+0000 front 2023-07-14T19:17:44.480662+0000 (oldest deadline 2023-07-14T19:18:06.780727+0000) 2023-07-14T19:18:31.472 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jul 14 19:18:30 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-11[120628]: 2023-07-14T19:18:30.983+0000 7f5ef7fea700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-07-14T19:17:50.782602+0000 front 2023-07-14T19:17:50.782583+0000 (oldest deadline 2023-07-14T19:18:14.282678+0000) 2023-07-14T19:18:31.473 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jul 14 19:18:30 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-11[120628]: 2023-07-14T19:18:30.983+0000 7f5ef7fea700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2023-07-14T19:17:58.983820+0000 front 2023-07-14T19:17:58.983785+0000 (oldest deadline 2023-07-14T19:18:25.984643+0000) 2023-07-14T19:18:31.473 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jul 14 19:18:30 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-11[120628]: 2023-07-14T19:18:30.983+0000 7f5ef7fea700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.40:6814 osd.3 since back 2023-07-14T19:18:05.985177+0000 front 2023-07-14T19:18:05.985194+0000 (oldest deadline 2023-07-14T19:18:30.685202+0000) 2023-07-14T19:18:31.871 INFO:journalctl@ceph.osd.9.smithi040.stdout:Jul 14 19:18:31 smithi040 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-9[125889]: 2023-07-14T19:18:31.484+0000 7fa29d625700 -1 osd.9 86 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2023-07-14T19:17:42.530269+0000 front 2023-07-14T19:17:42.530319+0000 (oldest deadline 2023-07-14T19:18:06.531071+0000) 2023-07-14T19:18:31.871 INFO:journalctl@ceph.osd.9.smithi040.stdout:Jul 14 19:18:31 smithi040 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-9[125889]: 2023-07-14T19:18:31.484+0000 7fa29d625700 -1 osd.9 86 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-07-14T19:17:50.632219+0000 front 2023-07-14T19:17:50.632182+0000 (oldest deadline 2023-07-14T19:18:16.532211+0000) 2023-07-14T19:18:31.872 INFO:journalctl@ceph.osd.9.smithi040.stdout:Jul 14 19:18:31 smithi040 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-9[125889]: 2023-07-14T19:18:31.484+0000 7fa29d625700 -1 osd.9 86 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2023-07-14T19:17:57.033248+0000 front 2023-07-14T19:17:57.033262+0000 (oldest deadline 2023-07-14T19:18:22.333292+0000) 2023-07-14T19:18:31.872 INFO:journalctl@ceph.osd.9.smithi040.stdout:Jul 14 19:18:31 smithi040 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-9[125889]: 2023-07-14T19:18:31.484+0000 7fa29d625700 -1 osd.9 86 heartbeat_check: no reply from 172.21.15.40:6814 osd.3 since back 2023-07-14T19:18:07.535132+0000 front 2023-07-14T19:18:07.535107+0000 (oldest deadline 2023-07-14T19:18:30.435355+0000) 2023-07-14T19:18:31.992 INFO:journalctl@ceph.osd.7.smithi130.stdout:Jul 14 19:18:31 smithi130 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-7[113739]: 2023-07-14T19:18:31.607+0000 7f621462a700 -1 osd.7 86 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-07-14T19:17:48.674688+0000 front 2023-07-14T19:17:48.674657+0000 (oldest deadline 2023-07-14T19:18:14.574846+0000) 2023-07-14T19:18:31.992 INFO:journalctl@ceph.osd.7.smithi130.stdout:Jul 14 19:18:31 smithi130 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-7[113739]: 2023-07-14T19:18:31.607+0000 7f621462a700 -1 osd.7 86 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2023-07-14T19:17:57.475991+0000 front 2023-07-14T19:17:57.475868+0000 (oldest deadline 2023-07-14T19:18:23.376038+0000) 2023-07-14T19:18:31.993 INFO:journalctl@ceph.osd.7.smithi130.stdout:Jul 14 19:18:31 smithi130 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-7[113739]: 2023-07-14T19:18:31.607+0000 7f621462a700 -1 osd.7 86 heartbeat_check: no reply from 172.21.15.40:6814 osd.3 since back 2023-07-14T19:18:03.376649+0000 front 2023-07-14T19:18:03.376725+0000 (oldest deadline 2023-07-14T19:18:28.676747+0000) 2023-07-14T19:18:31.993 INFO:journalctl@ceph.osd.10.smithi130.stdout:Jul 14 19:18:31 smithi130 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-10[119160]: 2023-07-14T19:18:31.599+0000 7fa1fdfb9700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2023-07-14T19:17:41.727090+0000 front 2023-07-14T19:17:41.727036+0000 (oldest deadline 2023-07-14T19:18:06.427174+0000) 2023-07-14T19:18:31.993 INFO:journalctl@ceph.osd.10.smithi130.stdout:Jul 14 19:18:31 smithi130 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-10[119160]: 2023-07-14T19:18:31.599+0000 7fa1fdfb9700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-07-14T19:17:52.828799+0000 front 2023-07-14T19:17:52.828840+0000 (oldest deadline 2023-07-14T19:18:14.528908+0000) 2023-07-14T19:18:31.994 INFO:journalctl@ceph.osd.10.smithi130.stdout:Jul 14 19:18:31 smithi130 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-10[119160]: 2023-07-14T19:18:31.599+0000 7fa1fdfb9700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2023-07-14T19:17:56.730545+0000 front 2023-07-14T19:17:56.730590+0000 (oldest deadline 2023-07-14T19:18:22.630599+0000) 2023-07-14T19:18:31.994 INFO:journalctl@ceph.osd.10.smithi130.stdout:Jul 14 19:18:31 smithi130 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-10[119160]: 2023-07-14T19:18:31.599+0000 7fa1fdfb9700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.40:6814 osd.3 since back 2023-07-14T19:18:05.432244+0000 front 2023-07-14T19:18:05.432347+0000 (oldest deadline 2023-07-14T19:18:31.331990+0000) 2023-07-14T19:18:32.472 INFO:journalctl@ceph.osd.8.smithi191.stdout:Jul 14 19:18:32 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-8[115063]: 2023-07-14T19:18:32.009+0000 7f42fdd64700 -1 osd.8 86 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-07-14T19:17:52.163739+0000 front 2023-07-14T19:17:52.163632+0000 (oldest deadline 2023-07-14T19:18:17.964264+0000) 2023-07-14T19:18:32.472 INFO:journalctl@ceph.osd.8.smithi191.stdout:Jul 14 19:18:32 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-8[115063]: 2023-07-14T19:18:32.009+0000 7f42fdd64700 -1 osd.8 86 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2023-07-14T19:17:57.964682+0000 front 2023-07-14T19:17:57.964797+0000 (oldest deadline 2023-07-14T19:18:20.864905+0000) 2023-07-14T19:18:32.472 INFO:journalctl@ceph.osd.8.smithi191.stdout:Jul 14 19:18:32 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-8[115063]: 2023-07-14T19:18:32.009+0000 7f42fdd64700 -1 osd.8 86 heartbeat_check: no reply from 172.21.15.40:6814 osd.3 since back 2023-07-14T19:18:03.765725+0000 front 2023-07-14T19:18:03.765750+0000 (oldest deadline 2023-07-14T19:18:29.665790+0000) 2023-07-14T19:18:32.473 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jul 14 19:18:31 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-11[120628]: 2023-07-14T19:18:31.994+0000 7f5ef7fea700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2023-07-14T19:17:44.480647+0000 front 2023-07-14T19:17:44.480662+0000 (oldest deadline 2023-07-14T19:18:06.780727+0000) 2023-07-14T19:18:32.473 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jul 14 19:18:31 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-11[120628]: 2023-07-14T19:18:31.994+0000 7f5ef7fea700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-07-14T19:17:50.782602+0000 front 2023-07-14T19:17:50.782583+0000 (oldest deadline 2023-07-14T19:18:14.282678+0000) 2023-07-14T19:18:32.473 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jul 14 19:18:31 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-11[120628]: 2023-07-14T19:18:31.994+0000 7f5ef7fea700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2023-07-14T19:17:58.983820+0000 front 2023-07-14T19:17:58.983785+0000 (oldest deadline 2023-07-14T19:18:25.984643+0000) 2023-07-14T19:18:32.473 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jul 14 19:18:31 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-11[120628]: 2023-07-14T19:18:31.994+0000 7f5ef7fea700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.40:6814 osd.3 since back 2023-07-14T19:18:05.985177+0000 front 2023-07-14T19:18:05.985194+0000 (oldest deadline 2023-07-14T19:18:30.685202+0000) 2023-07-14T19:18:32.654 INFO:journalctl@ceph.osd.9.smithi040.stdout:Jul 14 19:18:32 smithi040 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-9[125889]: 2023-07-14T19:18:32.531+0000 7fa29d625700 -1 osd.9 86 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2023-07-14T19:17:42.530269+0000 front 2023-07-14T19:17:42.530319+0000 (oldest deadline 2023-07-14T19:18:06.531071+0000) 2023-07-14T19:18:32.654 INFO:journalctl@ceph.osd.9.smithi040.stdout:Jul 14 19:18:32 smithi040 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-9[125889]: 2023-07-14T19:18:32.531+0000 7fa29d625700 -1 osd.9 86 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-07-14T19:17:50.632219+0000 front 2023-07-14T19:17:50.632182+0000 (oldest deadline 2023-07-14T19:18:16.532211+0000) 2023-07-14T19:18:32.654 INFO:journalctl@ceph.osd.9.smithi040.stdout:Jul 14 19:18:32 smithi040 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-9[125889]: 2023-07-14T19:18:32.531+0000 7fa29d625700 -1 osd.9 86 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2023-07-14T19:17:57.033248+0000 front 2023-07-14T19:17:57.033262+0000 (oldest deadline 2023-07-14T19:18:22.333292+0000) 2023-07-14T19:18:32.655 INFO:journalctl@ceph.osd.9.smithi040.stdout:Jul 14 19:18:32 smithi040 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-9[125889]: 2023-07-14T19:18:32.531+0000 7fa29d625700 -1 osd.9 86 heartbeat_check: no reply from 172.21.15.40:6814 osd.3 since back 2023-07-14T19:18:07.535132+0000 front 2023-07-14T19:18:07.535107+0000 (oldest deadline 2023-07-14T19:18:30.435355+0000) 2023-07-14T19:18:32.661 DEBUG:teuthology.orchestra.run.smithi040:> sudo pkill -f 'journalctl -f -n 0 -u ceph-adc52f04-2279-11ee-9b33-001a4aab830c@osd.6.service' 2023-07-14T19:18:32.706 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-14T19:18:32.706 INFO:tasks.cephadm.osd.6:Stopped osd.6 2023-07-14T19:18:32.707 INFO:tasks.cephadm.osd.7:Stopping osd.7... 2023-07-14T19:18:32.707 DEBUG:teuthology.orchestra.run.smithi130:> sudo systemctl stop ceph-adc52f04-2279-11ee-9b33-001a4aab830c@osd.7 2023-07-14T19:18:32.992 INFO:journalctl@ceph.osd.7.smithi130.stdout:Jul 14 19:18:32 smithi130 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-7[113739]: 2023-07-14T19:18:32.572+0000 7f621462a700 -1 osd.7 86 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-07-14T19:17:48.674688+0000 front 2023-07-14T19:17:48.674657+0000 (oldest deadline 2023-07-14T19:18:14.574846+0000) 2023-07-14T19:18:32.992 INFO:journalctl@ceph.osd.7.smithi130.stdout:Jul 14 19:18:32 smithi130 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-7[113739]: 2023-07-14T19:18:32.572+0000 7f621462a700 -1 osd.7 86 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2023-07-14T19:17:57.475991+0000 front 2023-07-14T19:17:57.475868+0000 (oldest deadline 2023-07-14T19:18:23.376038+0000) 2023-07-14T19:18:32.992 INFO:journalctl@ceph.osd.7.smithi130.stdout:Jul 14 19:18:32 smithi130 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-7[113739]: 2023-07-14T19:18:32.572+0000 7f621462a700 -1 osd.7 86 heartbeat_check: no reply from 172.21.15.40:6814 osd.3 since back 2023-07-14T19:18:03.376649+0000 front 2023-07-14T19:18:03.376725+0000 (oldest deadline 2023-07-14T19:18:28.676747+0000) 2023-07-14T19:18:32.993 INFO:journalctl@ceph.osd.7.smithi130.stdout:Jul 14 19:18:32 smithi130 systemd[1]: Stopping Ceph osd.7 for adc52f04-2279-11ee-9b33-001a4aab830c... 2023-07-14T19:18:32.993 INFO:journalctl@ceph.osd.10.smithi130.stdout:Jul 14 19:18:32 smithi130 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-10[119160]: 2023-07-14T19:18:32.611+0000 7fa1fdfb9700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2023-07-14T19:17:41.727090+0000 front 2023-07-14T19:17:41.727036+0000 (oldest deadline 2023-07-14T19:18:06.427174+0000) 2023-07-14T19:18:32.993 INFO:journalctl@ceph.osd.10.smithi130.stdout:Jul 14 19:18:32 smithi130 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-10[119160]: 2023-07-14T19:18:32.611+0000 7fa1fdfb9700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-07-14T19:17:52.828799+0000 front 2023-07-14T19:17:52.828840+0000 (oldest deadline 2023-07-14T19:18:14.528908+0000) 2023-07-14T19:18:32.993 INFO:journalctl@ceph.osd.10.smithi130.stdout:Jul 14 19:18:32 smithi130 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-10[119160]: 2023-07-14T19:18:32.611+0000 7fa1fdfb9700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2023-07-14T19:17:56.730545+0000 front 2023-07-14T19:17:56.730590+0000 (oldest deadline 2023-07-14T19:18:22.630599+0000) 2023-07-14T19:18:32.994 INFO:journalctl@ceph.osd.10.smithi130.stdout:Jul 14 19:18:32 smithi130 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-10[119160]: 2023-07-14T19:18:32.611+0000 7fa1fdfb9700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.40:6814 osd.3 since back 2023-07-14T19:18:05.432244+0000 front 2023-07-14T19:18:05.432347+0000 (oldest deadline 2023-07-14T19:18:31.331990+0000) 2023-07-14T19:18:33.472 INFO:journalctl@ceph.osd.8.smithi191.stdout:Jul 14 19:18:33 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-8[115063]: 2023-07-14T19:18:33.026+0000 7f42fdd64700 -1 osd.8 86 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-07-14T19:17:52.163739+0000 front 2023-07-14T19:17:52.163632+0000 (oldest deadline 2023-07-14T19:18:17.964264+0000) 2023-07-14T19:18:33.472 INFO:journalctl@ceph.osd.8.smithi191.stdout:Jul 14 19:18:33 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-8[115063]: 2023-07-14T19:18:33.026+0000 7f42fdd64700 -1 osd.8 86 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2023-07-14T19:17:57.964682+0000 front 2023-07-14T19:17:57.964797+0000 (oldest deadline 2023-07-14T19:18:20.864905+0000) 2023-07-14T19:18:33.473 INFO:journalctl@ceph.osd.8.smithi191.stdout:Jul 14 19:18:33 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-8[115063]: 2023-07-14T19:18:33.026+0000 7f42fdd64700 -1 osd.8 86 heartbeat_check: no reply from 172.21.15.40:6814 osd.3 since back 2023-07-14T19:18:03.765725+0000 front 2023-07-14T19:18:03.765750+0000 (oldest deadline 2023-07-14T19:18:29.665790+0000) 2023-07-14T19:18:33.473 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jul 14 19:18:32 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-11[120628]: 2023-07-14T19:18:32.988+0000 7f5ef7fea700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2023-07-14T19:17:44.480647+0000 front 2023-07-14T19:17:44.480662+0000 (oldest deadline 2023-07-14T19:18:06.780727+0000) 2023-07-14T19:18:33.473 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jul 14 19:18:32 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-11[120628]: 2023-07-14T19:18:32.988+0000 7f5ef7fea700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-07-14T19:17:50.782602+0000 front 2023-07-14T19:17:50.782583+0000 (oldest deadline 2023-07-14T19:18:14.282678+0000) 2023-07-14T19:18:33.473 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jul 14 19:18:32 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-11[120628]: 2023-07-14T19:18:32.988+0000 7f5ef7fea700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2023-07-14T19:17:58.983820+0000 front 2023-07-14T19:17:58.983785+0000 (oldest deadline 2023-07-14T19:18:25.984643+0000) 2023-07-14T19:18:33.474 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jul 14 19:18:32 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-11[120628]: 2023-07-14T19:18:32.988+0000 7f5ef7fea700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.40:6814 osd.3 since back 2023-07-14T19:18:05.985177+0000 front 2023-07-14T19:18:05.985194+0000 (oldest deadline 2023-07-14T19:18:30.685202+0000) 2023-07-14T19:18:33.492 INFO:journalctl@ceph.osd.7.smithi130.stdout:Jul 14 19:18:33 smithi130 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-7[113739]: 2023-07-14T19:18:33.061+0000 7f621a037700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-07-14T19:18:33.492 INFO:journalctl@ceph.osd.7.smithi130.stdout:Jul 14 19:18:33 smithi130 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-7[113739]: 2023-07-14T19:18:33.061+0000 7f621a037700 -1 osd.7 86 *** Got signal Terminated *** 2023-07-14T19:18:33.492 INFO:journalctl@ceph.osd.7.smithi130.stdout:Jul 14 19:18:33 smithi130 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-7[113739]: 2023-07-14T19:18:33.061+0000 7f621a037700 -1 osd.7 86 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-07-14T19:18:33.971 INFO:journalctl@ceph.osd.9.smithi040.stdout:Jul 14 19:18:33 smithi040 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-9[125889]: 2023-07-14T19:18:33.519+0000 7fa29d625700 -1 osd.9 86 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2023-07-14T19:17:42.530269+0000 front 2023-07-14T19:17:42.530319+0000 (oldest deadline 2023-07-14T19:18:06.531071+0000) 2023-07-14T19:18:33.971 INFO:journalctl@ceph.osd.9.smithi040.stdout:Jul 14 19:18:33 smithi040 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-9[125889]: 2023-07-14T19:18:33.519+0000 7fa29d625700 -1 osd.9 86 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-07-14T19:17:50.632219+0000 front 2023-07-14T19:17:50.632182+0000 (oldest deadline 2023-07-14T19:18:16.532211+0000) 2023-07-14T19:18:33.971 INFO:journalctl@ceph.osd.9.smithi040.stdout:Jul 14 19:18:33 smithi040 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-9[125889]: 2023-07-14T19:18:33.519+0000 7fa29d625700 -1 osd.9 86 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2023-07-14T19:17:57.033248+0000 front 2023-07-14T19:17:57.033262+0000 (oldest deadline 2023-07-14T19:18:22.333292+0000) 2023-07-14T19:18:33.972 INFO:journalctl@ceph.osd.9.smithi040.stdout:Jul 14 19:18:33 smithi040 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-9[125889]: 2023-07-14T19:18:33.519+0000 7fa29d625700 -1 osd.9 86 heartbeat_check: no reply from 172.21.15.40:6814 osd.3 since back 2023-07-14T19:18:07.535132+0000 front 2023-07-14T19:18:07.535107+0000 (oldest deadline 2023-07-14T19:18:30.435355+0000) 2023-07-14T19:18:33.992 INFO:journalctl@ceph.osd.7.smithi130.stdout:Jul 14 19:18:33 smithi130 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-7[113739]: 2023-07-14T19:18:33.534+0000 7f621462a700 -1 osd.7 86 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-07-14T19:17:48.674688+0000 front 2023-07-14T19:17:48.674657+0000 (oldest deadline 2023-07-14T19:18:14.574846+0000) 2023-07-14T19:18:33.992 INFO:journalctl@ceph.osd.7.smithi130.stdout:Jul 14 19:18:33 smithi130 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-7[113739]: 2023-07-14T19:18:33.534+0000 7f621462a700 -1 osd.7 86 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2023-07-14T19:17:57.475991+0000 front 2023-07-14T19:17:57.475868+0000 (oldest deadline 2023-07-14T19:18:23.376038+0000) 2023-07-14T19:18:33.992 INFO:journalctl@ceph.osd.7.smithi130.stdout:Jul 14 19:18:33 smithi130 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-7[113739]: 2023-07-14T19:18:33.534+0000 7f621462a700 -1 osd.7 86 heartbeat_check: no reply from 172.21.15.40:6814 osd.3 since back 2023-07-14T19:18:03.376649+0000 front 2023-07-14T19:18:03.376725+0000 (oldest deadline 2023-07-14T19:18:28.676747+0000) 2023-07-14T19:18:33.993 INFO:journalctl@ceph.osd.10.smithi130.stdout:Jul 14 19:18:33 smithi130 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-10[119160]: 2023-07-14T19:18:33.645+0000 7fa1fdfb9700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2023-07-14T19:17:41.727090+0000 front 2023-07-14T19:17:41.727036+0000 (oldest deadline 2023-07-14T19:18:06.427174+0000) 2023-07-14T19:18:33.993 INFO:journalctl@ceph.osd.10.smithi130.stdout:Jul 14 19:18:33 smithi130 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-10[119160]: 2023-07-14T19:18:33.645+0000 7fa1fdfb9700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-07-14T19:17:52.828799+0000 front 2023-07-14T19:17:52.828840+0000 (oldest deadline 2023-07-14T19:18:14.528908+0000) 2023-07-14T19:18:33.993 INFO:journalctl@ceph.osd.10.smithi130.stdout:Jul 14 19:18:33 smithi130 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-10[119160]: 2023-07-14T19:18:33.645+0000 7fa1fdfb9700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2023-07-14T19:17:56.730545+0000 front 2023-07-14T19:17:56.730590+0000 (oldest deadline 2023-07-14T19:18:22.630599+0000) 2023-07-14T19:18:33.993 INFO:journalctl@ceph.osd.10.smithi130.stdout:Jul 14 19:18:33 smithi130 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-10[119160]: 2023-07-14T19:18:33.645+0000 7fa1fdfb9700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.40:6814 osd.3 since back 2023-07-14T19:18:05.432244+0000 front 2023-07-14T19:18:05.432347+0000 (oldest deadline 2023-07-14T19:18:31.331990+0000) 2023-07-14T19:18:34.472 INFO:journalctl@ceph.osd.8.smithi191.stdout:Jul 14 19:18:34 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-8[115063]: 2023-07-14T19:18:34.057+0000 7f42fdd64700 -1 osd.8 86 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-07-14T19:17:52.163739+0000 front 2023-07-14T19:17:52.163632+0000 (oldest deadline 2023-07-14T19:18:17.964264+0000) 2023-07-14T19:18:34.472 INFO:journalctl@ceph.osd.8.smithi191.stdout:Jul 14 19:18:34 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-8[115063]: 2023-07-14T19:18:34.057+0000 7f42fdd64700 -1 osd.8 86 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2023-07-14T19:17:57.964682+0000 front 2023-07-14T19:17:57.964797+0000 (oldest deadline 2023-07-14T19:18:20.864905+0000) 2023-07-14T19:18:34.473 INFO:journalctl@ceph.osd.8.smithi191.stdout:Jul 14 19:18:34 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-8[115063]: 2023-07-14T19:18:34.057+0000 7f42fdd64700 -1 osd.8 86 heartbeat_check: no reply from 172.21.15.40:6814 osd.3 since back 2023-07-14T19:18:03.765725+0000 front 2023-07-14T19:18:03.765750+0000 (oldest deadline 2023-07-14T19:18:29.665790+0000) 2023-07-14T19:18:34.473 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jul 14 19:18:34 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-11[120628]: 2023-07-14T19:18:34.005+0000 7f5ef7fea700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2023-07-14T19:17:44.480647+0000 front 2023-07-14T19:17:44.480662+0000 (oldest deadline 2023-07-14T19:18:06.780727+0000) 2023-07-14T19:18:34.473 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jul 14 19:18:34 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-11[120628]: 2023-07-14T19:18:34.005+0000 7f5ef7fea700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-07-14T19:17:50.782602+0000 front 2023-07-14T19:17:50.782583+0000 (oldest deadline 2023-07-14T19:18:14.282678+0000) 2023-07-14T19:18:34.473 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jul 14 19:18:34 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-11[120628]: 2023-07-14T19:18:34.005+0000 7f5ef7fea700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2023-07-14T19:17:58.983820+0000 front 2023-07-14T19:17:58.983785+0000 (oldest deadline 2023-07-14T19:18:25.984643+0000) 2023-07-14T19:18:34.474 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jul 14 19:18:34 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-11[120628]: 2023-07-14T19:18:34.005+0000 7f5ef7fea700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.40:6814 osd.3 since back 2023-07-14T19:18:05.985177+0000 front 2023-07-14T19:18:05.985194+0000 (oldest deadline 2023-07-14T19:18:30.685202+0000) 2023-07-14T19:18:34.971 INFO:journalctl@ceph.osd.9.smithi040.stdout:Jul 14 19:18:34 smithi040 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-9[125889]: 2023-07-14T19:18:34.570+0000 7fa29d625700 -1 osd.9 86 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2023-07-14T19:17:42.530269+0000 front 2023-07-14T19:17:42.530319+0000 (oldest deadline 2023-07-14T19:18:06.531071+0000) 2023-07-14T19:18:34.971 INFO:journalctl@ceph.osd.9.smithi040.stdout:Jul 14 19:18:34 smithi040 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-9[125889]: 2023-07-14T19:18:34.570+0000 7fa29d625700 -1 osd.9 86 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-07-14T19:17:50.632219+0000 front 2023-07-14T19:17:50.632182+0000 (oldest deadline 2023-07-14T19:18:16.532211+0000) 2023-07-14T19:18:34.971 INFO:journalctl@ceph.osd.9.smithi040.stdout:Jul 14 19:18:34 smithi040 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-9[125889]: 2023-07-14T19:18:34.570+0000 7fa29d625700 -1 osd.9 86 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2023-07-14T19:17:57.033248+0000 front 2023-07-14T19:17:57.033262+0000 (oldest deadline 2023-07-14T19:18:22.333292+0000) 2023-07-14T19:18:34.972 INFO:journalctl@ceph.osd.9.smithi040.stdout:Jul 14 19:18:34 smithi040 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-9[125889]: 2023-07-14T19:18:34.570+0000 7fa29d625700 -1 osd.9 86 heartbeat_check: no reply from 172.21.15.40:6814 osd.3 since back 2023-07-14T19:18:07.535132+0000 front 2023-07-14T19:18:07.535107+0000 (oldest deadline 2023-07-14T19:18:30.435355+0000) 2023-07-14T19:18:34.992 INFO:journalctl@ceph.osd.7.smithi130.stdout:Jul 14 19:18:34 smithi130 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-7[113739]: 2023-07-14T19:18:34.547+0000 7f621462a700 -1 osd.7 86 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-07-14T19:17:48.674688+0000 front 2023-07-14T19:17:48.674657+0000 (oldest deadline 2023-07-14T19:18:14.574846+0000) 2023-07-14T19:18:34.992 INFO:journalctl@ceph.osd.7.smithi130.stdout:Jul 14 19:18:34 smithi130 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-7[113739]: 2023-07-14T19:18:34.547+0000 7f621462a700 -1 osd.7 86 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2023-07-14T19:17:57.475991+0000 front 2023-07-14T19:17:57.475868+0000 (oldest deadline 2023-07-14T19:18:23.376038+0000) 2023-07-14T19:18:34.993 INFO:journalctl@ceph.osd.7.smithi130.stdout:Jul 14 19:18:34 smithi130 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-7[113739]: 2023-07-14T19:18:34.547+0000 7f621462a700 -1 osd.7 86 heartbeat_check: no reply from 172.21.15.40:6814 osd.3 since back 2023-07-14T19:18:03.376649+0000 front 2023-07-14T19:18:03.376725+0000 (oldest deadline 2023-07-14T19:18:28.676747+0000) 2023-07-14T19:18:34.993 INFO:journalctl@ceph.osd.10.smithi130.stdout:Jul 14 19:18:34 smithi130 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-10[119160]: 2023-07-14T19:18:34.616+0000 7fa1fdfb9700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2023-07-14T19:17:41.727090+0000 front 2023-07-14T19:17:41.727036+0000 (oldest deadline 2023-07-14T19:18:06.427174+0000) 2023-07-14T19:18:34.993 INFO:journalctl@ceph.osd.10.smithi130.stdout:Jul 14 19:18:34 smithi130 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-10[119160]: 2023-07-14T19:18:34.616+0000 7fa1fdfb9700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-07-14T19:17:52.828799+0000 front 2023-07-14T19:17:52.828840+0000 (oldest deadline 2023-07-14T19:18:14.528908+0000) 2023-07-14T19:18:34.993 INFO:journalctl@ceph.osd.10.smithi130.stdout:Jul 14 19:18:34 smithi130 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-10[119160]: 2023-07-14T19:18:34.616+0000 7fa1fdfb9700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2023-07-14T19:17:56.730545+0000 front 2023-07-14T19:17:56.730590+0000 (oldest deadline 2023-07-14T19:18:22.630599+0000) 2023-07-14T19:18:34.994 INFO:journalctl@ceph.osd.10.smithi130.stdout:Jul 14 19:18:34 smithi130 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-10[119160]: 2023-07-14T19:18:34.616+0000 7fa1fdfb9700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.40:6814 osd.3 since back 2023-07-14T19:18:05.432244+0000 front 2023-07-14T19:18:05.432347+0000 (oldest deadline 2023-07-14T19:18:31.331990+0000) 2023-07-14T19:18:35.222 INFO:journalctl@ceph.osd.8.smithi191.stdout:Jul 14 19:18:35 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-8[115063]: 2023-07-14T19:18:35.104+0000 7f42fdd64700 -1 osd.8 86 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-07-14T19:17:52.163739+0000 front 2023-07-14T19:17:52.163632+0000 (oldest deadline 2023-07-14T19:18:17.964264+0000) 2023-07-14T19:18:35.222 INFO:journalctl@ceph.osd.8.smithi191.stdout:Jul 14 19:18:35 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-8[115063]: 2023-07-14T19:18:35.104+0000 7f42fdd64700 -1 osd.8 86 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2023-07-14T19:17:57.964682+0000 front 2023-07-14T19:17:57.964797+0000 (oldest deadline 2023-07-14T19:18:20.864905+0000) 2023-07-14T19:18:35.223 INFO:journalctl@ceph.osd.8.smithi191.stdout:Jul 14 19:18:35 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-8[115063]: 2023-07-14T19:18:35.104+0000 7f42fdd64700 -1 osd.8 86 heartbeat_check: no reply from 172.21.15.40:6814 osd.3 since back 2023-07-14T19:18:03.765725+0000 front 2023-07-14T19:18:03.765750+0000 (oldest deadline 2023-07-14T19:18:29.665790+0000) 2023-07-14T19:18:35.223 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jul 14 19:18:34 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-11[120628]: 2023-07-14T19:18:34.969+0000 7f5ef7fea700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2023-07-14T19:17:44.480647+0000 front 2023-07-14T19:17:44.480662+0000 (oldest deadline 2023-07-14T19:18:06.780727+0000) 2023-07-14T19:18:35.223 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jul 14 19:18:34 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-11[120628]: 2023-07-14T19:18:34.969+0000 7f5ef7fea700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-07-14T19:17:50.782602+0000 front 2023-07-14T19:17:50.782583+0000 (oldest deadline 2023-07-14T19:18:14.282678+0000) 2023-07-14T19:18:35.224 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jul 14 19:18:34 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-11[120628]: 2023-07-14T19:18:34.969+0000 7f5ef7fea700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2023-07-14T19:17:58.983820+0000 front 2023-07-14T19:17:58.983785+0000 (oldest deadline 2023-07-14T19:18:25.984643+0000) 2023-07-14T19:18:35.224 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jul 14 19:18:34 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-11[120628]: 2023-07-14T19:18:34.969+0000 7f5ef7fea700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.40:6814 osd.3 since back 2023-07-14T19:18:05.985177+0000 front 2023-07-14T19:18:05.985194+0000 (oldest deadline 2023-07-14T19:18:30.685202+0000) 2023-07-14T19:18:35.971 INFO:journalctl@ceph.osd.9.smithi040.stdout:Jul 14 19:18:35 smithi040 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-9[125889]: 2023-07-14T19:18:35.597+0000 7fa29d625700 -1 osd.9 86 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2023-07-14T19:17:42.530269+0000 front 2023-07-14T19:17:42.530319+0000 (oldest deadline 2023-07-14T19:18:06.531071+0000) 2023-07-14T19:18:35.971 INFO:journalctl@ceph.osd.9.smithi040.stdout:Jul 14 19:18:35 smithi040 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-9[125889]: 2023-07-14T19:18:35.597+0000 7fa29d625700 -1 osd.9 86 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-07-14T19:17:50.632219+0000 front 2023-07-14T19:17:50.632182+0000 (oldest deadline 2023-07-14T19:18:16.532211+0000) 2023-07-14T19:18:35.971 INFO:journalctl@ceph.osd.9.smithi040.stdout:Jul 14 19:18:35 smithi040 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-9[125889]: 2023-07-14T19:18:35.597+0000 7fa29d625700 -1 osd.9 86 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2023-07-14T19:17:57.033248+0000 front 2023-07-14T19:17:57.033262+0000 (oldest deadline 2023-07-14T19:18:22.333292+0000) 2023-07-14T19:18:35.972 INFO:journalctl@ceph.osd.9.smithi040.stdout:Jul 14 19:18:35 smithi040 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-9[125889]: 2023-07-14T19:18:35.597+0000 7fa29d625700 -1 osd.9 86 heartbeat_check: no reply from 172.21.15.40:6814 osd.3 since back 2023-07-14T19:18:07.535132+0000 front 2023-07-14T19:18:07.535107+0000 (oldest deadline 2023-07-14T19:18:30.435355+0000) 2023-07-14T19:18:35.992 INFO:journalctl@ceph.osd.7.smithi130.stdout:Jul 14 19:18:35 smithi130 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-7[113739]: 2023-07-14T19:18:35.570+0000 7f621462a700 -1 osd.7 86 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-07-14T19:17:48.674688+0000 front 2023-07-14T19:17:48.674657+0000 (oldest deadline 2023-07-14T19:18:14.574846+0000) 2023-07-14T19:18:35.992 INFO:journalctl@ceph.osd.7.smithi130.stdout:Jul 14 19:18:35 smithi130 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-7[113739]: 2023-07-14T19:18:35.570+0000 7f621462a700 -1 osd.7 86 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2023-07-14T19:17:57.475991+0000 front 2023-07-14T19:17:57.475868+0000 (oldest deadline 2023-07-14T19:18:23.376038+0000) 2023-07-14T19:18:35.992 INFO:journalctl@ceph.osd.7.smithi130.stdout:Jul 14 19:18:35 smithi130 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-7[113739]: 2023-07-14T19:18:35.570+0000 7f621462a700 -1 osd.7 86 heartbeat_check: no reply from 172.21.15.40:6814 osd.3 since back 2023-07-14T19:18:03.376649+0000 front 2023-07-14T19:18:03.376725+0000 (oldest deadline 2023-07-14T19:18:28.676747+0000) 2023-07-14T19:18:35.993 INFO:journalctl@ceph.osd.10.smithi130.stdout:Jul 14 19:18:35 smithi130 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-10[119160]: 2023-07-14T19:18:35.642+0000 7fa1fdfb9700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2023-07-14T19:17:41.727090+0000 front 2023-07-14T19:17:41.727036+0000 (oldest deadline 2023-07-14T19:18:06.427174+0000) 2023-07-14T19:18:35.993 INFO:journalctl@ceph.osd.10.smithi130.stdout:Jul 14 19:18:35 smithi130 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-10[119160]: 2023-07-14T19:18:35.642+0000 7fa1fdfb9700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-07-14T19:17:52.828799+0000 front 2023-07-14T19:17:52.828840+0000 (oldest deadline 2023-07-14T19:18:14.528908+0000) 2023-07-14T19:18:35.993 INFO:journalctl@ceph.osd.10.smithi130.stdout:Jul 14 19:18:35 smithi130 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-10[119160]: 2023-07-14T19:18:35.642+0000 7fa1fdfb9700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2023-07-14T19:17:56.730545+0000 front 2023-07-14T19:17:56.730590+0000 (oldest deadline 2023-07-14T19:18:22.630599+0000) 2023-07-14T19:18:35.993 INFO:journalctl@ceph.osd.10.smithi130.stdout:Jul 14 19:18:35 smithi130 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-10[119160]: 2023-07-14T19:18:35.642+0000 7fa1fdfb9700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.40:6814 osd.3 since back 2023-07-14T19:18:05.432244+0000 front 2023-07-14T19:18:05.432347+0000 (oldest deadline 2023-07-14T19:18:31.331990+0000) 2023-07-14T19:18:36.222 INFO:journalctl@ceph.osd.8.smithi191.stdout:Jul 14 19:18:36 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-8[115063]: 2023-07-14T19:18:36.123+0000 7f42fdd64700 -1 osd.8 86 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-07-14T19:17:52.163739+0000 front 2023-07-14T19:17:52.163632+0000 (oldest deadline 2023-07-14T19:18:17.964264+0000) 2023-07-14T19:18:36.222 INFO:journalctl@ceph.osd.8.smithi191.stdout:Jul 14 19:18:36 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-8[115063]: 2023-07-14T19:18:36.123+0000 7f42fdd64700 -1 osd.8 86 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2023-07-14T19:17:57.964682+0000 front 2023-07-14T19:17:57.964797+0000 (oldest deadline 2023-07-14T19:18:20.864905+0000) 2023-07-14T19:18:36.222 INFO:journalctl@ceph.osd.8.smithi191.stdout:Jul 14 19:18:36 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-8[115063]: 2023-07-14T19:18:36.123+0000 7f42fdd64700 -1 osd.8 86 heartbeat_check: no reply from 172.21.15.40:6814 osd.3 since back 2023-07-14T19:18:03.765725+0000 front 2023-07-14T19:18:03.765750+0000 (oldest deadline 2023-07-14T19:18:29.665790+0000) 2023-07-14T19:18:36.223 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jul 14 19:18:35 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-11[120628]: 2023-07-14T19:18:35.924+0000 7f5ef7fea700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2023-07-14T19:17:44.480647+0000 front 2023-07-14T19:17:44.480662+0000 (oldest deadline 2023-07-14T19:18:06.780727+0000) 2023-07-14T19:18:36.223 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jul 14 19:18:35 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-11[120628]: 2023-07-14T19:18:35.924+0000 7f5ef7fea700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-07-14T19:17:50.782602+0000 front 2023-07-14T19:17:50.782583+0000 (oldest deadline 2023-07-14T19:18:14.282678+0000) 2023-07-14T19:18:36.223 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jul 14 19:18:35 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-11[120628]: 2023-07-14T19:18:35.924+0000 7f5ef7fea700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2023-07-14T19:17:58.983820+0000 front 2023-07-14T19:17:58.983785+0000 (oldest deadline 2023-07-14T19:18:25.984643+0000) 2023-07-14T19:18:36.224 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jul 14 19:18:35 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-11[120628]: 2023-07-14T19:18:35.924+0000 7f5ef7fea700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.40:6814 osd.3 since back 2023-07-14T19:18:05.985177+0000 front 2023-07-14T19:18:05.985194+0000 (oldest deadline 2023-07-14T19:18:30.685202+0000) 2023-07-14T19:18:36.970 INFO:journalctl@ceph.osd.9.smithi040.stdout:Jul 14 19:18:36 smithi040 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-9[125889]: 2023-07-14T19:18:36.634+0000 7fa29d625700 -1 osd.9 86 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2023-07-14T19:17:42.530269+0000 front 2023-07-14T19:17:42.530319+0000 (oldest deadline 2023-07-14T19:18:06.531071+0000) 2023-07-14T19:18:36.971 INFO:journalctl@ceph.osd.9.smithi040.stdout:Jul 14 19:18:36 smithi040 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-9[125889]: 2023-07-14T19:18:36.634+0000 7fa29d625700 -1 osd.9 86 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-07-14T19:17:50.632219+0000 front 2023-07-14T19:17:50.632182+0000 (oldest deadline 2023-07-14T19:18:16.532211+0000) 2023-07-14T19:18:36.971 INFO:journalctl@ceph.osd.9.smithi040.stdout:Jul 14 19:18:36 smithi040 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-9[125889]: 2023-07-14T19:18:36.634+0000 7fa29d625700 -1 osd.9 86 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2023-07-14T19:17:57.033248+0000 front 2023-07-14T19:17:57.033262+0000 (oldest deadline 2023-07-14T19:18:22.333292+0000) 2023-07-14T19:18:36.971 INFO:journalctl@ceph.osd.9.smithi040.stdout:Jul 14 19:18:36 smithi040 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-9[125889]: 2023-07-14T19:18:36.634+0000 7fa29d625700 -1 osd.9 86 heartbeat_check: no reply from 172.21.15.40:6814 osd.3 since back 2023-07-14T19:18:07.535132+0000 front 2023-07-14T19:18:07.535107+0000 (oldest deadline 2023-07-14T19:18:30.435355+0000) 2023-07-14T19:18:36.992 INFO:journalctl@ceph.osd.7.smithi130.stdout:Jul 14 19:18:36 smithi130 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-7[113739]: 2023-07-14T19:18:36.528+0000 7f621462a700 -1 osd.7 86 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-07-14T19:17:48.674688+0000 front 2023-07-14T19:17:48.674657+0000 (oldest deadline 2023-07-14T19:18:14.574846+0000) 2023-07-14T19:18:36.992 INFO:journalctl@ceph.osd.7.smithi130.stdout:Jul 14 19:18:36 smithi130 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-7[113739]: 2023-07-14T19:18:36.528+0000 7f621462a700 -1 osd.7 86 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2023-07-14T19:17:57.475991+0000 front 2023-07-14T19:17:57.475868+0000 (oldest deadline 2023-07-14T19:18:23.376038+0000) 2023-07-14T19:18:36.992 INFO:journalctl@ceph.osd.7.smithi130.stdout:Jul 14 19:18:36 smithi130 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-7[113739]: 2023-07-14T19:18:36.528+0000 7f621462a700 -1 osd.7 86 heartbeat_check: no reply from 172.21.15.40:6814 osd.3 since back 2023-07-14T19:18:03.376649+0000 front 2023-07-14T19:18:03.376725+0000 (oldest deadline 2023-07-14T19:18:28.676747+0000) 2023-07-14T19:18:36.993 INFO:journalctl@ceph.osd.10.smithi130.stdout:Jul 14 19:18:36 smithi130 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-10[119160]: 2023-07-14T19:18:36.654+0000 7fa1fdfb9700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2023-07-14T19:17:41.727090+0000 front 2023-07-14T19:17:41.727036+0000 (oldest deadline 2023-07-14T19:18:06.427174+0000) 2023-07-14T19:18:36.993 INFO:journalctl@ceph.osd.10.smithi130.stdout:Jul 14 19:18:36 smithi130 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-10[119160]: 2023-07-14T19:18:36.654+0000 7fa1fdfb9700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-07-14T19:17:52.828799+0000 front 2023-07-14T19:17:52.828840+0000 (oldest deadline 2023-07-14T19:18:14.528908+0000) 2023-07-14T19:18:36.993 INFO:journalctl@ceph.osd.10.smithi130.stdout:Jul 14 19:18:36 smithi130 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-10[119160]: 2023-07-14T19:18:36.654+0000 7fa1fdfb9700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2023-07-14T19:17:56.730545+0000 front 2023-07-14T19:17:56.730590+0000 (oldest deadline 2023-07-14T19:18:22.630599+0000) 2023-07-14T19:18:36.993 INFO:journalctl@ceph.osd.10.smithi130.stdout:Jul 14 19:18:36 smithi130 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-10[119160]: 2023-07-14T19:18:36.654+0000 7fa1fdfb9700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.40:6814 osd.3 since back 2023-07-14T19:18:05.432244+0000 front 2023-07-14T19:18:05.432347+0000 (oldest deadline 2023-07-14T19:18:31.331990+0000) 2023-07-14T19:18:37.222 INFO:journalctl@ceph.osd.8.smithi191.stdout:Jul 14 19:18:37 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-8[115063]: 2023-07-14T19:18:37.130+0000 7f42fdd64700 -1 osd.8 86 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-07-14T19:17:52.163739+0000 front 2023-07-14T19:17:52.163632+0000 (oldest deadline 2023-07-14T19:18:17.964264+0000) 2023-07-14T19:18:37.222 INFO:journalctl@ceph.osd.8.smithi191.stdout:Jul 14 19:18:37 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-8[115063]: 2023-07-14T19:18:37.130+0000 7f42fdd64700 -1 osd.8 86 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2023-07-14T19:17:57.964682+0000 front 2023-07-14T19:17:57.964797+0000 (oldest deadline 2023-07-14T19:18:20.864905+0000) 2023-07-14T19:18:37.222 INFO:journalctl@ceph.osd.8.smithi191.stdout:Jul 14 19:18:37 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-8[115063]: 2023-07-14T19:18:37.130+0000 7f42fdd64700 -1 osd.8 86 heartbeat_check: no reply from 172.21.15.40:6814 osd.3 since back 2023-07-14T19:18:03.765725+0000 front 2023-07-14T19:18:03.765750+0000 (oldest deadline 2023-07-14T19:18:29.665790+0000) 2023-07-14T19:18:37.223 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jul 14 19:18:36 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-11[120628]: 2023-07-14T19:18:36.944+0000 7f5ef7fea700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2023-07-14T19:17:44.480647+0000 front 2023-07-14T19:17:44.480662+0000 (oldest deadline 2023-07-14T19:18:06.780727+0000) 2023-07-14T19:18:37.223 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jul 14 19:18:36 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-11[120628]: 2023-07-14T19:18:36.944+0000 7f5ef7fea700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-07-14T19:17:50.782602+0000 front 2023-07-14T19:17:50.782583+0000 (oldest deadline 2023-07-14T19:18:14.282678+0000) 2023-07-14T19:18:37.223 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jul 14 19:18:36 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-11[120628]: 2023-07-14T19:18:36.944+0000 7f5ef7fea700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2023-07-14T19:17:58.983820+0000 front 2023-07-14T19:17:58.983785+0000 (oldest deadline 2023-07-14T19:18:25.984643+0000) 2023-07-14T19:18:37.224 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jul 14 19:18:36 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-11[120628]: 2023-07-14T19:18:36.944+0000 7f5ef7fea700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.40:6814 osd.3 since back 2023-07-14T19:18:05.985177+0000 front 2023-07-14T19:18:05.985194+0000 (oldest deadline 2023-07-14T19:18:30.685202+0000) 2023-07-14T19:18:37.970 INFO:journalctl@ceph.osd.9.smithi040.stdout:Jul 14 19:18:37 smithi040 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-9[125889]: 2023-07-14T19:18:37.607+0000 7fa29d625700 -1 osd.9 86 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2023-07-14T19:17:42.530269+0000 front 2023-07-14T19:17:42.530319+0000 (oldest deadline 2023-07-14T19:18:06.531071+0000) 2023-07-14T19:18:37.971 INFO:journalctl@ceph.osd.9.smithi040.stdout:Jul 14 19:18:37 smithi040 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-9[125889]: 2023-07-14T19:18:37.607+0000 7fa29d625700 -1 osd.9 86 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-07-14T19:17:50.632219+0000 front 2023-07-14T19:17:50.632182+0000 (oldest deadline 2023-07-14T19:18:16.532211+0000) 2023-07-14T19:18:37.971 INFO:journalctl@ceph.osd.9.smithi040.stdout:Jul 14 19:18:37 smithi040 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-9[125889]: 2023-07-14T19:18:37.607+0000 7fa29d625700 -1 osd.9 86 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2023-07-14T19:17:57.033248+0000 front 2023-07-14T19:17:57.033262+0000 (oldest deadline 2023-07-14T19:18:22.333292+0000) 2023-07-14T19:18:37.971 INFO:journalctl@ceph.osd.9.smithi040.stdout:Jul 14 19:18:37 smithi040 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-9[125889]: 2023-07-14T19:18:37.607+0000 7fa29d625700 -1 osd.9 86 heartbeat_check: no reply from 172.21.15.40:6814 osd.3 since back 2023-07-14T19:18:07.535132+0000 front 2023-07-14T19:18:07.535107+0000 (oldest deadline 2023-07-14T19:18:30.435355+0000) 2023-07-14T19:18:37.992 INFO:journalctl@ceph.osd.7.smithi130.stdout:Jul 14 19:18:37 smithi130 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-7[113739]: 2023-07-14T19:18:37.553+0000 7f621462a700 -1 osd.7 86 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-07-14T19:17:48.674688+0000 front 2023-07-14T19:17:48.674657+0000 (oldest deadline 2023-07-14T19:18:14.574846+0000) 2023-07-14T19:18:37.992 INFO:journalctl@ceph.osd.7.smithi130.stdout:Jul 14 19:18:37 smithi130 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-7[113739]: 2023-07-14T19:18:37.553+0000 7f621462a700 -1 osd.7 86 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2023-07-14T19:17:57.475991+0000 front 2023-07-14T19:17:57.475868+0000 (oldest deadline 2023-07-14T19:18:23.376038+0000) 2023-07-14T19:18:37.992 INFO:journalctl@ceph.osd.7.smithi130.stdout:Jul 14 19:18:37 smithi130 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-7[113739]: 2023-07-14T19:18:37.553+0000 7f621462a700 -1 osd.7 86 heartbeat_check: no reply from 172.21.15.40:6814 osd.3 since back 2023-07-14T19:18:03.376649+0000 front 2023-07-14T19:18:03.376725+0000 (oldest deadline 2023-07-14T19:18:28.676747+0000) 2023-07-14T19:18:37.993 INFO:journalctl@ceph.osd.10.smithi130.stdout:Jul 14 19:18:37 smithi130 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-10[119160]: 2023-07-14T19:18:37.618+0000 7fa1fdfb9700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2023-07-14T19:17:41.727090+0000 front 2023-07-14T19:17:41.727036+0000 (oldest deadline 2023-07-14T19:18:06.427174+0000) 2023-07-14T19:18:37.993 INFO:journalctl@ceph.osd.10.smithi130.stdout:Jul 14 19:18:37 smithi130 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-10[119160]: 2023-07-14T19:18:37.618+0000 7fa1fdfb9700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-07-14T19:17:52.828799+0000 front 2023-07-14T19:17:52.828840+0000 (oldest deadline 2023-07-14T19:18:14.528908+0000) 2023-07-14T19:18:37.993 INFO:journalctl@ceph.osd.10.smithi130.stdout:Jul 14 19:18:37 smithi130 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-10[119160]: 2023-07-14T19:18:37.618+0000 7fa1fdfb9700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2023-07-14T19:17:56.730545+0000 front 2023-07-14T19:17:56.730590+0000 (oldest deadline 2023-07-14T19:18:22.630599+0000) 2023-07-14T19:18:37.993 INFO:journalctl@ceph.osd.10.smithi130.stdout:Jul 14 19:18:37 smithi130 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-10[119160]: 2023-07-14T19:18:37.618+0000 7fa1fdfb9700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.40:6814 osd.3 since back 2023-07-14T19:18:05.432244+0000 front 2023-07-14T19:18:05.432347+0000 (oldest deadline 2023-07-14T19:18:31.331990+0000) 2023-07-14T19:18:37.993 INFO:journalctl@ceph.osd.10.smithi130.stdout:Jul 14 19:18:37 smithi130 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-10[119160]: 2023-07-14T19:18:37.618+0000 7fa1fdfb9700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.130:6812 osd.4 since back 2023-07-14T19:18:14.232731+0000 front 2023-07-14T19:18:14.232687+0000 (oldest deadline 2023-07-14T19:18:37.132833+0000) 2023-07-14T19:18:38.221 INFO:journalctl@ceph.osd.8.smithi191.stdout:Jul 14 19:18:38 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-8[115063]: 2023-07-14T19:18:38.150+0000 7f42fdd64700 -1 osd.8 86 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-07-14T19:17:52.163739+0000 front 2023-07-14T19:17:52.163632+0000 (oldest deadline 2023-07-14T19:18:17.964264+0000) 2023-07-14T19:18:38.222 INFO:journalctl@ceph.osd.8.smithi191.stdout:Jul 14 19:18:38 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-8[115063]: 2023-07-14T19:18:38.150+0000 7f42fdd64700 -1 osd.8 86 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2023-07-14T19:17:57.964682+0000 front 2023-07-14T19:17:57.964797+0000 (oldest deadline 2023-07-14T19:18:20.864905+0000) 2023-07-14T19:18:38.222 INFO:journalctl@ceph.osd.8.smithi191.stdout:Jul 14 19:18:38 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-8[115063]: 2023-07-14T19:18:38.150+0000 7f42fdd64700 -1 osd.8 86 heartbeat_check: no reply from 172.21.15.40:6814 osd.3 since back 2023-07-14T19:18:03.765725+0000 front 2023-07-14T19:18:03.765750+0000 (oldest deadline 2023-07-14T19:18:29.665790+0000) 2023-07-14T19:18:38.222 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jul 14 19:18:37 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-11[120628]: 2023-07-14T19:18:37.928+0000 7f5ef7fea700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2023-07-14T19:17:44.480647+0000 front 2023-07-14T19:17:44.480662+0000 (oldest deadline 2023-07-14T19:18:06.780727+0000) 2023-07-14T19:18:38.223 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jul 14 19:18:37 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-11[120628]: 2023-07-14T19:18:37.928+0000 7f5ef7fea700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-07-14T19:17:50.782602+0000 front 2023-07-14T19:17:50.782583+0000 (oldest deadline 2023-07-14T19:18:14.282678+0000) 2023-07-14T19:18:38.223 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jul 14 19:18:37 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-11[120628]: 2023-07-14T19:18:37.928+0000 7f5ef7fea700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2023-07-14T19:17:58.983820+0000 front 2023-07-14T19:17:58.983785+0000 (oldest deadline 2023-07-14T19:18:25.984643+0000) 2023-07-14T19:18:38.223 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jul 14 19:18:37 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-11[120628]: 2023-07-14T19:18:37.928+0000 7f5ef7fea700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.40:6814 osd.3 since back 2023-07-14T19:18:05.985177+0000 front 2023-07-14T19:18:05.985194+0000 (oldest deadline 2023-07-14T19:18:30.685202+0000) 2023-07-14T19:18:38.742 INFO:journalctl@ceph.osd.7.smithi130.stdout:Jul 14 19:18:38 smithi130 bash[125674]: ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-7 2023-07-14T19:18:38.743 INFO:journalctl@ceph.osd.10.smithi130.stdout:Jul 14 19:18:38 smithi130 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-10[119160]: 2023-07-14T19:18:38.601+0000 7fa1fdfb9700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2023-07-14T19:17:41.727090+0000 front 2023-07-14T19:17:41.727036+0000 (oldest deadline 2023-07-14T19:18:06.427174+0000) 2023-07-14T19:18:38.744 INFO:journalctl@ceph.osd.10.smithi130.stdout:Jul 14 19:18:38 smithi130 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-10[119160]: 2023-07-14T19:18:38.601+0000 7fa1fdfb9700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-07-14T19:17:52.828799+0000 front 2023-07-14T19:17:52.828840+0000 (oldest deadline 2023-07-14T19:18:14.528908+0000) 2023-07-14T19:18:38.744 INFO:journalctl@ceph.osd.10.smithi130.stdout:Jul 14 19:18:38 smithi130 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-10[119160]: 2023-07-14T19:18:38.601+0000 7fa1fdfb9700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2023-07-14T19:17:56.730545+0000 front 2023-07-14T19:17:56.730590+0000 (oldest deadline 2023-07-14T19:18:22.630599+0000) 2023-07-14T19:18:38.744 INFO:journalctl@ceph.osd.10.smithi130.stdout:Jul 14 19:18:38 smithi130 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-10[119160]: 2023-07-14T19:18:38.601+0000 7fa1fdfb9700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.40:6814 osd.3 since back 2023-07-14T19:18:05.432244+0000 front 2023-07-14T19:18:05.432347+0000 (oldest deadline 2023-07-14T19:18:31.331990+0000) 2023-07-14T19:18:38.745 INFO:journalctl@ceph.osd.10.smithi130.stdout:Jul 14 19:18:38 smithi130 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-10[119160]: 2023-07-14T19:18:38.601+0000 7fa1fdfb9700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.130:6812 osd.4 since back 2023-07-14T19:18:14.232731+0000 front 2023-07-14T19:18:14.232687+0000 (oldest deadline 2023-07-14T19:18:37.132833+0000) 2023-07-14T19:18:38.970 INFO:journalctl@ceph.osd.9.smithi040.stdout:Jul 14 19:18:38 smithi040 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-9[125889]: 2023-07-14T19:18:38.607+0000 7fa29d625700 -1 osd.9 86 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2023-07-14T19:17:42.530269+0000 front 2023-07-14T19:17:42.530319+0000 (oldest deadline 2023-07-14T19:18:06.531071+0000) 2023-07-14T19:18:38.971 INFO:journalctl@ceph.osd.9.smithi040.stdout:Jul 14 19:18:38 smithi040 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-9[125889]: 2023-07-14T19:18:38.607+0000 7fa29d625700 -1 osd.9 86 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-07-14T19:17:50.632219+0000 front 2023-07-14T19:17:50.632182+0000 (oldest deadline 2023-07-14T19:18:16.532211+0000) 2023-07-14T19:18:38.971 INFO:journalctl@ceph.osd.9.smithi040.stdout:Jul 14 19:18:38 smithi040 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-9[125889]: 2023-07-14T19:18:38.607+0000 7fa29d625700 -1 osd.9 86 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2023-07-14T19:17:57.033248+0000 front 2023-07-14T19:17:57.033262+0000 (oldest deadline 2023-07-14T19:18:22.333292+0000) 2023-07-14T19:18:38.971 INFO:journalctl@ceph.osd.9.smithi040.stdout:Jul 14 19:18:38 smithi040 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-9[125889]: 2023-07-14T19:18:38.607+0000 7fa29d625700 -1 osd.9 86 heartbeat_check: no reply from 172.21.15.40:6814 osd.3 since back 2023-07-14T19:18:07.535132+0000 front 2023-07-14T19:18:07.535107+0000 (oldest deadline 2023-07-14T19:18:30.435355+0000) 2023-07-14T19:18:39.222 INFO:journalctl@ceph.osd.8.smithi191.stdout:Jul 14 19:18:39 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-8[115063]: 2023-07-14T19:18:39.159+0000 7f42fdd64700 -1 osd.8 86 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-07-14T19:17:52.163739+0000 front 2023-07-14T19:17:52.163632+0000 (oldest deadline 2023-07-14T19:18:17.964264+0000) 2023-07-14T19:18:39.222 INFO:journalctl@ceph.osd.8.smithi191.stdout:Jul 14 19:18:39 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-8[115063]: 2023-07-14T19:18:39.159+0000 7f42fdd64700 -1 osd.8 86 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2023-07-14T19:17:57.964682+0000 front 2023-07-14T19:17:57.964797+0000 (oldest deadline 2023-07-14T19:18:20.864905+0000) 2023-07-14T19:18:39.223 INFO:journalctl@ceph.osd.8.smithi191.stdout:Jul 14 19:18:39 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-8[115063]: 2023-07-14T19:18:39.159+0000 7f42fdd64700 -1 osd.8 86 heartbeat_check: no reply from 172.21.15.40:6814 osd.3 since back 2023-07-14T19:18:03.765725+0000 front 2023-07-14T19:18:03.765750+0000 (oldest deadline 2023-07-14T19:18:29.665790+0000) 2023-07-14T19:18:39.223 INFO:journalctl@ceph.osd.8.smithi191.stdout:Jul 14 19:18:39 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-8[115063]: 2023-07-14T19:18:39.159+0000 7f42fdd64700 -1 osd.8 86 heartbeat_check: no reply from 172.21.15.130:6812 osd.4 since back 2023-07-14T19:18:14.366878+0000 front 2023-07-14T19:18:14.366798+0000 (oldest deadline 2023-07-14T19:18:39.066716+0000) 2023-07-14T19:18:39.223 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jul 14 19:18:38 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-11[120628]: 2023-07-14T19:18:38.913+0000 7f5ef7fea700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2023-07-14T19:17:44.480647+0000 front 2023-07-14T19:17:44.480662+0000 (oldest deadline 2023-07-14T19:18:06.780727+0000) 2023-07-14T19:18:39.224 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jul 14 19:18:38 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-11[120628]: 2023-07-14T19:18:38.913+0000 7f5ef7fea700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-07-14T19:17:50.782602+0000 front 2023-07-14T19:17:50.782583+0000 (oldest deadline 2023-07-14T19:18:14.282678+0000) 2023-07-14T19:18:39.224 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jul 14 19:18:38 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-11[120628]: 2023-07-14T19:18:38.913+0000 7f5ef7fea700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2023-07-14T19:17:58.983820+0000 front 2023-07-14T19:17:58.983785+0000 (oldest deadline 2023-07-14T19:18:25.984643+0000) 2023-07-14T19:18:39.224 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jul 14 19:18:38 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-11[120628]: 2023-07-14T19:18:38.913+0000 7f5ef7fea700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.40:6814 osd.3 since back 2023-07-14T19:18:05.985177+0000 front 2023-07-14T19:18:05.985194+0000 (oldest deadline 2023-07-14T19:18:30.685202+0000) 2023-07-14T19:18:39.224 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jul 14 19:18:38 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-11[120628]: 2023-07-14T19:18:38.913+0000 7f5ef7fea700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.130:6812 osd.4 since back 2023-07-14T19:18:14.186314+0000 front 2023-07-14T19:18:14.186278+0000 (oldest deadline 2023-07-14T19:18:38.886251+0000) 2023-07-14T19:18:39.971 INFO:journalctl@ceph.osd.9.smithi040.stdout:Jul 14 19:18:39 smithi040 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-9[125889]: 2023-07-14T19:18:39.637+0000 7fa29d625700 -1 osd.9 86 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2023-07-14T19:17:42.530269+0000 front 2023-07-14T19:17:42.530319+0000 (oldest deadline 2023-07-14T19:18:06.531071+0000) 2023-07-14T19:18:39.971 INFO:journalctl@ceph.osd.9.smithi040.stdout:Jul 14 19:18:39 smithi040 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-9[125889]: 2023-07-14T19:18:39.637+0000 7fa29d625700 -1 osd.9 86 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-07-14T19:17:50.632219+0000 front 2023-07-14T19:17:50.632182+0000 (oldest deadline 2023-07-14T19:18:16.532211+0000) 2023-07-14T19:18:39.971 INFO:journalctl@ceph.osd.9.smithi040.stdout:Jul 14 19:18:39 smithi040 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-9[125889]: 2023-07-14T19:18:39.637+0000 7fa29d625700 -1 osd.9 86 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2023-07-14T19:17:57.033248+0000 front 2023-07-14T19:17:57.033262+0000 (oldest deadline 2023-07-14T19:18:22.333292+0000) 2023-07-14T19:18:39.971 INFO:journalctl@ceph.osd.9.smithi040.stdout:Jul 14 19:18:39 smithi040 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-9[125889]: 2023-07-14T19:18:39.637+0000 7fa29d625700 -1 osd.9 86 heartbeat_check: no reply from 172.21.15.40:6814 osd.3 since back 2023-07-14T19:18:07.535132+0000 front 2023-07-14T19:18:07.535107+0000 (oldest deadline 2023-07-14T19:18:30.435355+0000) 2023-07-14T19:18:39.972 INFO:journalctl@ceph.osd.9.smithi040.stdout:Jul 14 19:18:39 smithi040 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-9[125889]: 2023-07-14T19:18:39.637+0000 7fa29d625700 -1 osd.9 86 heartbeat_check: no reply from 172.21.15.130:6812 osd.4 since back 2023-07-14T19:18:10.435794+0000 front 2023-07-14T19:18:10.435752+0000 (oldest deadline 2023-07-14T19:18:39.236194+0000) 2023-07-14T19:18:39.992 INFO:journalctl@ceph.osd.10.smithi130.stdout:Jul 14 19:18:39 smithi130 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-10[119160]: 2023-07-14T19:18:39.600+0000 7fa1fdfb9700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2023-07-14T19:17:41.727090+0000 front 2023-07-14T19:17:41.727036+0000 (oldest deadline 2023-07-14T19:18:06.427174+0000) 2023-07-14T19:18:39.993 INFO:journalctl@ceph.osd.10.smithi130.stdout:Jul 14 19:18:39 smithi130 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-10[119160]: 2023-07-14T19:18:39.600+0000 7fa1fdfb9700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-07-14T19:17:52.828799+0000 front 2023-07-14T19:17:52.828840+0000 (oldest deadline 2023-07-14T19:18:14.528908+0000) 2023-07-14T19:18:39.993 INFO:journalctl@ceph.osd.10.smithi130.stdout:Jul 14 19:18:39 smithi130 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-10[119160]: 2023-07-14T19:18:39.600+0000 7fa1fdfb9700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2023-07-14T19:17:56.730545+0000 front 2023-07-14T19:17:56.730590+0000 (oldest deadline 2023-07-14T19:18:22.630599+0000) 2023-07-14T19:18:39.994 INFO:journalctl@ceph.osd.10.smithi130.stdout:Jul 14 19:18:39 smithi130 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-10[119160]: 2023-07-14T19:18:39.600+0000 7fa1fdfb9700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.40:6814 osd.3 since back 2023-07-14T19:18:05.432244+0000 front 2023-07-14T19:18:05.432347+0000 (oldest deadline 2023-07-14T19:18:31.331990+0000) 2023-07-14T19:18:39.994 INFO:journalctl@ceph.osd.10.smithi130.stdout:Jul 14 19:18:39 smithi130 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-10[119160]: 2023-07-14T19:18:39.600+0000 7fa1fdfb9700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.130:6812 osd.4 since back 2023-07-14T19:18:14.232731+0000 front 2023-07-14T19:18:14.232687+0000 (oldest deadline 2023-07-14T19:18:37.132833+0000) 2023-07-14T19:18:40.011 DEBUG:teuthology.orchestra.run.smithi130:> sudo pkill -f 'journalctl -f -n 0 -u ceph-adc52f04-2279-11ee-9b33-001a4aab830c@osd.7.service' 2023-07-14T19:18:40.052 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-14T19:18:40.053 INFO:tasks.cephadm.osd.7:Stopped osd.7 2023-07-14T19:18:40.053 INFO:tasks.cephadm.osd.8:Stopping osd.8... 2023-07-14T19:18:40.054 DEBUG:teuthology.orchestra.run.smithi191:> sudo systemctl stop ceph-adc52f04-2279-11ee-9b33-001a4aab830c@osd.8 2023-07-14T19:18:40.222 INFO:journalctl@ceph.osd.8.smithi191.stdout:Jul 14 19:18:40 smithi191 systemd[1]: Stopping Ceph osd.8 for adc52f04-2279-11ee-9b33-001a4aab830c... 2023-07-14T19:18:40.222 INFO:journalctl@ceph.osd.8.smithi191.stdout:Jul 14 19:18:40 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-8[115063]: 2023-07-14T19:18:40.202+0000 7f42fdd64700 -1 osd.8 86 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-07-14T19:17:52.163739+0000 front 2023-07-14T19:17:52.163632+0000 (oldest deadline 2023-07-14T19:18:17.964264+0000) 2023-07-14T19:18:40.222 INFO:journalctl@ceph.osd.8.smithi191.stdout:Jul 14 19:18:40 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-8[115063]: 2023-07-14T19:18:40.202+0000 7f42fdd64700 -1 osd.8 86 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2023-07-14T19:17:57.964682+0000 front 2023-07-14T19:17:57.964797+0000 (oldest deadline 2023-07-14T19:18:20.864905+0000) 2023-07-14T19:18:40.223 INFO:journalctl@ceph.osd.8.smithi191.stdout:Jul 14 19:18:40 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-8[115063]: 2023-07-14T19:18:40.202+0000 7f42fdd64700 -1 osd.8 86 heartbeat_check: no reply from 172.21.15.40:6814 osd.3 since back 2023-07-14T19:18:03.765725+0000 front 2023-07-14T19:18:03.765750+0000 (oldest deadline 2023-07-14T19:18:29.665790+0000) 2023-07-14T19:18:40.223 INFO:journalctl@ceph.osd.8.smithi191.stdout:Jul 14 19:18:40 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-8[115063]: 2023-07-14T19:18:40.202+0000 7f42fdd64700 -1 osd.8 86 heartbeat_check: no reply from 172.21.15.130:6812 osd.4 since back 2023-07-14T19:18:14.366878+0000 front 2023-07-14T19:18:14.366798+0000 (oldest deadline 2023-07-14T19:18:39.066716+0000) 2023-07-14T19:18:40.223 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jul 14 19:18:39 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-11[120628]: 2023-07-14T19:18:39.953+0000 7f5ef7fea700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2023-07-14T19:17:44.480647+0000 front 2023-07-14T19:17:44.480662+0000 (oldest deadline 2023-07-14T19:18:06.780727+0000) 2023-07-14T19:18:40.223 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jul 14 19:18:39 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-11[120628]: 2023-07-14T19:18:39.953+0000 7f5ef7fea700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-07-14T19:17:50.782602+0000 front 2023-07-14T19:17:50.782583+0000 (oldest deadline 2023-07-14T19:18:14.282678+0000) 2023-07-14T19:18:40.224 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jul 14 19:18:39 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-11[120628]: 2023-07-14T19:18:39.953+0000 7f5ef7fea700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2023-07-14T19:17:58.983820+0000 front 2023-07-14T19:17:58.983785+0000 (oldest deadline 2023-07-14T19:18:25.984643+0000) 2023-07-14T19:18:40.224 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jul 14 19:18:39 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-11[120628]: 2023-07-14T19:18:39.953+0000 7f5ef7fea700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.40:6814 osd.3 since back 2023-07-14T19:18:05.985177+0000 front 2023-07-14T19:18:05.985194+0000 (oldest deadline 2023-07-14T19:18:30.685202+0000) 2023-07-14T19:18:40.224 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jul 14 19:18:39 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-11[120628]: 2023-07-14T19:18:39.953+0000 7f5ef7fea700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.130:6812 osd.4 since back 2023-07-14T19:18:14.186314+0000 front 2023-07-14T19:18:14.186278+0000 (oldest deadline 2023-07-14T19:18:38.886251+0000) 2023-07-14T19:18:40.722 INFO:journalctl@ceph.osd.8.smithi191.stdout:Jul 14 19:18:40 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-8[115063]: 2023-07-14T19:18:40.415+0000 7f4302f70700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-osd -n osd.8 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-07-14T19:18:40.722 INFO:journalctl@ceph.osd.8.smithi191.stdout:Jul 14 19:18:40 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-8[115063]: 2023-07-14T19:18:40.415+0000 7f4302f70700 -1 osd.8 86 *** Got signal Terminated *** 2023-07-14T19:18:40.722 INFO:journalctl@ceph.osd.8.smithi191.stdout:Jul 14 19:18:40 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-8[115063]: 2023-07-14T19:18:40.415+0000 7f4302f70700 -1 osd.8 86 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-07-14T19:18:40.971 INFO:journalctl@ceph.osd.9.smithi040.stdout:Jul 14 19:18:40 smithi040 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-9[125889]: 2023-07-14T19:18:40.661+0000 7fa29d625700 -1 osd.9 86 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2023-07-14T19:17:42.530269+0000 front 2023-07-14T19:17:42.530319+0000 (oldest deadline 2023-07-14T19:18:06.531071+0000) 2023-07-14T19:18:40.971 INFO:journalctl@ceph.osd.9.smithi040.stdout:Jul 14 19:18:40 smithi040 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-9[125889]: 2023-07-14T19:18:40.661+0000 7fa29d625700 -1 osd.9 86 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-07-14T19:17:50.632219+0000 front 2023-07-14T19:17:50.632182+0000 (oldest deadline 2023-07-14T19:18:16.532211+0000) 2023-07-14T19:18:40.971 INFO:journalctl@ceph.osd.9.smithi040.stdout:Jul 14 19:18:40 smithi040 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-9[125889]: 2023-07-14T19:18:40.661+0000 7fa29d625700 -1 osd.9 86 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2023-07-14T19:17:57.033248+0000 front 2023-07-14T19:17:57.033262+0000 (oldest deadline 2023-07-14T19:18:22.333292+0000) 2023-07-14T19:18:40.972 INFO:journalctl@ceph.osd.9.smithi040.stdout:Jul 14 19:18:40 smithi040 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-9[125889]: 2023-07-14T19:18:40.661+0000 7fa29d625700 -1 osd.9 86 heartbeat_check: no reply from 172.21.15.40:6814 osd.3 since back 2023-07-14T19:18:07.535132+0000 front 2023-07-14T19:18:07.535107+0000 (oldest deadline 2023-07-14T19:18:30.435355+0000) 2023-07-14T19:18:40.972 INFO:journalctl@ceph.osd.9.smithi040.stdout:Jul 14 19:18:40 smithi040 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-9[125889]: 2023-07-14T19:18:40.661+0000 7fa29d625700 -1 osd.9 86 heartbeat_check: no reply from 172.21.15.130:6812 osd.4 since back 2023-07-14T19:18:10.435794+0000 front 2023-07-14T19:18:10.435752+0000 (oldest deadline 2023-07-14T19:18:39.236194+0000) 2023-07-14T19:18:40.992 INFO:journalctl@ceph.osd.10.smithi130.stdout:Jul 14 19:18:40 smithi130 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-10[119160]: 2023-07-14T19:18:40.558+0000 7fa1fdfb9700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2023-07-14T19:17:41.727090+0000 front 2023-07-14T19:17:41.727036+0000 (oldest deadline 2023-07-14T19:18:06.427174+0000) 2023-07-14T19:18:40.993 INFO:journalctl@ceph.osd.10.smithi130.stdout:Jul 14 19:18:40 smithi130 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-10[119160]: 2023-07-14T19:18:40.558+0000 7fa1fdfb9700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-07-14T19:17:52.828799+0000 front 2023-07-14T19:17:52.828840+0000 (oldest deadline 2023-07-14T19:18:14.528908+0000) 2023-07-14T19:18:40.993 INFO:journalctl@ceph.osd.10.smithi130.stdout:Jul 14 19:18:40 smithi130 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-10[119160]: 2023-07-14T19:18:40.558+0000 7fa1fdfb9700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2023-07-14T19:17:56.730545+0000 front 2023-07-14T19:17:56.730590+0000 (oldest deadline 2023-07-14T19:18:22.630599+0000) 2023-07-14T19:18:40.994 INFO:journalctl@ceph.osd.10.smithi130.stdout:Jul 14 19:18:40 smithi130 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-10[119160]: 2023-07-14T19:18:40.558+0000 7fa1fdfb9700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.40:6814 osd.3 since back 2023-07-14T19:18:05.432244+0000 front 2023-07-14T19:18:05.432347+0000 (oldest deadline 2023-07-14T19:18:31.331990+0000) 2023-07-14T19:18:40.994 INFO:journalctl@ceph.osd.10.smithi130.stdout:Jul 14 19:18:40 smithi130 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-10[119160]: 2023-07-14T19:18:40.558+0000 7fa1fdfb9700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.130:6812 osd.4 since back 2023-07-14T19:18:14.232731+0000 front 2023-07-14T19:18:14.232687+0000 (oldest deadline 2023-07-14T19:18:37.132833+0000) 2023-07-14T19:18:41.222 INFO:journalctl@ceph.osd.8.smithi191.stdout:Jul 14 19:18:41 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-8[115063]: 2023-07-14T19:18:41.199+0000 7f42fdd64700 -1 osd.8 86 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-07-14T19:17:52.163739+0000 front 2023-07-14T19:17:52.163632+0000 (oldest deadline 2023-07-14T19:18:17.964264+0000) 2023-07-14T19:18:41.222 INFO:journalctl@ceph.osd.8.smithi191.stdout:Jul 14 19:18:41 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-8[115063]: 2023-07-14T19:18:41.199+0000 7f42fdd64700 -1 osd.8 86 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2023-07-14T19:17:57.964682+0000 front 2023-07-14T19:17:57.964797+0000 (oldest deadline 2023-07-14T19:18:20.864905+0000) 2023-07-14T19:18:41.222 INFO:journalctl@ceph.osd.8.smithi191.stdout:Jul 14 19:18:41 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-8[115063]: 2023-07-14T19:18:41.199+0000 7f42fdd64700 -1 osd.8 86 heartbeat_check: no reply from 172.21.15.40:6814 osd.3 since back 2023-07-14T19:18:03.765725+0000 front 2023-07-14T19:18:03.765750+0000 (oldest deadline 2023-07-14T19:18:29.665790+0000) 2023-07-14T19:18:41.223 INFO:journalctl@ceph.osd.8.smithi191.stdout:Jul 14 19:18:41 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-8[115063]: 2023-07-14T19:18:41.199+0000 7f42fdd64700 -1 osd.8 86 heartbeat_check: no reply from 172.21.15.130:6812 osd.4 since back 2023-07-14T19:18:14.366878+0000 front 2023-07-14T19:18:14.366798+0000 (oldest deadline 2023-07-14T19:18:39.066716+0000) 2023-07-14T19:18:41.223 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jul 14 19:18:40 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-11[120628]: 2023-07-14T19:18:40.965+0000 7f5ef7fea700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2023-07-14T19:17:44.480647+0000 front 2023-07-14T19:17:44.480662+0000 (oldest deadline 2023-07-14T19:18:06.780727+0000) 2023-07-14T19:18:41.223 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jul 14 19:18:40 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-11[120628]: 2023-07-14T19:18:40.965+0000 7f5ef7fea700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-07-14T19:17:50.782602+0000 front 2023-07-14T19:17:50.782583+0000 (oldest deadline 2023-07-14T19:18:14.282678+0000) 2023-07-14T19:18:41.223 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jul 14 19:18:40 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-11[120628]: 2023-07-14T19:18:40.965+0000 7f5ef7fea700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2023-07-14T19:17:58.983820+0000 front 2023-07-14T19:17:58.983785+0000 (oldest deadline 2023-07-14T19:18:25.984643+0000) 2023-07-14T19:18:41.224 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jul 14 19:18:40 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-11[120628]: 2023-07-14T19:18:40.965+0000 7f5ef7fea700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.40:6814 osd.3 since back 2023-07-14T19:18:05.985177+0000 front 2023-07-14T19:18:05.985194+0000 (oldest deadline 2023-07-14T19:18:30.685202+0000) 2023-07-14T19:18:41.224 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jul 14 19:18:40 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-11[120628]: 2023-07-14T19:18:40.965+0000 7f5ef7fea700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.130:6812 osd.4 since back 2023-07-14T19:18:14.186314+0000 front 2023-07-14T19:18:14.186278+0000 (oldest deadline 2023-07-14T19:18:38.886251+0000) 2023-07-14T19:18:41.970 INFO:journalctl@ceph.osd.9.smithi040.stdout:Jul 14 19:18:41 smithi040 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-9[125889]: 2023-07-14T19:18:41.687+0000 7fa29d625700 -1 osd.9 86 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2023-07-14T19:17:42.530269+0000 front 2023-07-14T19:17:42.530319+0000 (oldest deadline 2023-07-14T19:18:06.531071+0000) 2023-07-14T19:18:41.971 INFO:journalctl@ceph.osd.9.smithi040.stdout:Jul 14 19:18:41 smithi040 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-9[125889]: 2023-07-14T19:18:41.687+0000 7fa29d625700 -1 osd.9 86 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-07-14T19:17:50.632219+0000 front 2023-07-14T19:17:50.632182+0000 (oldest deadline 2023-07-14T19:18:16.532211+0000) 2023-07-14T19:18:41.971 INFO:journalctl@ceph.osd.9.smithi040.stdout:Jul 14 19:18:41 smithi040 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-9[125889]: 2023-07-14T19:18:41.687+0000 7fa29d625700 -1 osd.9 86 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2023-07-14T19:17:57.033248+0000 front 2023-07-14T19:17:57.033262+0000 (oldest deadline 2023-07-14T19:18:22.333292+0000) 2023-07-14T19:18:41.971 INFO:journalctl@ceph.osd.9.smithi040.stdout:Jul 14 19:18:41 smithi040 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-9[125889]: 2023-07-14T19:18:41.687+0000 7fa29d625700 -1 osd.9 86 heartbeat_check: no reply from 172.21.15.40:6814 osd.3 since back 2023-07-14T19:18:07.535132+0000 front 2023-07-14T19:18:07.535107+0000 (oldest deadline 2023-07-14T19:18:30.435355+0000) 2023-07-14T19:18:41.972 INFO:journalctl@ceph.osd.9.smithi040.stdout:Jul 14 19:18:41 smithi040 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-9[125889]: 2023-07-14T19:18:41.687+0000 7fa29d625700 -1 osd.9 86 heartbeat_check: no reply from 172.21.15.130:6812 osd.4 since back 2023-07-14T19:18:10.435794+0000 front 2023-07-14T19:18:10.435752+0000 (oldest deadline 2023-07-14T19:18:39.236194+0000) 2023-07-14T19:18:41.992 INFO:journalctl@ceph.osd.10.smithi130.stdout:Jul 14 19:18:41 smithi130 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-10[119160]: 2023-07-14T19:18:41.533+0000 7fa1fdfb9700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2023-07-14T19:17:41.727090+0000 front 2023-07-14T19:17:41.727036+0000 (oldest deadline 2023-07-14T19:18:06.427174+0000) 2023-07-14T19:18:41.992 INFO:journalctl@ceph.osd.10.smithi130.stdout:Jul 14 19:18:41 smithi130 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-10[119160]: 2023-07-14T19:18:41.533+0000 7fa1fdfb9700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-07-14T19:17:52.828799+0000 front 2023-07-14T19:17:52.828840+0000 (oldest deadline 2023-07-14T19:18:14.528908+0000) 2023-07-14T19:18:41.992 INFO:journalctl@ceph.osd.10.smithi130.stdout:Jul 14 19:18:41 smithi130 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-10[119160]: 2023-07-14T19:18:41.533+0000 7fa1fdfb9700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2023-07-14T19:17:56.730545+0000 front 2023-07-14T19:17:56.730590+0000 (oldest deadline 2023-07-14T19:18:22.630599+0000) 2023-07-14T19:18:41.992 INFO:journalctl@ceph.osd.10.smithi130.stdout:Jul 14 19:18:41 smithi130 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-10[119160]: 2023-07-14T19:18:41.533+0000 7fa1fdfb9700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.40:6814 osd.3 since back 2023-07-14T19:18:05.432244+0000 front 2023-07-14T19:18:05.432347+0000 (oldest deadline 2023-07-14T19:18:31.331990+0000) 2023-07-14T19:18:41.993 INFO:journalctl@ceph.osd.10.smithi130.stdout:Jul 14 19:18:41 smithi130 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-10[119160]: 2023-07-14T19:18:41.533+0000 7fa1fdfb9700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.130:6812 osd.4 since back 2023-07-14T19:18:14.232731+0000 front 2023-07-14T19:18:14.232687+0000 (oldest deadline 2023-07-14T19:18:37.132833+0000) 2023-07-14T19:18:42.222 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jul 14 19:18:41 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-11[120628]: 2023-07-14T19:18:41.949+0000 7f5ef7fea700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2023-07-14T19:17:44.480647+0000 front 2023-07-14T19:17:44.480662+0000 (oldest deadline 2023-07-14T19:18:06.780727+0000) 2023-07-14T19:18:42.222 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jul 14 19:18:41 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-11[120628]: 2023-07-14T19:18:41.949+0000 7f5ef7fea700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-07-14T19:17:50.782602+0000 front 2023-07-14T19:17:50.782583+0000 (oldest deadline 2023-07-14T19:18:14.282678+0000) 2023-07-14T19:18:42.222 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jul 14 19:18:41 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-11[120628]: 2023-07-14T19:18:41.949+0000 7f5ef7fea700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2023-07-14T19:17:58.983820+0000 front 2023-07-14T19:17:58.983785+0000 (oldest deadline 2023-07-14T19:18:25.984643+0000) 2023-07-14T19:18:42.223 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jul 14 19:18:41 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-11[120628]: 2023-07-14T19:18:41.949+0000 7f5ef7fea700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.40:6814 osd.3 since back 2023-07-14T19:18:05.985177+0000 front 2023-07-14T19:18:05.985194+0000 (oldest deadline 2023-07-14T19:18:30.685202+0000) 2023-07-14T19:18:42.223 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jul 14 19:18:41 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-11[120628]: 2023-07-14T19:18:41.949+0000 7f5ef7fea700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.130:6812 osd.4 since back 2023-07-14T19:18:14.186314+0000 front 2023-07-14T19:18:14.186278+0000 (oldest deadline 2023-07-14T19:18:38.886251+0000) 2023-07-14T19:18:42.722 INFO:journalctl@ceph.osd.8.smithi191.stdout:Jul 14 19:18:42 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-8[115063]: 2023-07-14T19:18:42.238+0000 7f42fdd64700 -1 osd.8 86 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-07-14T19:17:52.163739+0000 front 2023-07-14T19:17:52.163632+0000 (oldest deadline 2023-07-14T19:18:17.964264+0000) 2023-07-14T19:18:42.722 INFO:journalctl@ceph.osd.8.smithi191.stdout:Jul 14 19:18:42 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-8[115063]: 2023-07-14T19:18:42.238+0000 7f42fdd64700 -1 osd.8 86 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2023-07-14T19:17:57.964682+0000 front 2023-07-14T19:17:57.964797+0000 (oldest deadline 2023-07-14T19:18:20.864905+0000) 2023-07-14T19:18:42.722 INFO:journalctl@ceph.osd.8.smithi191.stdout:Jul 14 19:18:42 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-8[115063]: 2023-07-14T19:18:42.238+0000 7f42fdd64700 -1 osd.8 86 heartbeat_check: no reply from 172.21.15.40:6814 osd.3 since back 2023-07-14T19:18:03.765725+0000 front 2023-07-14T19:18:03.765750+0000 (oldest deadline 2023-07-14T19:18:29.665790+0000) 2023-07-14T19:18:42.722 INFO:journalctl@ceph.osd.8.smithi191.stdout:Jul 14 19:18:42 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-8[115063]: 2023-07-14T19:18:42.238+0000 7f42fdd64700 -1 osd.8 86 heartbeat_check: no reply from 172.21.15.130:6812 osd.4 since back 2023-07-14T19:18:14.366878+0000 front 2023-07-14T19:18:14.366798+0000 (oldest deadline 2023-07-14T19:18:39.066716+0000) 2023-07-14T19:18:42.971 INFO:journalctl@ceph.osd.9.smithi040.stdout:Jul 14 19:18:42 smithi040 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-9[125889]: 2023-07-14T19:18:42.691+0000 7fa29d625700 -1 osd.9 86 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2023-07-14T19:17:42.530269+0000 front 2023-07-14T19:17:42.530319+0000 (oldest deadline 2023-07-14T19:18:06.531071+0000) 2023-07-14T19:18:42.972 INFO:journalctl@ceph.osd.9.smithi040.stdout:Jul 14 19:18:42 smithi040 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-9[125889]: 2023-07-14T19:18:42.691+0000 7fa29d625700 -1 osd.9 86 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-07-14T19:17:50.632219+0000 front 2023-07-14T19:17:50.632182+0000 (oldest deadline 2023-07-14T19:18:16.532211+0000) 2023-07-14T19:18:42.972 INFO:journalctl@ceph.osd.9.smithi040.stdout:Jul 14 19:18:42 smithi040 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-9[125889]: 2023-07-14T19:18:42.691+0000 7fa29d625700 -1 osd.9 86 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2023-07-14T19:17:57.033248+0000 front 2023-07-14T19:17:57.033262+0000 (oldest deadline 2023-07-14T19:18:22.333292+0000) 2023-07-14T19:18:42.972 INFO:journalctl@ceph.osd.9.smithi040.stdout:Jul 14 19:18:42 smithi040 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-9[125889]: 2023-07-14T19:18:42.691+0000 7fa29d625700 -1 osd.9 86 heartbeat_check: no reply from 172.21.15.40:6814 osd.3 since back 2023-07-14T19:18:07.535132+0000 front 2023-07-14T19:18:07.535107+0000 (oldest deadline 2023-07-14T19:18:30.435355+0000) 2023-07-14T19:18:42.972 INFO:journalctl@ceph.osd.9.smithi040.stdout:Jul 14 19:18:42 smithi040 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-9[125889]: 2023-07-14T19:18:42.691+0000 7fa29d625700 -1 osd.9 86 heartbeat_check: no reply from 172.21.15.130:6812 osd.4 since back 2023-07-14T19:18:10.435794+0000 front 2023-07-14T19:18:10.435752+0000 (oldest deadline 2023-07-14T19:18:39.236194+0000) 2023-07-14T19:18:42.992 INFO:journalctl@ceph.osd.10.smithi130.stdout:Jul 14 19:18:42 smithi130 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-10[119160]: 2023-07-14T19:18:42.507+0000 7fa1fdfb9700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2023-07-14T19:17:41.727090+0000 front 2023-07-14T19:17:41.727036+0000 (oldest deadline 2023-07-14T19:18:06.427174+0000) 2023-07-14T19:18:42.993 INFO:journalctl@ceph.osd.10.smithi130.stdout:Jul 14 19:18:42 smithi130 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-10[119160]: 2023-07-14T19:18:42.507+0000 7fa1fdfb9700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-07-14T19:17:52.828799+0000 front 2023-07-14T19:17:52.828840+0000 (oldest deadline 2023-07-14T19:18:14.528908+0000) 2023-07-14T19:18:42.993 INFO:journalctl@ceph.osd.10.smithi130.stdout:Jul 14 19:18:42 smithi130 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-10[119160]: 2023-07-14T19:18:42.507+0000 7fa1fdfb9700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2023-07-14T19:17:56.730545+0000 front 2023-07-14T19:17:56.730590+0000 (oldest deadline 2023-07-14T19:18:22.630599+0000) 2023-07-14T19:18:42.993 INFO:journalctl@ceph.osd.10.smithi130.stdout:Jul 14 19:18:42 smithi130 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-10[119160]: 2023-07-14T19:18:42.507+0000 7fa1fdfb9700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.40:6814 osd.3 since back 2023-07-14T19:18:05.432244+0000 front 2023-07-14T19:18:05.432347+0000 (oldest deadline 2023-07-14T19:18:31.331990+0000) 2023-07-14T19:18:42.994 INFO:journalctl@ceph.osd.10.smithi130.stdout:Jul 14 19:18:42 smithi130 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-10[119160]: 2023-07-14T19:18:42.507+0000 7fa1fdfb9700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.130:6812 osd.4 since back 2023-07-14T19:18:14.232731+0000 front 2023-07-14T19:18:14.232687+0000 (oldest deadline 2023-07-14T19:18:37.132833+0000) 2023-07-14T19:18:43.472 INFO:journalctl@ceph.osd.8.smithi191.stdout:Jul 14 19:18:43 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-8[115063]: 2023-07-14T19:18:43.223+0000 7f42fdd64700 -1 osd.8 86 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-07-14T19:17:52.163739+0000 front 2023-07-14T19:17:52.163632+0000 (oldest deadline 2023-07-14T19:18:17.964264+0000) 2023-07-14T19:18:43.472 INFO:journalctl@ceph.osd.8.smithi191.stdout:Jul 14 19:18:43 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-8[115063]: 2023-07-14T19:18:43.223+0000 7f42fdd64700 -1 osd.8 86 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2023-07-14T19:17:57.964682+0000 front 2023-07-14T19:17:57.964797+0000 (oldest deadline 2023-07-14T19:18:20.864905+0000) 2023-07-14T19:18:43.472 INFO:journalctl@ceph.osd.8.smithi191.stdout:Jul 14 19:18:43 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-8[115063]: 2023-07-14T19:18:43.223+0000 7f42fdd64700 -1 osd.8 86 heartbeat_check: no reply from 172.21.15.40:6814 osd.3 since back 2023-07-14T19:18:03.765725+0000 front 2023-07-14T19:18:03.765750+0000 (oldest deadline 2023-07-14T19:18:29.665790+0000) 2023-07-14T19:18:43.473 INFO:journalctl@ceph.osd.8.smithi191.stdout:Jul 14 19:18:43 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-8[115063]: 2023-07-14T19:18:43.223+0000 7f42fdd64700 -1 osd.8 86 heartbeat_check: no reply from 172.21.15.130:6812 osd.4 since back 2023-07-14T19:18:14.366878+0000 front 2023-07-14T19:18:14.366798+0000 (oldest deadline 2023-07-14T19:18:39.066716+0000) 2023-07-14T19:18:43.473 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jul 14 19:18:42 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-11[120628]: 2023-07-14T19:18:42.986+0000 7f5ef7fea700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2023-07-14T19:17:44.480647+0000 front 2023-07-14T19:17:44.480662+0000 (oldest deadline 2023-07-14T19:18:06.780727+0000) 2023-07-14T19:18:43.473 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jul 14 19:18:42 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-11[120628]: 2023-07-14T19:18:42.986+0000 7f5ef7fea700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-07-14T19:17:50.782602+0000 front 2023-07-14T19:17:50.782583+0000 (oldest deadline 2023-07-14T19:18:14.282678+0000) 2023-07-14T19:18:43.474 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jul 14 19:18:42 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-11[120628]: 2023-07-14T19:18:42.986+0000 7f5ef7fea700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2023-07-14T19:17:58.983820+0000 front 2023-07-14T19:17:58.983785+0000 (oldest deadline 2023-07-14T19:18:25.984643+0000) 2023-07-14T19:18:43.474 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jul 14 19:18:42 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-11[120628]: 2023-07-14T19:18:42.986+0000 7f5ef7fea700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.40:6814 osd.3 since back 2023-07-14T19:18:05.985177+0000 front 2023-07-14T19:18:05.985194+0000 (oldest deadline 2023-07-14T19:18:30.685202+0000) 2023-07-14T19:18:43.474 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jul 14 19:18:42 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-11[120628]: 2023-07-14T19:18:42.986+0000 7f5ef7fea700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.130:6812 osd.4 since back 2023-07-14T19:18:14.186314+0000 front 2023-07-14T19:18:14.186278+0000 (oldest deadline 2023-07-14T19:18:38.886251+0000) 2023-07-14T19:18:43.742 INFO:journalctl@ceph.osd.10.smithi130.stdout:Jul 14 19:18:43 smithi130 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-10[119160]: 2023-07-14T19:18:43.487+0000 7fa1fdfb9700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2023-07-14T19:17:41.727090+0000 front 2023-07-14T19:17:41.727036+0000 (oldest deadline 2023-07-14T19:18:06.427174+0000) 2023-07-14T19:18:43.742 INFO:journalctl@ceph.osd.10.smithi130.stdout:Jul 14 19:18:43 smithi130 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-10[119160]: 2023-07-14T19:18:43.487+0000 7fa1fdfb9700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-07-14T19:17:52.828799+0000 front 2023-07-14T19:17:52.828840+0000 (oldest deadline 2023-07-14T19:18:14.528908+0000) 2023-07-14T19:18:43.742 INFO:journalctl@ceph.osd.10.smithi130.stdout:Jul 14 19:18:43 smithi130 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-10[119160]: 2023-07-14T19:18:43.487+0000 7fa1fdfb9700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2023-07-14T19:17:56.730545+0000 front 2023-07-14T19:17:56.730590+0000 (oldest deadline 2023-07-14T19:18:22.630599+0000) 2023-07-14T19:18:43.743 INFO:journalctl@ceph.osd.10.smithi130.stdout:Jul 14 19:18:43 smithi130 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-10[119160]: 2023-07-14T19:18:43.487+0000 7fa1fdfb9700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.40:6814 osd.3 since back 2023-07-14T19:18:05.432244+0000 front 2023-07-14T19:18:05.432347+0000 (oldest deadline 2023-07-14T19:18:31.331990+0000) 2023-07-14T19:18:43.743 INFO:journalctl@ceph.osd.10.smithi130.stdout:Jul 14 19:18:43 smithi130 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-10[119160]: 2023-07-14T19:18:43.487+0000 7fa1fdfb9700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.130:6812 osd.4 since back 2023-07-14T19:18:14.232731+0000 front 2023-07-14T19:18:14.232687+0000 (oldest deadline 2023-07-14T19:18:37.132833+0000) 2023-07-14T19:18:43.971 INFO:journalctl@ceph.osd.9.smithi040.stdout:Jul 14 19:18:43 smithi040 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-9[125889]: 2023-07-14T19:18:43.668+0000 7fa29d625700 -1 osd.9 86 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2023-07-14T19:17:42.530269+0000 front 2023-07-14T19:17:42.530319+0000 (oldest deadline 2023-07-14T19:18:06.531071+0000) 2023-07-14T19:18:43.971 INFO:journalctl@ceph.osd.9.smithi040.stdout:Jul 14 19:18:43 smithi040 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-9[125889]: 2023-07-14T19:18:43.668+0000 7fa29d625700 -1 osd.9 86 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-07-14T19:17:50.632219+0000 front 2023-07-14T19:17:50.632182+0000 (oldest deadline 2023-07-14T19:18:16.532211+0000) 2023-07-14T19:18:43.971 INFO:journalctl@ceph.osd.9.smithi040.stdout:Jul 14 19:18:43 smithi040 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-9[125889]: 2023-07-14T19:18:43.668+0000 7fa29d625700 -1 osd.9 86 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2023-07-14T19:17:57.033248+0000 front 2023-07-14T19:17:57.033262+0000 (oldest deadline 2023-07-14T19:18:22.333292+0000) 2023-07-14T19:18:43.972 INFO:journalctl@ceph.osd.9.smithi040.stdout:Jul 14 19:18:43 smithi040 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-9[125889]: 2023-07-14T19:18:43.668+0000 7fa29d625700 -1 osd.9 86 heartbeat_check: no reply from 172.21.15.40:6814 osd.3 since back 2023-07-14T19:18:07.535132+0000 front 2023-07-14T19:18:07.535107+0000 (oldest deadline 2023-07-14T19:18:30.435355+0000) 2023-07-14T19:18:43.972 INFO:journalctl@ceph.osd.9.smithi040.stdout:Jul 14 19:18:43 smithi040 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-9[125889]: 2023-07-14T19:18:43.668+0000 7fa29d625700 -1 osd.9 86 heartbeat_check: no reply from 172.21.15.130:6812 osd.4 since back 2023-07-14T19:18:10.435794+0000 front 2023-07-14T19:18:10.435752+0000 (oldest deadline 2023-07-14T19:18:39.236194+0000) 2023-07-14T19:18:44.472 INFO:journalctl@ceph.osd.8.smithi191.stdout:Jul 14 19:18:44 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-8[115063]: 2023-07-14T19:18:44.181+0000 7f42fdd64700 -1 osd.8 86 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-07-14T19:17:52.163739+0000 front 2023-07-14T19:17:52.163632+0000 (oldest deadline 2023-07-14T19:18:17.964264+0000) 2023-07-14T19:18:44.472 INFO:journalctl@ceph.osd.8.smithi191.stdout:Jul 14 19:18:44 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-8[115063]: 2023-07-14T19:18:44.181+0000 7f42fdd64700 -1 osd.8 86 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2023-07-14T19:17:57.964682+0000 front 2023-07-14T19:17:57.964797+0000 (oldest deadline 2023-07-14T19:18:20.864905+0000) 2023-07-14T19:18:44.472 INFO:journalctl@ceph.osd.8.smithi191.stdout:Jul 14 19:18:44 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-8[115063]: 2023-07-14T19:18:44.182+0000 7f42fdd64700 -1 osd.8 86 heartbeat_check: no reply from 172.21.15.40:6814 osd.3 since back 2023-07-14T19:18:03.765725+0000 front 2023-07-14T19:18:03.765750+0000 (oldest deadline 2023-07-14T19:18:29.665790+0000) 2023-07-14T19:18:44.473 INFO:journalctl@ceph.osd.8.smithi191.stdout:Jul 14 19:18:44 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-8[115063]: 2023-07-14T19:18:44.182+0000 7f42fdd64700 -1 osd.8 86 heartbeat_check: no reply from 172.21.15.130:6812 osd.4 since back 2023-07-14T19:18:14.366878+0000 front 2023-07-14T19:18:14.366798+0000 (oldest deadline 2023-07-14T19:18:39.066716+0000) 2023-07-14T19:18:44.473 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jul 14 19:18:43 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-11[120628]: 2023-07-14T19:18:43.985+0000 7f5ef7fea700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2023-07-14T19:17:44.480647+0000 front 2023-07-14T19:17:44.480662+0000 (oldest deadline 2023-07-14T19:18:06.780727+0000) 2023-07-14T19:18:44.473 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jul 14 19:18:43 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-11[120628]: 2023-07-14T19:18:43.985+0000 7f5ef7fea700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-07-14T19:17:50.782602+0000 front 2023-07-14T19:17:50.782583+0000 (oldest deadline 2023-07-14T19:18:14.282678+0000) 2023-07-14T19:18:44.473 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jul 14 19:18:43 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-11[120628]: 2023-07-14T19:18:43.985+0000 7f5ef7fea700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2023-07-14T19:17:58.983820+0000 front 2023-07-14T19:17:58.983785+0000 (oldest deadline 2023-07-14T19:18:25.984643+0000) 2023-07-14T19:18:44.474 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jul 14 19:18:43 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-11[120628]: 2023-07-14T19:18:43.985+0000 7f5ef7fea700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.40:6814 osd.3 since back 2023-07-14T19:18:05.985177+0000 front 2023-07-14T19:18:05.985194+0000 (oldest deadline 2023-07-14T19:18:30.685202+0000) 2023-07-14T19:18:44.474 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jul 14 19:18:43 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-11[120628]: 2023-07-14T19:18:43.985+0000 7f5ef7fea700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.130:6812 osd.4 since back 2023-07-14T19:18:14.186314+0000 front 2023-07-14T19:18:14.186278+0000 (oldest deadline 2023-07-14T19:18:38.886251+0000) 2023-07-14T19:18:44.474 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jul 14 19:18:43 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-11[120628]: 2023-07-14T19:18:43.985+0000 7f5ef7fea700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.191:6812 osd.5 since back 2023-07-14T19:18:21.787193+0000 front 2023-07-14T19:18:21.787218+0000 (oldest deadline 2023-07-14T19:18:43.487275+0000) 2023-07-14T19:18:44.742 INFO:journalctl@ceph.osd.10.smithi130.stdout:Jul 14 19:18:44 smithi130 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-10[119160]: 2023-07-14T19:18:44.511+0000 7fa1fdfb9700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2023-07-14T19:17:41.727090+0000 front 2023-07-14T19:17:41.727036+0000 (oldest deadline 2023-07-14T19:18:06.427174+0000) 2023-07-14T19:18:44.743 INFO:journalctl@ceph.osd.10.smithi130.stdout:Jul 14 19:18:44 smithi130 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-10[119160]: 2023-07-14T19:18:44.511+0000 7fa1fdfb9700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-07-14T19:17:52.828799+0000 front 2023-07-14T19:17:52.828840+0000 (oldest deadline 2023-07-14T19:18:14.528908+0000) 2023-07-14T19:18:44.743 INFO:journalctl@ceph.osd.10.smithi130.stdout:Jul 14 19:18:44 smithi130 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-10[119160]: 2023-07-14T19:18:44.511+0000 7fa1fdfb9700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2023-07-14T19:17:56.730545+0000 front 2023-07-14T19:17:56.730590+0000 (oldest deadline 2023-07-14T19:18:22.630599+0000) 2023-07-14T19:18:44.743 INFO:journalctl@ceph.osd.10.smithi130.stdout:Jul 14 19:18:44 smithi130 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-10[119160]: 2023-07-14T19:18:44.511+0000 7fa1fdfb9700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.40:6814 osd.3 since back 2023-07-14T19:18:05.432244+0000 front 2023-07-14T19:18:05.432347+0000 (oldest deadline 2023-07-14T19:18:31.331990+0000) 2023-07-14T19:18:44.743 INFO:journalctl@ceph.osd.10.smithi130.stdout:Jul 14 19:18:44 smithi130 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-10[119160]: 2023-07-14T19:18:44.511+0000 7fa1fdfb9700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.130:6812 osd.4 since back 2023-07-14T19:18:14.232731+0000 front 2023-07-14T19:18:14.232687+0000 (oldest deadline 2023-07-14T19:18:37.132833+0000) 2023-07-14T19:18:44.971 INFO:journalctl@ceph.osd.9.smithi040.stdout:Jul 14 19:18:44 smithi040 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-9[125889]: 2023-07-14T19:18:44.663+0000 7fa29d625700 -1 osd.9 86 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2023-07-14T19:17:42.530269+0000 front 2023-07-14T19:17:42.530319+0000 (oldest deadline 2023-07-14T19:18:06.531071+0000) 2023-07-14T19:18:44.971 INFO:journalctl@ceph.osd.9.smithi040.stdout:Jul 14 19:18:44 smithi040 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-9[125889]: 2023-07-14T19:18:44.663+0000 7fa29d625700 -1 osd.9 86 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-07-14T19:17:50.632219+0000 front 2023-07-14T19:17:50.632182+0000 (oldest deadline 2023-07-14T19:18:16.532211+0000) 2023-07-14T19:18:44.971 INFO:journalctl@ceph.osd.9.smithi040.stdout:Jul 14 19:18:44 smithi040 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-9[125889]: 2023-07-14T19:18:44.663+0000 7fa29d625700 -1 osd.9 86 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2023-07-14T19:17:57.033248+0000 front 2023-07-14T19:17:57.033262+0000 (oldest deadline 2023-07-14T19:18:22.333292+0000) 2023-07-14T19:18:44.972 INFO:journalctl@ceph.osd.9.smithi040.stdout:Jul 14 19:18:44 smithi040 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-9[125889]: 2023-07-14T19:18:44.663+0000 7fa29d625700 -1 osd.9 86 heartbeat_check: no reply from 172.21.15.40:6814 osd.3 since back 2023-07-14T19:18:07.535132+0000 front 2023-07-14T19:18:07.535107+0000 (oldest deadline 2023-07-14T19:18:30.435355+0000) 2023-07-14T19:18:44.972 INFO:journalctl@ceph.osd.9.smithi040.stdout:Jul 14 19:18:44 smithi040 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-9[125889]: 2023-07-14T19:18:44.663+0000 7fa29d625700 -1 osd.9 86 heartbeat_check: no reply from 172.21.15.130:6812 osd.4 since back 2023-07-14T19:18:10.435794+0000 front 2023-07-14T19:18:10.435752+0000 (oldest deadline 2023-07-14T19:18:39.236194+0000) 2023-07-14T19:18:45.222 INFO:journalctl@ceph.osd.8.smithi191.stdout:Jul 14 19:18:45 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-8[115063]: 2023-07-14T19:18:45.156+0000 7f42fdd64700 -1 osd.8 86 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-07-14T19:17:52.163739+0000 front 2023-07-14T19:17:52.163632+0000 (oldest deadline 2023-07-14T19:18:17.964264+0000) 2023-07-14T19:18:45.222 INFO:journalctl@ceph.osd.8.smithi191.stdout:Jul 14 19:18:45 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-8[115063]: 2023-07-14T19:18:45.156+0000 7f42fdd64700 -1 osd.8 86 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2023-07-14T19:17:57.964682+0000 front 2023-07-14T19:17:57.964797+0000 (oldest deadline 2023-07-14T19:18:20.864905+0000) 2023-07-14T19:18:45.222 INFO:journalctl@ceph.osd.8.smithi191.stdout:Jul 14 19:18:45 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-8[115063]: 2023-07-14T19:18:45.156+0000 7f42fdd64700 -1 osd.8 86 heartbeat_check: no reply from 172.21.15.40:6814 osd.3 since back 2023-07-14T19:18:03.765725+0000 front 2023-07-14T19:18:03.765750+0000 (oldest deadline 2023-07-14T19:18:29.665790+0000) 2023-07-14T19:18:45.223 INFO:journalctl@ceph.osd.8.smithi191.stdout:Jul 14 19:18:45 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-8[115063]: 2023-07-14T19:18:45.156+0000 7f42fdd64700 -1 osd.8 86 heartbeat_check: no reply from 172.21.15.130:6812 osd.4 since back 2023-07-14T19:18:14.366878+0000 front 2023-07-14T19:18:14.366798+0000 (oldest deadline 2023-07-14T19:18:39.066716+0000) 2023-07-14T19:18:45.223 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jul 14 19:18:44 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-11[120628]: 2023-07-14T19:18:44.943+0000 7f5ef7fea700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2023-07-14T19:17:44.480647+0000 front 2023-07-14T19:17:44.480662+0000 (oldest deadline 2023-07-14T19:18:06.780727+0000) 2023-07-14T19:18:45.223 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jul 14 19:18:44 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-11[120628]: 2023-07-14T19:18:44.943+0000 7f5ef7fea700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-07-14T19:17:50.782602+0000 front 2023-07-14T19:17:50.782583+0000 (oldest deadline 2023-07-14T19:18:14.282678+0000) 2023-07-14T19:18:45.223 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jul 14 19:18:44 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-11[120628]: 2023-07-14T19:18:44.943+0000 7f5ef7fea700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2023-07-14T19:17:58.983820+0000 front 2023-07-14T19:17:58.983785+0000 (oldest deadline 2023-07-14T19:18:25.984643+0000) 2023-07-14T19:18:45.224 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jul 14 19:18:44 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-11[120628]: 2023-07-14T19:18:44.943+0000 7f5ef7fea700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.40:6814 osd.3 since back 2023-07-14T19:18:05.985177+0000 front 2023-07-14T19:18:05.985194+0000 (oldest deadline 2023-07-14T19:18:30.685202+0000) 2023-07-14T19:18:45.224 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jul 14 19:18:44 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-11[120628]: 2023-07-14T19:18:44.943+0000 7f5ef7fea700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.130:6812 osd.4 since back 2023-07-14T19:18:14.186314+0000 front 2023-07-14T19:18:14.186278+0000 (oldest deadline 2023-07-14T19:18:38.886251+0000) 2023-07-14T19:18:45.224 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jul 14 19:18:44 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-11[120628]: 2023-07-14T19:18:44.943+0000 7f5ef7fea700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.191:6812 osd.5 since back 2023-07-14T19:18:21.787193+0000 front 2023-07-14T19:18:21.787218+0000 (oldest deadline 2023-07-14T19:18:43.487275+0000) 2023-07-14T19:18:45.742 INFO:journalctl@ceph.osd.10.smithi130.stdout:Jul 14 19:18:45 smithi130 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-10[119160]: 2023-07-14T19:18:45.472+0000 7fa1fdfb9700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2023-07-14T19:17:41.727090+0000 front 2023-07-14T19:17:41.727036+0000 (oldest deadline 2023-07-14T19:18:06.427174+0000) 2023-07-14T19:18:45.742 INFO:journalctl@ceph.osd.10.smithi130.stdout:Jul 14 19:18:45 smithi130 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-10[119160]: 2023-07-14T19:18:45.472+0000 7fa1fdfb9700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-07-14T19:17:52.828799+0000 front 2023-07-14T19:17:52.828840+0000 (oldest deadline 2023-07-14T19:18:14.528908+0000) 2023-07-14T19:18:45.742 INFO:journalctl@ceph.osd.10.smithi130.stdout:Jul 14 19:18:45 smithi130 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-10[119160]: 2023-07-14T19:18:45.472+0000 7fa1fdfb9700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2023-07-14T19:17:56.730545+0000 front 2023-07-14T19:17:56.730590+0000 (oldest deadline 2023-07-14T19:18:22.630599+0000) 2023-07-14T19:18:45.743 INFO:journalctl@ceph.osd.10.smithi130.stdout:Jul 14 19:18:45 smithi130 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-10[119160]: 2023-07-14T19:18:45.472+0000 7fa1fdfb9700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.40:6814 osd.3 since back 2023-07-14T19:18:05.432244+0000 front 2023-07-14T19:18:05.432347+0000 (oldest deadline 2023-07-14T19:18:31.331990+0000) 2023-07-14T19:18:45.743 INFO:journalctl@ceph.osd.10.smithi130.stdout:Jul 14 19:18:45 smithi130 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-10[119160]: 2023-07-14T19:18:45.472+0000 7fa1fdfb9700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.130:6812 osd.4 since back 2023-07-14T19:18:14.232731+0000 front 2023-07-14T19:18:14.232687+0000 (oldest deadline 2023-07-14T19:18:37.132833+0000) 2023-07-14T19:18:45.970 INFO:journalctl@ceph.osd.9.smithi040.stdout:Jul 14 19:18:45 smithi040 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-9[125889]: 2023-07-14T19:18:45.627+0000 7fa29d625700 -1 osd.9 86 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2023-07-14T19:17:42.530269+0000 front 2023-07-14T19:17:42.530319+0000 (oldest deadline 2023-07-14T19:18:06.531071+0000) 2023-07-14T19:18:45.971 INFO:journalctl@ceph.osd.9.smithi040.stdout:Jul 14 19:18:45 smithi040 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-9[125889]: 2023-07-14T19:18:45.627+0000 7fa29d625700 -1 osd.9 86 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-07-14T19:17:50.632219+0000 front 2023-07-14T19:17:50.632182+0000 (oldest deadline 2023-07-14T19:18:16.532211+0000) 2023-07-14T19:18:45.971 INFO:journalctl@ceph.osd.9.smithi040.stdout:Jul 14 19:18:45 smithi040 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-9[125889]: 2023-07-14T19:18:45.627+0000 7fa29d625700 -1 osd.9 86 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2023-07-14T19:17:57.033248+0000 front 2023-07-14T19:17:57.033262+0000 (oldest deadline 2023-07-14T19:18:22.333292+0000) 2023-07-14T19:18:45.972 INFO:journalctl@ceph.osd.9.smithi040.stdout:Jul 14 19:18:45 smithi040 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-9[125889]: 2023-07-14T19:18:45.627+0000 7fa29d625700 -1 osd.9 86 heartbeat_check: no reply from 172.21.15.40:6814 osd.3 since back 2023-07-14T19:18:07.535132+0000 front 2023-07-14T19:18:07.535107+0000 (oldest deadline 2023-07-14T19:18:30.435355+0000) 2023-07-14T19:18:45.972 INFO:journalctl@ceph.osd.9.smithi040.stdout:Jul 14 19:18:45 smithi040 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-9[125889]: 2023-07-14T19:18:45.627+0000 7fa29d625700 -1 osd.9 86 heartbeat_check: no reply from 172.21.15.130:6812 osd.4 since back 2023-07-14T19:18:10.435794+0000 front 2023-07-14T19:18:10.435752+0000 (oldest deadline 2023-07-14T19:18:39.236194+0000) 2023-07-14T19:18:46.139 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jul 14 19:18:45 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-11[120628]: 2023-07-14T19:18:45.933+0000 7f5ef7fea700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2023-07-14T19:17:44.480647+0000 front 2023-07-14T19:17:44.480662+0000 (oldest deadline 2023-07-14T19:18:06.780727+0000) 2023-07-14T19:18:46.140 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jul 14 19:18:45 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-11[120628]: 2023-07-14T19:18:45.933+0000 7f5ef7fea700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-07-14T19:17:50.782602+0000 front 2023-07-14T19:17:50.782583+0000 (oldest deadline 2023-07-14T19:18:14.282678+0000) 2023-07-14T19:18:46.140 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jul 14 19:18:45 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-11[120628]: 2023-07-14T19:18:45.933+0000 7f5ef7fea700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2023-07-14T19:17:58.983820+0000 front 2023-07-14T19:17:58.983785+0000 (oldest deadline 2023-07-14T19:18:25.984643+0000) 2023-07-14T19:18:46.140 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jul 14 19:18:45 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-11[120628]: 2023-07-14T19:18:45.933+0000 7f5ef7fea700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.40:6814 osd.3 since back 2023-07-14T19:18:05.985177+0000 front 2023-07-14T19:18:05.985194+0000 (oldest deadline 2023-07-14T19:18:30.685202+0000) 2023-07-14T19:18:46.141 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jul 14 19:18:45 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-11[120628]: 2023-07-14T19:18:45.933+0000 7f5ef7fea700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.130:6812 osd.4 since back 2023-07-14T19:18:14.186314+0000 front 2023-07-14T19:18:14.186278+0000 (oldest deadline 2023-07-14T19:18:38.886251+0000) 2023-07-14T19:18:46.141 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jul 14 19:18:45 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-11[120628]: 2023-07-14T19:18:45.933+0000 7f5ef7fea700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.191:6812 osd.5 since back 2023-07-14T19:18:21.787193+0000 front 2023-07-14T19:18:21.787218+0000 (oldest deadline 2023-07-14T19:18:43.487275+0000) 2023-07-14T19:18:46.471 INFO:journalctl@ceph.osd.8.smithi191.stdout:Jul 14 19:18:46 smithi191 bash[127381]: ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-8 2023-07-14T19:18:46.742 INFO:journalctl@ceph.osd.10.smithi130.stdout:Jul 14 19:18:46 smithi130 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-10[119160]: 2023-07-14T19:18:46.461+0000 7fa1fdfb9700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2023-07-14T19:17:41.727090+0000 front 2023-07-14T19:17:41.727036+0000 (oldest deadline 2023-07-14T19:18:06.427174+0000) 2023-07-14T19:18:46.742 INFO:journalctl@ceph.osd.10.smithi130.stdout:Jul 14 19:18:46 smithi130 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-10[119160]: 2023-07-14T19:18:46.461+0000 7fa1fdfb9700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-07-14T19:17:52.828799+0000 front 2023-07-14T19:17:52.828840+0000 (oldest deadline 2023-07-14T19:18:14.528908+0000) 2023-07-14T19:18:46.743 INFO:journalctl@ceph.osd.10.smithi130.stdout:Jul 14 19:18:46 smithi130 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-10[119160]: 2023-07-14T19:18:46.461+0000 7fa1fdfb9700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2023-07-14T19:17:56.730545+0000 front 2023-07-14T19:17:56.730590+0000 (oldest deadline 2023-07-14T19:18:22.630599+0000) 2023-07-14T19:18:46.743 INFO:journalctl@ceph.osd.10.smithi130.stdout:Jul 14 19:18:46 smithi130 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-10[119160]: 2023-07-14T19:18:46.461+0000 7fa1fdfb9700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.40:6814 osd.3 since back 2023-07-14T19:18:05.432244+0000 front 2023-07-14T19:18:05.432347+0000 (oldest deadline 2023-07-14T19:18:31.331990+0000) 2023-07-14T19:18:46.743 INFO:journalctl@ceph.osd.10.smithi130.stdout:Jul 14 19:18:46 smithi130 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-10[119160]: 2023-07-14T19:18:46.461+0000 7fa1fdfb9700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.130:6812 osd.4 since back 2023-07-14T19:18:14.232731+0000 front 2023-07-14T19:18:14.232687+0000 (oldest deadline 2023-07-14T19:18:37.132833+0000) 2023-07-14T19:18:46.971 INFO:journalctl@ceph.osd.9.smithi040.stdout:Jul 14 19:18:46 smithi040 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-9[125889]: 2023-07-14T19:18:46.673+0000 7fa29d625700 -1 osd.9 86 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2023-07-14T19:17:42.530269+0000 front 2023-07-14T19:17:42.530319+0000 (oldest deadline 2023-07-14T19:18:06.531071+0000) 2023-07-14T19:18:46.971 INFO:journalctl@ceph.osd.9.smithi040.stdout:Jul 14 19:18:46 smithi040 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-9[125889]: 2023-07-14T19:18:46.673+0000 7fa29d625700 -1 osd.9 86 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-07-14T19:17:50.632219+0000 front 2023-07-14T19:17:50.632182+0000 (oldest deadline 2023-07-14T19:18:16.532211+0000) 2023-07-14T19:18:46.971 INFO:journalctl@ceph.osd.9.smithi040.stdout:Jul 14 19:18:46 smithi040 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-9[125889]: 2023-07-14T19:18:46.673+0000 7fa29d625700 -1 osd.9 86 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2023-07-14T19:17:57.033248+0000 front 2023-07-14T19:17:57.033262+0000 (oldest deadline 2023-07-14T19:18:22.333292+0000) 2023-07-14T19:18:46.972 INFO:journalctl@ceph.osd.9.smithi040.stdout:Jul 14 19:18:46 smithi040 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-9[125889]: 2023-07-14T19:18:46.673+0000 7fa29d625700 -1 osd.9 86 heartbeat_check: no reply from 172.21.15.40:6814 osd.3 since back 2023-07-14T19:18:07.535132+0000 front 2023-07-14T19:18:07.535107+0000 (oldest deadline 2023-07-14T19:18:30.435355+0000) 2023-07-14T19:18:46.972 INFO:journalctl@ceph.osd.9.smithi040.stdout:Jul 14 19:18:46 smithi040 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-9[125889]: 2023-07-14T19:18:46.673+0000 7fa29d625700 -1 osd.9 86 heartbeat_check: no reply from 172.21.15.130:6812 osd.4 since back 2023-07-14T19:18:10.435794+0000 front 2023-07-14T19:18:10.435752+0000 (oldest deadline 2023-07-14T19:18:39.236194+0000) 2023-07-14T19:18:47.132 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jul 14 19:18:46 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-11[120628]: 2023-07-14T19:18:46.893+0000 7f5ef7fea700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2023-07-14T19:17:44.480647+0000 front 2023-07-14T19:17:44.480662+0000 (oldest deadline 2023-07-14T19:18:06.780727+0000) 2023-07-14T19:18:47.133 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jul 14 19:18:46 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-11[120628]: 2023-07-14T19:18:46.893+0000 7f5ef7fea700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-07-14T19:17:50.782602+0000 front 2023-07-14T19:17:50.782583+0000 (oldest deadline 2023-07-14T19:18:14.282678+0000) 2023-07-14T19:18:47.133 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jul 14 19:18:46 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-11[120628]: 2023-07-14T19:18:46.893+0000 7f5ef7fea700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2023-07-14T19:17:58.983820+0000 front 2023-07-14T19:17:58.983785+0000 (oldest deadline 2023-07-14T19:18:25.984643+0000) 2023-07-14T19:18:47.133 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jul 14 19:18:46 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-11[120628]: 2023-07-14T19:18:46.893+0000 7f5ef7fea700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.40:6814 osd.3 since back 2023-07-14T19:18:05.985177+0000 front 2023-07-14T19:18:05.985194+0000 (oldest deadline 2023-07-14T19:18:30.685202+0000) 2023-07-14T19:18:47.133 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jul 14 19:18:46 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-11[120628]: 2023-07-14T19:18:46.893+0000 7f5ef7fea700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.130:6812 osd.4 since back 2023-07-14T19:18:14.186314+0000 front 2023-07-14T19:18:14.186278+0000 (oldest deadline 2023-07-14T19:18:38.886251+0000) 2023-07-14T19:18:47.133 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jul 14 19:18:46 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-11[120628]: 2023-07-14T19:18:46.893+0000 7f5ef7fea700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.191:6812 osd.5 since back 2023-07-14T19:18:21.787193+0000 front 2023-07-14T19:18:21.787218+0000 (oldest deadline 2023-07-14T19:18:43.487275+0000) 2023-07-14T19:18:47.453 DEBUG:teuthology.orchestra.run.smithi191:> sudo pkill -f 'journalctl -f -n 0 -u ceph-adc52f04-2279-11ee-9b33-001a4aab830c@osd.8.service' 2023-07-14T19:18:47.493 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-14T19:18:47.494 INFO:tasks.cephadm.osd.8:Stopped osd.8 2023-07-14T19:18:47.494 INFO:tasks.cephadm.osd.9:Stopping osd.9... 2023-07-14T19:18:47.494 DEBUG:teuthology.orchestra.run.smithi040:> sudo systemctl stop ceph-adc52f04-2279-11ee-9b33-001a4aab830c@osd.9 2023-07-14T19:18:47.742 INFO:journalctl@ceph.osd.10.smithi130.stdout:Jul 14 19:18:47 smithi130 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-10[119160]: 2023-07-14T19:18:47.422+0000 7fa1fdfb9700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2023-07-14T19:17:41.727090+0000 front 2023-07-14T19:17:41.727036+0000 (oldest deadline 2023-07-14T19:18:06.427174+0000) 2023-07-14T19:18:47.742 INFO:journalctl@ceph.osd.10.smithi130.stdout:Jul 14 19:18:47 smithi130 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-10[119160]: 2023-07-14T19:18:47.422+0000 7fa1fdfb9700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-07-14T19:17:52.828799+0000 front 2023-07-14T19:17:52.828840+0000 (oldest deadline 2023-07-14T19:18:14.528908+0000) 2023-07-14T19:18:47.742 INFO:journalctl@ceph.osd.10.smithi130.stdout:Jul 14 19:18:47 smithi130 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-10[119160]: 2023-07-14T19:18:47.422+0000 7fa1fdfb9700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2023-07-14T19:17:56.730545+0000 front 2023-07-14T19:17:56.730590+0000 (oldest deadline 2023-07-14T19:18:22.630599+0000) 2023-07-14T19:18:47.743 INFO:journalctl@ceph.osd.10.smithi130.stdout:Jul 14 19:18:47 smithi130 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-10[119160]: 2023-07-14T19:18:47.422+0000 7fa1fdfb9700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.40:6814 osd.3 since back 2023-07-14T19:18:05.432244+0000 front 2023-07-14T19:18:05.432347+0000 (oldest deadline 2023-07-14T19:18:31.331990+0000) 2023-07-14T19:18:47.743 INFO:journalctl@ceph.osd.10.smithi130.stdout:Jul 14 19:18:47 smithi130 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-10[119160]: 2023-07-14T19:18:47.422+0000 7fa1fdfb9700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.130:6812 osd.4 since back 2023-07-14T19:18:14.232731+0000 front 2023-07-14T19:18:14.232687+0000 (oldest deadline 2023-07-14T19:18:37.132833+0000) 2023-07-14T19:18:47.743 INFO:journalctl@ceph.osd.10.smithi130.stdout:Jul 14 19:18:47 smithi130 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-10[119160]: 2023-07-14T19:18:47.422+0000 7fa1fdfb9700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.191:6812 osd.5 since back 2023-07-14T19:18:22.635891+0000 front 2023-07-14T19:18:22.636012+0000 (oldest deadline 2023-07-14T19:18:46.636128+0000) 2023-07-14T19:18:47.865 INFO:journalctl@ceph.osd.9.smithi040.stdout:Jul 14 19:18:47 smithi040 systemd[1]: Stopping Ceph osd.9 for adc52f04-2279-11ee-9b33-001a4aab830c... 2023-07-14T19:18:47.865 INFO:journalctl@ceph.osd.9.smithi040.stdout:Jul 14 19:18:47 smithi040 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-9[125889]: 2023-07-14T19:18:47.631+0000 7fa29d625700 -1 osd.9 86 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2023-07-14T19:17:42.530269+0000 front 2023-07-14T19:17:42.530319+0000 (oldest deadline 2023-07-14T19:18:06.531071+0000) 2023-07-14T19:18:47.865 INFO:journalctl@ceph.osd.9.smithi040.stdout:Jul 14 19:18:47 smithi040 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-9[125889]: 2023-07-14T19:18:47.631+0000 7fa29d625700 -1 osd.9 86 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-07-14T19:17:50.632219+0000 front 2023-07-14T19:17:50.632182+0000 (oldest deadline 2023-07-14T19:18:16.532211+0000) 2023-07-14T19:18:47.866 INFO:journalctl@ceph.osd.9.smithi040.stdout:Jul 14 19:18:47 smithi040 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-9[125889]: 2023-07-14T19:18:47.631+0000 7fa29d625700 -1 osd.9 86 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2023-07-14T19:17:57.033248+0000 front 2023-07-14T19:17:57.033262+0000 (oldest deadline 2023-07-14T19:18:22.333292+0000) 2023-07-14T19:18:47.866 INFO:journalctl@ceph.osd.9.smithi040.stdout:Jul 14 19:18:47 smithi040 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-9[125889]: 2023-07-14T19:18:47.631+0000 7fa29d625700 -1 osd.9 86 heartbeat_check: no reply from 172.21.15.40:6814 osd.3 since back 2023-07-14T19:18:07.535132+0000 front 2023-07-14T19:18:07.535107+0000 (oldest deadline 2023-07-14T19:18:30.435355+0000) 2023-07-14T19:18:47.866 INFO:journalctl@ceph.osd.9.smithi040.stdout:Jul 14 19:18:47 smithi040 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-9[125889]: 2023-07-14T19:18:47.631+0000 7fa29d625700 -1 osd.9 86 heartbeat_check: no reply from 172.21.15.130:6812 osd.4 since back 2023-07-14T19:18:10.435794+0000 front 2023-07-14T19:18:10.435752+0000 (oldest deadline 2023-07-14T19:18:39.236194+0000) 2023-07-14T19:18:48.220 INFO:journalctl@ceph.osd.9.smithi040.stdout:Jul 14 19:18:47 smithi040 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-9[125889]: 2023-07-14T19:18:47.863+0000 7fa2a102e700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-osd -n osd.9 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-07-14T19:18:48.221 INFO:journalctl@ceph.osd.9.smithi040.stdout:Jul 14 19:18:47 smithi040 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-9[125889]: 2023-07-14T19:18:47.863+0000 7fa2a102e700 -1 osd.9 86 *** Got signal Terminated *** 2023-07-14T19:18:48.221 INFO:journalctl@ceph.osd.9.smithi040.stdout:Jul 14 19:18:47 smithi040 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-9[125889]: 2023-07-14T19:18:47.863+0000 7fa2a102e700 -1 osd.9 86 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-07-14T19:18:48.222 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jul 14 19:18:47 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-11[120628]: 2023-07-14T19:18:47.903+0000 7f5ef7fea700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2023-07-14T19:17:44.480647+0000 front 2023-07-14T19:17:44.480662+0000 (oldest deadline 2023-07-14T19:18:06.780727+0000) 2023-07-14T19:18:48.222 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jul 14 19:18:47 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-11[120628]: 2023-07-14T19:18:47.903+0000 7f5ef7fea700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-07-14T19:17:50.782602+0000 front 2023-07-14T19:17:50.782583+0000 (oldest deadline 2023-07-14T19:18:14.282678+0000) 2023-07-14T19:18:48.222 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jul 14 19:18:47 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-11[120628]: 2023-07-14T19:18:47.903+0000 7f5ef7fea700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2023-07-14T19:17:58.983820+0000 front 2023-07-14T19:17:58.983785+0000 (oldest deadline 2023-07-14T19:18:25.984643+0000) 2023-07-14T19:18:48.222 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jul 14 19:18:47 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-11[120628]: 2023-07-14T19:18:47.903+0000 7f5ef7fea700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.40:6814 osd.3 since back 2023-07-14T19:18:05.985177+0000 front 2023-07-14T19:18:05.985194+0000 (oldest deadline 2023-07-14T19:18:30.685202+0000) 2023-07-14T19:18:48.223 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jul 14 19:18:47 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-11[120628]: 2023-07-14T19:18:47.903+0000 7f5ef7fea700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.130:6812 osd.4 since back 2023-07-14T19:18:14.186314+0000 front 2023-07-14T19:18:14.186278+0000 (oldest deadline 2023-07-14T19:18:38.886251+0000) 2023-07-14T19:18:48.223 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jul 14 19:18:47 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-11[120628]: 2023-07-14T19:18:47.903+0000 7f5ef7fea700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.191:6812 osd.5 since back 2023-07-14T19:18:21.787193+0000 front 2023-07-14T19:18:21.787218+0000 (oldest deadline 2023-07-14T19:18:43.487275+0000) 2023-07-14T19:18:48.742 INFO:journalctl@ceph.osd.10.smithi130.stdout:Jul 14 19:18:48 smithi130 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-10[119160]: 2023-07-14T19:18:48.470+0000 7fa1fdfb9700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2023-07-14T19:17:41.727090+0000 front 2023-07-14T19:17:41.727036+0000 (oldest deadline 2023-07-14T19:18:06.427174+0000) 2023-07-14T19:18:48.743 INFO:journalctl@ceph.osd.10.smithi130.stdout:Jul 14 19:18:48 smithi130 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-10[119160]: 2023-07-14T19:18:48.470+0000 7fa1fdfb9700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-07-14T19:17:52.828799+0000 front 2023-07-14T19:17:52.828840+0000 (oldest deadline 2023-07-14T19:18:14.528908+0000) 2023-07-14T19:18:48.743 INFO:journalctl@ceph.osd.10.smithi130.stdout:Jul 14 19:18:48 smithi130 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-10[119160]: 2023-07-14T19:18:48.470+0000 7fa1fdfb9700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2023-07-14T19:17:56.730545+0000 front 2023-07-14T19:17:56.730590+0000 (oldest deadline 2023-07-14T19:18:22.630599+0000) 2023-07-14T19:18:48.744 INFO:journalctl@ceph.osd.10.smithi130.stdout:Jul 14 19:18:48 smithi130 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-10[119160]: 2023-07-14T19:18:48.470+0000 7fa1fdfb9700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.40:6814 osd.3 since back 2023-07-14T19:18:05.432244+0000 front 2023-07-14T19:18:05.432347+0000 (oldest deadline 2023-07-14T19:18:31.331990+0000) 2023-07-14T19:18:48.744 INFO:journalctl@ceph.osd.10.smithi130.stdout:Jul 14 19:18:48 smithi130 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-10[119160]: 2023-07-14T19:18:48.470+0000 7fa1fdfb9700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.130:6812 osd.4 since back 2023-07-14T19:18:14.232731+0000 front 2023-07-14T19:18:14.232687+0000 (oldest deadline 2023-07-14T19:18:37.132833+0000) 2023-07-14T19:18:48.744 INFO:journalctl@ceph.osd.10.smithi130.stdout:Jul 14 19:18:48 smithi130 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-10[119160]: 2023-07-14T19:18:48.470+0000 7fa1fdfb9700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.191:6812 osd.5 since back 2023-07-14T19:18:22.635891+0000 front 2023-07-14T19:18:22.636012+0000 (oldest deadline 2023-07-14T19:18:46.636128+0000) 2023-07-14T19:18:48.970 INFO:journalctl@ceph.osd.9.smithi040.stdout:Jul 14 19:18:48 smithi040 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-9[125889]: 2023-07-14T19:18:48.647+0000 7fa29d625700 -1 osd.9 86 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2023-07-14T19:17:42.530269+0000 front 2023-07-14T19:17:42.530319+0000 (oldest deadline 2023-07-14T19:18:06.531071+0000) 2023-07-14T19:18:48.971 INFO:journalctl@ceph.osd.9.smithi040.stdout:Jul 14 19:18:48 smithi040 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-9[125889]: 2023-07-14T19:18:48.647+0000 7fa29d625700 -1 osd.9 86 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-07-14T19:17:50.632219+0000 front 2023-07-14T19:17:50.632182+0000 (oldest deadline 2023-07-14T19:18:16.532211+0000) 2023-07-14T19:18:48.971 INFO:journalctl@ceph.osd.9.smithi040.stdout:Jul 14 19:18:48 smithi040 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-9[125889]: 2023-07-14T19:18:48.647+0000 7fa29d625700 -1 osd.9 86 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2023-07-14T19:17:57.033248+0000 front 2023-07-14T19:17:57.033262+0000 (oldest deadline 2023-07-14T19:18:22.333292+0000) 2023-07-14T19:18:48.971 INFO:journalctl@ceph.osd.9.smithi040.stdout:Jul 14 19:18:48 smithi040 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-9[125889]: 2023-07-14T19:18:48.647+0000 7fa29d625700 -1 osd.9 86 heartbeat_check: no reply from 172.21.15.40:6814 osd.3 since back 2023-07-14T19:18:07.535132+0000 front 2023-07-14T19:18:07.535107+0000 (oldest deadline 2023-07-14T19:18:30.435355+0000) 2023-07-14T19:18:48.971 INFO:journalctl@ceph.osd.9.smithi040.stdout:Jul 14 19:18:48 smithi040 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-9[125889]: 2023-07-14T19:18:48.647+0000 7fa29d625700 -1 osd.9 86 heartbeat_check: no reply from 172.21.15.130:6812 osd.4 since back 2023-07-14T19:18:10.435794+0000 front 2023-07-14T19:18:10.435752+0000 (oldest deadline 2023-07-14T19:18:39.236194+0000) 2023-07-14T19:18:48.972 INFO:journalctl@ceph.osd.9.smithi040.stdout:Jul 14 19:18:48 smithi040 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-9[125889]: 2023-07-14T19:18:48.647+0000 7fa29d625700 -1 osd.9 86 heartbeat_check: no reply from 172.21.15.191:6812 osd.5 since back 2023-07-14T19:18:21.437664+0000 front 2023-07-14T19:18:21.437586+0000 (oldest deadline 2023-07-14T19:18:47.837788+0000) 2023-07-14T19:18:49.221 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jul 14 19:18:48 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-11[120628]: 2023-07-14T19:18:48.938+0000 7f5ef7fea700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2023-07-14T19:17:44.480647+0000 front 2023-07-14T19:17:44.480662+0000 (oldest deadline 2023-07-14T19:18:06.780727+0000) 2023-07-14T19:18:49.222 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jul 14 19:18:48 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-11[120628]: 2023-07-14T19:18:48.938+0000 7f5ef7fea700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-07-14T19:17:50.782602+0000 front 2023-07-14T19:17:50.782583+0000 (oldest deadline 2023-07-14T19:18:14.282678+0000) 2023-07-14T19:18:49.222 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jul 14 19:18:48 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-11[120628]: 2023-07-14T19:18:48.938+0000 7f5ef7fea700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2023-07-14T19:17:58.983820+0000 front 2023-07-14T19:17:58.983785+0000 (oldest deadline 2023-07-14T19:18:25.984643+0000) 2023-07-14T19:18:49.222 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jul 14 19:18:48 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-11[120628]: 2023-07-14T19:18:48.938+0000 7f5ef7fea700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.40:6814 osd.3 since back 2023-07-14T19:18:05.985177+0000 front 2023-07-14T19:18:05.985194+0000 (oldest deadline 2023-07-14T19:18:30.685202+0000) 2023-07-14T19:18:49.223 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jul 14 19:18:48 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-11[120628]: 2023-07-14T19:18:48.938+0000 7f5ef7fea700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.130:6812 osd.4 since back 2023-07-14T19:18:14.186314+0000 front 2023-07-14T19:18:14.186278+0000 (oldest deadline 2023-07-14T19:18:38.886251+0000) 2023-07-14T19:18:49.223 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jul 14 19:18:48 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-11[120628]: 2023-07-14T19:18:48.938+0000 7f5ef7fea700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.191:6812 osd.5 since back 2023-07-14T19:18:21.787193+0000 front 2023-07-14T19:18:21.787218+0000 (oldest deadline 2023-07-14T19:18:43.487275+0000) 2023-07-14T19:18:49.970 INFO:journalctl@ceph.osd.9.smithi040.stdout:Jul 14 19:18:49 smithi040 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-9[125889]: 2023-07-14T19:18:49.683+0000 7fa29d625700 -1 osd.9 86 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2023-07-14T19:17:42.530269+0000 front 2023-07-14T19:17:42.530319+0000 (oldest deadline 2023-07-14T19:18:06.531071+0000) 2023-07-14T19:18:49.971 INFO:journalctl@ceph.osd.9.smithi040.stdout:Jul 14 19:18:49 smithi040 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-9[125889]: 2023-07-14T19:18:49.683+0000 7fa29d625700 -1 osd.9 86 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-07-14T19:17:50.632219+0000 front 2023-07-14T19:17:50.632182+0000 (oldest deadline 2023-07-14T19:18:16.532211+0000) 2023-07-14T19:18:49.971 INFO:journalctl@ceph.osd.9.smithi040.stdout:Jul 14 19:18:49 smithi040 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-9[125889]: 2023-07-14T19:18:49.683+0000 7fa29d625700 -1 osd.9 86 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2023-07-14T19:17:57.033248+0000 front 2023-07-14T19:17:57.033262+0000 (oldest deadline 2023-07-14T19:18:22.333292+0000) 2023-07-14T19:18:49.971 INFO:journalctl@ceph.osd.9.smithi040.stdout:Jul 14 19:18:49 smithi040 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-9[125889]: 2023-07-14T19:18:49.683+0000 7fa29d625700 -1 osd.9 86 heartbeat_check: no reply from 172.21.15.40:6814 osd.3 since back 2023-07-14T19:18:07.535132+0000 front 2023-07-14T19:18:07.535107+0000 (oldest deadline 2023-07-14T19:18:30.435355+0000) 2023-07-14T19:18:49.972 INFO:journalctl@ceph.osd.9.smithi040.stdout:Jul 14 19:18:49 smithi040 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-9[125889]: 2023-07-14T19:18:49.683+0000 7fa29d625700 -1 osd.9 86 heartbeat_check: no reply from 172.21.15.130:6812 osd.4 since back 2023-07-14T19:18:10.435794+0000 front 2023-07-14T19:18:10.435752+0000 (oldest deadline 2023-07-14T19:18:39.236194+0000) 2023-07-14T19:18:49.972 INFO:journalctl@ceph.osd.9.smithi040.stdout:Jul 14 19:18:49 smithi040 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-9[125889]: 2023-07-14T19:18:49.683+0000 7fa29d625700 -1 osd.9 86 heartbeat_check: no reply from 172.21.15.191:6812 osd.5 since back 2023-07-14T19:18:21.437664+0000 front 2023-07-14T19:18:21.437586+0000 (oldest deadline 2023-07-14T19:18:47.837788+0000) 2023-07-14T19:18:49.992 INFO:journalctl@ceph.osd.10.smithi130.stdout:Jul 14 19:18:49 smithi130 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-10[119160]: 2023-07-14T19:18:49.492+0000 7fa1fdfb9700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2023-07-14T19:17:41.727090+0000 front 2023-07-14T19:17:41.727036+0000 (oldest deadline 2023-07-14T19:18:06.427174+0000) 2023-07-14T19:18:49.992 INFO:journalctl@ceph.osd.10.smithi130.stdout:Jul 14 19:18:49 smithi130 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-10[119160]: 2023-07-14T19:18:49.492+0000 7fa1fdfb9700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-07-14T19:17:52.828799+0000 front 2023-07-14T19:17:52.828840+0000 (oldest deadline 2023-07-14T19:18:14.528908+0000) 2023-07-14T19:18:49.993 INFO:journalctl@ceph.osd.10.smithi130.stdout:Jul 14 19:18:49 smithi130 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-10[119160]: 2023-07-14T19:18:49.492+0000 7fa1fdfb9700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2023-07-14T19:17:56.730545+0000 front 2023-07-14T19:17:56.730590+0000 (oldest deadline 2023-07-14T19:18:22.630599+0000) 2023-07-14T19:18:49.993 INFO:journalctl@ceph.osd.10.smithi130.stdout:Jul 14 19:18:49 smithi130 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-10[119160]: 2023-07-14T19:18:49.492+0000 7fa1fdfb9700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.40:6814 osd.3 since back 2023-07-14T19:18:05.432244+0000 front 2023-07-14T19:18:05.432347+0000 (oldest deadline 2023-07-14T19:18:31.331990+0000) 2023-07-14T19:18:49.993 INFO:journalctl@ceph.osd.10.smithi130.stdout:Jul 14 19:18:49 smithi130 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-10[119160]: 2023-07-14T19:18:49.492+0000 7fa1fdfb9700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.130:6812 osd.4 since back 2023-07-14T19:18:14.232731+0000 front 2023-07-14T19:18:14.232687+0000 (oldest deadline 2023-07-14T19:18:37.132833+0000) 2023-07-14T19:18:49.993 INFO:journalctl@ceph.osd.10.smithi130.stdout:Jul 14 19:18:49 smithi130 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-10[119160]: 2023-07-14T19:18:49.492+0000 7fa1fdfb9700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.191:6812 osd.5 since back 2023-07-14T19:18:22.635891+0000 front 2023-07-14T19:18:22.636012+0000 (oldest deadline 2023-07-14T19:18:46.636128+0000) 2023-07-14T19:18:50.222 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jul 14 19:18:49 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-11[120628]: 2023-07-14T19:18:49.948+0000 7f5ef7fea700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2023-07-14T19:17:44.480647+0000 front 2023-07-14T19:17:44.480662+0000 (oldest deadline 2023-07-14T19:18:06.780727+0000) 2023-07-14T19:18:50.222 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jul 14 19:18:49 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-11[120628]: 2023-07-14T19:18:49.948+0000 7f5ef7fea700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-07-14T19:17:50.782602+0000 front 2023-07-14T19:17:50.782583+0000 (oldest deadline 2023-07-14T19:18:14.282678+0000) 2023-07-14T19:18:50.222 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jul 14 19:18:49 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-11[120628]: 2023-07-14T19:18:49.948+0000 7f5ef7fea700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2023-07-14T19:17:58.983820+0000 front 2023-07-14T19:17:58.983785+0000 (oldest deadline 2023-07-14T19:18:25.984643+0000) 2023-07-14T19:18:50.223 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jul 14 19:18:49 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-11[120628]: 2023-07-14T19:18:49.948+0000 7f5ef7fea700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.40:6814 osd.3 since back 2023-07-14T19:18:05.985177+0000 front 2023-07-14T19:18:05.985194+0000 (oldest deadline 2023-07-14T19:18:30.685202+0000) 2023-07-14T19:18:50.223 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jul 14 19:18:49 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-11[120628]: 2023-07-14T19:18:49.948+0000 7f5ef7fea700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.130:6812 osd.4 since back 2023-07-14T19:18:14.186314+0000 front 2023-07-14T19:18:14.186278+0000 (oldest deadline 2023-07-14T19:18:38.886251+0000) 2023-07-14T19:18:50.223 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jul 14 19:18:49 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-11[120628]: 2023-07-14T19:18:49.948+0000 7f5ef7fea700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.191:6812 osd.5 since back 2023-07-14T19:18:21.787193+0000 front 2023-07-14T19:18:21.787218+0000 (oldest deadline 2023-07-14T19:18:43.487275+0000) 2023-07-14T19:18:50.992 INFO:journalctl@ceph.osd.10.smithi130.stdout:Jul 14 19:18:50 smithi130 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-10[119160]: 2023-07-14T19:18:50.510+0000 7fa1fdfb9700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2023-07-14T19:17:41.727090+0000 front 2023-07-14T19:17:41.727036+0000 (oldest deadline 2023-07-14T19:18:06.427174+0000) 2023-07-14T19:18:50.992 INFO:journalctl@ceph.osd.10.smithi130.stdout:Jul 14 19:18:50 smithi130 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-10[119160]: 2023-07-14T19:18:50.510+0000 7fa1fdfb9700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-07-14T19:17:52.828799+0000 front 2023-07-14T19:17:52.828840+0000 (oldest deadline 2023-07-14T19:18:14.528908+0000) 2023-07-14T19:18:50.993 INFO:journalctl@ceph.osd.10.smithi130.stdout:Jul 14 19:18:50 smithi130 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-10[119160]: 2023-07-14T19:18:50.510+0000 7fa1fdfb9700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2023-07-14T19:17:56.730545+0000 front 2023-07-14T19:17:56.730590+0000 (oldest deadline 2023-07-14T19:18:22.630599+0000) 2023-07-14T19:18:50.993 INFO:journalctl@ceph.osd.10.smithi130.stdout:Jul 14 19:18:50 smithi130 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-10[119160]: 2023-07-14T19:18:50.510+0000 7fa1fdfb9700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.40:6814 osd.3 since back 2023-07-14T19:18:05.432244+0000 front 2023-07-14T19:18:05.432347+0000 (oldest deadline 2023-07-14T19:18:31.331990+0000) 2023-07-14T19:18:50.993 INFO:journalctl@ceph.osd.10.smithi130.stdout:Jul 14 19:18:50 smithi130 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-10[119160]: 2023-07-14T19:18:50.510+0000 7fa1fdfb9700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.130:6812 osd.4 since back 2023-07-14T19:18:14.232731+0000 front 2023-07-14T19:18:14.232687+0000 (oldest deadline 2023-07-14T19:18:37.132833+0000) 2023-07-14T19:18:50.993 INFO:journalctl@ceph.osd.10.smithi130.stdout:Jul 14 19:18:50 smithi130 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-10[119160]: 2023-07-14T19:18:50.510+0000 7fa1fdfb9700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.191:6812 osd.5 since back 2023-07-14T19:18:22.635891+0000 front 2023-07-14T19:18:22.636012+0000 (oldest deadline 2023-07-14T19:18:46.636128+0000) 2023-07-14T19:18:51.221 INFO:journalctl@ceph.osd.9.smithi040.stdout:Jul 14 19:18:50 smithi040 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-9[125889]: 2023-07-14T19:18:50.719+0000 7fa29d625700 -1 osd.9 86 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2023-07-14T19:17:42.530269+0000 front 2023-07-14T19:17:42.530319+0000 (oldest deadline 2023-07-14T19:18:06.531071+0000) 2023-07-14T19:18:51.221 INFO:journalctl@ceph.osd.9.smithi040.stdout:Jul 14 19:18:50 smithi040 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-9[125889]: 2023-07-14T19:18:50.719+0000 7fa29d625700 -1 osd.9 86 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-07-14T19:17:50.632219+0000 front 2023-07-14T19:17:50.632182+0000 (oldest deadline 2023-07-14T19:18:16.532211+0000) 2023-07-14T19:18:51.221 INFO:journalctl@ceph.osd.9.smithi040.stdout:Jul 14 19:18:50 smithi040 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-9[125889]: 2023-07-14T19:18:50.719+0000 7fa29d625700 -1 osd.9 86 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2023-07-14T19:17:57.033248+0000 front 2023-07-14T19:17:57.033262+0000 (oldest deadline 2023-07-14T19:18:22.333292+0000) 2023-07-14T19:18:51.222 INFO:journalctl@ceph.osd.9.smithi040.stdout:Jul 14 19:18:50 smithi040 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-9[125889]: 2023-07-14T19:18:50.719+0000 7fa29d625700 -1 osd.9 86 heartbeat_check: no reply from 172.21.15.40:6814 osd.3 since back 2023-07-14T19:18:07.535132+0000 front 2023-07-14T19:18:07.535107+0000 (oldest deadline 2023-07-14T19:18:30.435355+0000) 2023-07-14T19:18:51.222 INFO:journalctl@ceph.osd.9.smithi040.stdout:Jul 14 19:18:50 smithi040 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-9[125889]: 2023-07-14T19:18:50.720+0000 7fa29d625700 -1 osd.9 86 heartbeat_check: no reply from 172.21.15.130:6812 osd.4 since back 2023-07-14T19:18:10.435794+0000 front 2023-07-14T19:18:10.435752+0000 (oldest deadline 2023-07-14T19:18:39.236194+0000) 2023-07-14T19:18:51.222 INFO:journalctl@ceph.osd.9.smithi040.stdout:Jul 14 19:18:50 smithi040 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-9[125889]: 2023-07-14T19:18:50.720+0000 7fa29d625700 -1 osd.9 86 heartbeat_check: no reply from 172.21.15.191:6812 osd.5 since back 2023-07-14T19:18:21.437664+0000 front 2023-07-14T19:18:21.437586+0000 (oldest deadline 2023-07-14T19:18:47.837788+0000) 2023-07-14T19:18:51.472 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jul 14 19:18:50 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-11[120628]: 2023-07-14T19:18:50.984+0000 7f5ef7fea700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2023-07-14T19:17:44.480647+0000 front 2023-07-14T19:17:44.480662+0000 (oldest deadline 2023-07-14T19:18:06.780727+0000) 2023-07-14T19:18:51.472 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jul 14 19:18:50 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-11[120628]: 2023-07-14T19:18:50.984+0000 7f5ef7fea700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-07-14T19:17:50.782602+0000 front 2023-07-14T19:17:50.782583+0000 (oldest deadline 2023-07-14T19:18:14.282678+0000) 2023-07-14T19:18:51.472 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jul 14 19:18:50 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-11[120628]: 2023-07-14T19:18:50.984+0000 7f5ef7fea700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2023-07-14T19:17:58.983820+0000 front 2023-07-14T19:17:58.983785+0000 (oldest deadline 2023-07-14T19:18:25.984643+0000) 2023-07-14T19:18:51.473 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jul 14 19:18:50 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-11[120628]: 2023-07-14T19:18:50.984+0000 7f5ef7fea700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.40:6814 osd.3 since back 2023-07-14T19:18:05.985177+0000 front 2023-07-14T19:18:05.985194+0000 (oldest deadline 2023-07-14T19:18:30.685202+0000) 2023-07-14T19:18:51.473 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jul 14 19:18:50 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-11[120628]: 2023-07-14T19:18:50.984+0000 7f5ef7fea700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.130:6812 osd.4 since back 2023-07-14T19:18:14.186314+0000 front 2023-07-14T19:18:14.186278+0000 (oldest deadline 2023-07-14T19:18:38.886251+0000) 2023-07-14T19:18:51.473 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jul 14 19:18:50 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-11[120628]: 2023-07-14T19:18:50.984+0000 7f5ef7fea700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.191:6812 osd.5 since back 2023-07-14T19:18:21.787193+0000 front 2023-07-14T19:18:21.787218+0000 (oldest deadline 2023-07-14T19:18:43.487275+0000) 2023-07-14T19:18:51.742 INFO:journalctl@ceph.osd.10.smithi130.stdout:Jul 14 19:18:51 smithi130 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-10[119160]: 2023-07-14T19:18:51.463+0000 7fa1fdfb9700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2023-07-14T19:17:41.727090+0000 front 2023-07-14T19:17:41.727036+0000 (oldest deadline 2023-07-14T19:18:06.427174+0000) 2023-07-14T19:18:51.742 INFO:journalctl@ceph.osd.10.smithi130.stdout:Jul 14 19:18:51 smithi130 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-10[119160]: 2023-07-14T19:18:51.463+0000 7fa1fdfb9700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-07-14T19:17:52.828799+0000 front 2023-07-14T19:17:52.828840+0000 (oldest deadline 2023-07-14T19:18:14.528908+0000) 2023-07-14T19:18:51.743 INFO:journalctl@ceph.osd.10.smithi130.stdout:Jul 14 19:18:51 smithi130 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-10[119160]: 2023-07-14T19:18:51.463+0000 7fa1fdfb9700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2023-07-14T19:17:56.730545+0000 front 2023-07-14T19:17:56.730590+0000 (oldest deadline 2023-07-14T19:18:22.630599+0000) 2023-07-14T19:18:51.743 INFO:journalctl@ceph.osd.10.smithi130.stdout:Jul 14 19:18:51 smithi130 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-10[119160]: 2023-07-14T19:18:51.463+0000 7fa1fdfb9700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.40:6814 osd.3 since back 2023-07-14T19:18:05.432244+0000 front 2023-07-14T19:18:05.432347+0000 (oldest deadline 2023-07-14T19:18:31.331990+0000) 2023-07-14T19:18:51.743 INFO:journalctl@ceph.osd.10.smithi130.stdout:Jul 14 19:18:51 smithi130 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-10[119160]: 2023-07-14T19:18:51.463+0000 7fa1fdfb9700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.130:6812 osd.4 since back 2023-07-14T19:18:14.232731+0000 front 2023-07-14T19:18:14.232687+0000 (oldest deadline 2023-07-14T19:18:37.132833+0000) 2023-07-14T19:18:51.743 INFO:journalctl@ceph.osd.10.smithi130.stdout:Jul 14 19:18:51 smithi130 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-10[119160]: 2023-07-14T19:18:51.463+0000 7fa1fdfb9700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.191:6812 osd.5 since back 2023-07-14T19:18:22.635891+0000 front 2023-07-14T19:18:22.636012+0000 (oldest deadline 2023-07-14T19:18:46.636128+0000) 2023-07-14T19:18:51.971 INFO:journalctl@ceph.osd.9.smithi040.stdout:Jul 14 19:18:51 smithi040 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-9[125889]: 2023-07-14T19:18:51.683+0000 7fa29d625700 -1 osd.9 86 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2023-07-14T19:17:42.530269+0000 front 2023-07-14T19:17:42.530319+0000 (oldest deadline 2023-07-14T19:18:06.531071+0000) 2023-07-14T19:18:51.972 INFO:journalctl@ceph.osd.9.smithi040.stdout:Jul 14 19:18:51 smithi040 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-9[125889]: 2023-07-14T19:18:51.683+0000 7fa29d625700 -1 osd.9 86 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-07-14T19:17:50.632219+0000 front 2023-07-14T19:17:50.632182+0000 (oldest deadline 2023-07-14T19:18:16.532211+0000) 2023-07-14T19:18:51.972 INFO:journalctl@ceph.osd.9.smithi040.stdout:Jul 14 19:18:51 smithi040 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-9[125889]: 2023-07-14T19:18:51.683+0000 7fa29d625700 -1 osd.9 86 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2023-07-14T19:17:57.033248+0000 front 2023-07-14T19:17:57.033262+0000 (oldest deadline 2023-07-14T19:18:22.333292+0000) 2023-07-14T19:18:51.973 INFO:journalctl@ceph.osd.9.smithi040.stdout:Jul 14 19:18:51 smithi040 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-9[125889]: 2023-07-14T19:18:51.683+0000 7fa29d625700 -1 osd.9 86 heartbeat_check: no reply from 172.21.15.40:6814 osd.3 since back 2023-07-14T19:18:07.535132+0000 front 2023-07-14T19:18:07.535107+0000 (oldest deadline 2023-07-14T19:18:30.435355+0000) 2023-07-14T19:18:51.973 INFO:journalctl@ceph.osd.9.smithi040.stdout:Jul 14 19:18:51 smithi040 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-9[125889]: 2023-07-14T19:18:51.683+0000 7fa29d625700 -1 osd.9 86 heartbeat_check: no reply from 172.21.15.130:6812 osd.4 since back 2023-07-14T19:18:10.435794+0000 front 2023-07-14T19:18:10.435752+0000 (oldest deadline 2023-07-14T19:18:39.236194+0000) 2023-07-14T19:18:51.974 INFO:journalctl@ceph.osd.9.smithi040.stdout:Jul 14 19:18:51 smithi040 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-9[125889]: 2023-07-14T19:18:51.683+0000 7fa29d625700 -1 osd.9 86 heartbeat_check: no reply from 172.21.15.191:6812 osd.5 since back 2023-07-14T19:18:21.437664+0000 front 2023-07-14T19:18:21.437586+0000 (oldest deadline 2023-07-14T19:18:47.837788+0000) 2023-07-14T19:18:52.472 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jul 14 19:18:51 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-11[120628]: 2023-07-14T19:18:51.974+0000 7f5ef7fea700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2023-07-14T19:17:44.480647+0000 front 2023-07-14T19:17:44.480662+0000 (oldest deadline 2023-07-14T19:18:06.780727+0000) 2023-07-14T19:18:52.472 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jul 14 19:18:51 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-11[120628]: 2023-07-14T19:18:51.974+0000 7f5ef7fea700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-07-14T19:17:50.782602+0000 front 2023-07-14T19:17:50.782583+0000 (oldest deadline 2023-07-14T19:18:14.282678+0000) 2023-07-14T19:18:52.472 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jul 14 19:18:51 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-11[120628]: 2023-07-14T19:18:51.974+0000 7f5ef7fea700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2023-07-14T19:17:58.983820+0000 front 2023-07-14T19:17:58.983785+0000 (oldest deadline 2023-07-14T19:18:25.984643+0000) 2023-07-14T19:18:52.473 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jul 14 19:18:51 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-11[120628]: 2023-07-14T19:18:51.974+0000 7f5ef7fea700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.40:6814 osd.3 since back 2023-07-14T19:18:05.985177+0000 front 2023-07-14T19:18:05.985194+0000 (oldest deadline 2023-07-14T19:18:30.685202+0000) 2023-07-14T19:18:52.473 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jul 14 19:18:51 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-11[120628]: 2023-07-14T19:18:51.974+0000 7f5ef7fea700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.130:6812 osd.4 since back 2023-07-14T19:18:14.186314+0000 front 2023-07-14T19:18:14.186278+0000 (oldest deadline 2023-07-14T19:18:38.886251+0000) 2023-07-14T19:18:52.473 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jul 14 19:18:51 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-11[120628]: 2023-07-14T19:18:51.974+0000 7f5ef7fea700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.191:6812 osd.5 since back 2023-07-14T19:18:21.787193+0000 front 2023-07-14T19:18:21.787218+0000 (oldest deadline 2023-07-14T19:18:43.487275+0000) 2023-07-14T19:18:52.970 INFO:journalctl@ceph.osd.9.smithi040.stdout:Jul 14 19:18:52 smithi040 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-9[125889]: 2023-07-14T19:18:52.672+0000 7fa29d625700 -1 osd.9 86 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2023-07-14T19:17:42.530269+0000 front 2023-07-14T19:17:42.530319+0000 (oldest deadline 2023-07-14T19:18:06.531071+0000) 2023-07-14T19:18:52.971 INFO:journalctl@ceph.osd.9.smithi040.stdout:Jul 14 19:18:52 smithi040 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-9[125889]: 2023-07-14T19:18:52.672+0000 7fa29d625700 -1 osd.9 86 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-07-14T19:17:50.632219+0000 front 2023-07-14T19:17:50.632182+0000 (oldest deadline 2023-07-14T19:18:16.532211+0000) 2023-07-14T19:18:52.971 INFO:journalctl@ceph.osd.9.smithi040.stdout:Jul 14 19:18:52 smithi040 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-9[125889]: 2023-07-14T19:18:52.672+0000 7fa29d625700 -1 osd.9 86 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2023-07-14T19:17:57.033248+0000 front 2023-07-14T19:17:57.033262+0000 (oldest deadline 2023-07-14T19:18:22.333292+0000) 2023-07-14T19:18:52.971 INFO:journalctl@ceph.osd.9.smithi040.stdout:Jul 14 19:18:52 smithi040 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-9[125889]: 2023-07-14T19:18:52.672+0000 7fa29d625700 -1 osd.9 86 heartbeat_check: no reply from 172.21.15.40:6814 osd.3 since back 2023-07-14T19:18:07.535132+0000 front 2023-07-14T19:18:07.535107+0000 (oldest deadline 2023-07-14T19:18:30.435355+0000) 2023-07-14T19:18:52.972 INFO:journalctl@ceph.osd.9.smithi040.stdout:Jul 14 19:18:52 smithi040 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-9[125889]: 2023-07-14T19:18:52.672+0000 7fa29d625700 -1 osd.9 86 heartbeat_check: no reply from 172.21.15.130:6812 osd.4 since back 2023-07-14T19:18:10.435794+0000 front 2023-07-14T19:18:10.435752+0000 (oldest deadline 2023-07-14T19:18:39.236194+0000) 2023-07-14T19:18:52.972 INFO:journalctl@ceph.osd.9.smithi040.stdout:Jul 14 19:18:52 smithi040 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-9[125889]: 2023-07-14T19:18:52.672+0000 7fa29d625700 -1 osd.9 86 heartbeat_check: no reply from 172.21.15.191:6812 osd.5 since back 2023-07-14T19:18:21.437664+0000 front 2023-07-14T19:18:21.437586+0000 (oldest deadline 2023-07-14T19:18:47.837788+0000) 2023-07-14T19:18:52.992 INFO:journalctl@ceph.osd.10.smithi130.stdout:Jul 14 19:18:52 smithi130 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-10[119160]: 2023-07-14T19:18:52.502+0000 7fa1fdfb9700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2023-07-14T19:17:41.727090+0000 front 2023-07-14T19:17:41.727036+0000 (oldest deadline 2023-07-14T19:18:06.427174+0000) 2023-07-14T19:18:52.992 INFO:journalctl@ceph.osd.10.smithi130.stdout:Jul 14 19:18:52 smithi130 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-10[119160]: 2023-07-14T19:18:52.502+0000 7fa1fdfb9700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-07-14T19:17:52.828799+0000 front 2023-07-14T19:17:52.828840+0000 (oldest deadline 2023-07-14T19:18:14.528908+0000) 2023-07-14T19:18:52.992 INFO:journalctl@ceph.osd.10.smithi130.stdout:Jul 14 19:18:52 smithi130 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-10[119160]: 2023-07-14T19:18:52.502+0000 7fa1fdfb9700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2023-07-14T19:17:56.730545+0000 front 2023-07-14T19:17:56.730590+0000 (oldest deadline 2023-07-14T19:18:22.630599+0000) 2023-07-14T19:18:52.993 INFO:journalctl@ceph.osd.10.smithi130.stdout:Jul 14 19:18:52 smithi130 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-10[119160]: 2023-07-14T19:18:52.502+0000 7fa1fdfb9700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.40:6814 osd.3 since back 2023-07-14T19:18:05.432244+0000 front 2023-07-14T19:18:05.432347+0000 (oldest deadline 2023-07-14T19:18:31.331990+0000) 2023-07-14T19:18:52.993 INFO:journalctl@ceph.osd.10.smithi130.stdout:Jul 14 19:18:52 smithi130 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-10[119160]: 2023-07-14T19:18:52.502+0000 7fa1fdfb9700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.130:6812 osd.4 since back 2023-07-14T19:18:14.232731+0000 front 2023-07-14T19:18:14.232687+0000 (oldest deadline 2023-07-14T19:18:37.132833+0000) 2023-07-14T19:18:52.993 INFO:journalctl@ceph.osd.10.smithi130.stdout:Jul 14 19:18:52 smithi130 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-10[119160]: 2023-07-14T19:18:52.502+0000 7fa1fdfb9700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.191:6812 osd.5 since back 2023-07-14T19:18:22.635891+0000 front 2023-07-14T19:18:22.636012+0000 (oldest deadline 2023-07-14T19:18:46.636128+0000) 2023-07-14T19:18:52.993 INFO:journalctl@ceph.osd.10.smithi130.stdout:Jul 14 19:18:52 smithi130 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-10[119160]: 2023-07-14T19:18:52.502+0000 7fa1fdfb9700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.40:6822 osd.6 since back 2023-07-14T19:18:28.937071+0000 front 2023-07-14T19:18:28.937134+0000 (oldest deadline 2023-07-14T19:18:52.436878+0000) 2023-07-14T19:18:53.472 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jul 14 19:18:52 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-11[120628]: 2023-07-14T19:18:52.977+0000 7f5ef7fea700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2023-07-14T19:17:44.480647+0000 front 2023-07-14T19:17:44.480662+0000 (oldest deadline 2023-07-14T19:18:06.780727+0000) 2023-07-14T19:18:53.472 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jul 14 19:18:52 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-11[120628]: 2023-07-14T19:18:52.977+0000 7f5ef7fea700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-07-14T19:17:50.782602+0000 front 2023-07-14T19:17:50.782583+0000 (oldest deadline 2023-07-14T19:18:14.282678+0000) 2023-07-14T19:18:53.472 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jul 14 19:18:52 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-11[120628]: 2023-07-14T19:18:52.977+0000 7f5ef7fea700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2023-07-14T19:17:58.983820+0000 front 2023-07-14T19:17:58.983785+0000 (oldest deadline 2023-07-14T19:18:25.984643+0000) 2023-07-14T19:18:53.473 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jul 14 19:18:52 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-11[120628]: 2023-07-14T19:18:52.977+0000 7f5ef7fea700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.40:6814 osd.3 since back 2023-07-14T19:18:05.985177+0000 front 2023-07-14T19:18:05.985194+0000 (oldest deadline 2023-07-14T19:18:30.685202+0000) 2023-07-14T19:18:53.473 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jul 14 19:18:52 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-11[120628]: 2023-07-14T19:18:52.977+0000 7f5ef7fea700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.130:6812 osd.4 since back 2023-07-14T19:18:14.186314+0000 front 2023-07-14T19:18:14.186278+0000 (oldest deadline 2023-07-14T19:18:38.886251+0000) 2023-07-14T19:18:53.473 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jul 14 19:18:52 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-11[120628]: 2023-07-14T19:18:52.977+0000 7f5ef7fea700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.191:6812 osd.5 since back 2023-07-14T19:18:21.787193+0000 front 2023-07-14T19:18:21.787218+0000 (oldest deadline 2023-07-14T19:18:43.487275+0000) 2023-07-14T19:18:53.720 INFO:journalctl@ceph.osd.9.smithi040.stdout:Jul 14 19:18:53 smithi040 bash[147005]: ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-9 2023-07-14T19:18:53.742 INFO:journalctl@ceph.osd.10.smithi130.stdout:Jul 14 19:18:53 smithi130 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-10[119160]: 2023-07-14T19:18:53.466+0000 7fa1fdfb9700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2023-07-14T19:17:41.727090+0000 front 2023-07-14T19:17:41.727036+0000 (oldest deadline 2023-07-14T19:18:06.427174+0000) 2023-07-14T19:18:53.742 INFO:journalctl@ceph.osd.10.smithi130.stdout:Jul 14 19:18:53 smithi130 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-10[119160]: 2023-07-14T19:18:53.466+0000 7fa1fdfb9700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-07-14T19:17:52.828799+0000 front 2023-07-14T19:17:52.828840+0000 (oldest deadline 2023-07-14T19:18:14.528908+0000) 2023-07-14T19:18:53.742 INFO:journalctl@ceph.osd.10.smithi130.stdout:Jul 14 19:18:53 smithi130 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-10[119160]: 2023-07-14T19:18:53.466+0000 7fa1fdfb9700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2023-07-14T19:17:56.730545+0000 front 2023-07-14T19:17:56.730590+0000 (oldest deadline 2023-07-14T19:18:22.630599+0000) 2023-07-14T19:18:53.743 INFO:journalctl@ceph.osd.10.smithi130.stdout:Jul 14 19:18:53 smithi130 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-10[119160]: 2023-07-14T19:18:53.466+0000 7fa1fdfb9700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.40:6814 osd.3 since back 2023-07-14T19:18:05.432244+0000 front 2023-07-14T19:18:05.432347+0000 (oldest deadline 2023-07-14T19:18:31.331990+0000) 2023-07-14T19:18:53.743 INFO:journalctl@ceph.osd.10.smithi130.stdout:Jul 14 19:18:53 smithi130 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-10[119160]: 2023-07-14T19:18:53.466+0000 7fa1fdfb9700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.130:6812 osd.4 since back 2023-07-14T19:18:14.232731+0000 front 2023-07-14T19:18:14.232687+0000 (oldest deadline 2023-07-14T19:18:37.132833+0000) 2023-07-14T19:18:53.743 INFO:journalctl@ceph.osd.10.smithi130.stdout:Jul 14 19:18:53 smithi130 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-10[119160]: 2023-07-14T19:18:53.466+0000 7fa1fdfb9700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.191:6812 osd.5 since back 2023-07-14T19:18:22.635891+0000 front 2023-07-14T19:18:22.636012+0000 (oldest deadline 2023-07-14T19:18:46.636128+0000) 2023-07-14T19:18:53.744 INFO:journalctl@ceph.osd.10.smithi130.stdout:Jul 14 19:18:53 smithi130 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-10[119160]: 2023-07-14T19:18:53.466+0000 7fa1fdfb9700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.40:6822 osd.6 since back 2023-07-14T19:18:28.937071+0000 front 2023-07-14T19:18:28.937134+0000 (oldest deadline 2023-07-14T19:18:52.436878+0000) 2023-07-14T19:18:54.472 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jul 14 19:18:54 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-11[120628]: 2023-07-14T19:18:54.010+0000 7f5ef7fea700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2023-07-14T19:17:44.480647+0000 front 2023-07-14T19:17:44.480662+0000 (oldest deadline 2023-07-14T19:18:06.780727+0000) 2023-07-14T19:18:54.472 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jul 14 19:18:54 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-11[120628]: 2023-07-14T19:18:54.010+0000 7f5ef7fea700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-07-14T19:17:50.782602+0000 front 2023-07-14T19:17:50.782583+0000 (oldest deadline 2023-07-14T19:18:14.282678+0000) 2023-07-14T19:18:54.473 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jul 14 19:18:54 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-11[120628]: 2023-07-14T19:18:54.010+0000 7f5ef7fea700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2023-07-14T19:17:58.983820+0000 front 2023-07-14T19:17:58.983785+0000 (oldest deadline 2023-07-14T19:18:25.984643+0000) 2023-07-14T19:18:54.473 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jul 14 19:18:54 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-11[120628]: 2023-07-14T19:18:54.010+0000 7f5ef7fea700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.40:6814 osd.3 since back 2023-07-14T19:18:05.985177+0000 front 2023-07-14T19:18:05.985194+0000 (oldest deadline 2023-07-14T19:18:30.685202+0000) 2023-07-14T19:18:54.473 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jul 14 19:18:54 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-11[120628]: 2023-07-14T19:18:54.010+0000 7f5ef7fea700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.130:6812 osd.4 since back 2023-07-14T19:18:14.186314+0000 front 2023-07-14T19:18:14.186278+0000 (oldest deadline 2023-07-14T19:18:38.886251+0000) 2023-07-14T19:18:54.473 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jul 14 19:18:54 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-11[120628]: 2023-07-14T19:18:54.010+0000 7f5ef7fea700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.191:6812 osd.5 since back 2023-07-14T19:18:21.787193+0000 front 2023-07-14T19:18:21.787218+0000 (oldest deadline 2023-07-14T19:18:43.487275+0000) 2023-07-14T19:18:54.742 INFO:journalctl@ceph.osd.10.smithi130.stdout:Jul 14 19:18:54 smithi130 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-10[119160]: 2023-07-14T19:18:54.459+0000 7fa1fdfb9700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2023-07-14T19:17:41.727090+0000 front 2023-07-14T19:17:41.727036+0000 (oldest deadline 2023-07-14T19:18:06.427174+0000) 2023-07-14T19:18:54.742 INFO:journalctl@ceph.osd.10.smithi130.stdout:Jul 14 19:18:54 smithi130 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-10[119160]: 2023-07-14T19:18:54.459+0000 7fa1fdfb9700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-07-14T19:17:52.828799+0000 front 2023-07-14T19:17:52.828840+0000 (oldest deadline 2023-07-14T19:18:14.528908+0000) 2023-07-14T19:18:54.743 INFO:journalctl@ceph.osd.10.smithi130.stdout:Jul 14 19:18:54 smithi130 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-10[119160]: 2023-07-14T19:18:54.459+0000 7fa1fdfb9700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2023-07-14T19:17:56.730545+0000 front 2023-07-14T19:17:56.730590+0000 (oldest deadline 2023-07-14T19:18:22.630599+0000) 2023-07-14T19:18:54.743 INFO:journalctl@ceph.osd.10.smithi130.stdout:Jul 14 19:18:54 smithi130 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-10[119160]: 2023-07-14T19:18:54.459+0000 7fa1fdfb9700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.40:6814 osd.3 since back 2023-07-14T19:18:05.432244+0000 front 2023-07-14T19:18:05.432347+0000 (oldest deadline 2023-07-14T19:18:31.331990+0000) 2023-07-14T19:18:54.744 INFO:journalctl@ceph.osd.10.smithi130.stdout:Jul 14 19:18:54 smithi130 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-10[119160]: 2023-07-14T19:18:54.459+0000 7fa1fdfb9700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.130:6812 osd.4 since back 2023-07-14T19:18:14.232731+0000 front 2023-07-14T19:18:14.232687+0000 (oldest deadline 2023-07-14T19:18:37.132833+0000) 2023-07-14T19:18:54.744 INFO:journalctl@ceph.osd.10.smithi130.stdout:Jul 14 19:18:54 smithi130 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-10[119160]: 2023-07-14T19:18:54.459+0000 7fa1fdfb9700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.191:6812 osd.5 since back 2023-07-14T19:18:22.635891+0000 front 2023-07-14T19:18:22.636012+0000 (oldest deadline 2023-07-14T19:18:46.636128+0000) 2023-07-14T19:18:54.745 INFO:journalctl@ceph.osd.10.smithi130.stdout:Jul 14 19:18:54 smithi130 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-10[119160]: 2023-07-14T19:18:54.459+0000 7fa1fdfb9700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.40:6822 osd.6 since back 2023-07-14T19:18:28.937071+0000 front 2023-07-14T19:18:28.937134+0000 (oldest deadline 2023-07-14T19:18:52.436878+0000) 2023-07-14T19:18:54.995 DEBUG:teuthology.orchestra.run.smithi040:> sudo pkill -f 'journalctl -f -n 0 -u ceph-adc52f04-2279-11ee-9b33-001a4aab830c@osd.9.service' 2023-07-14T19:18:55.036 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-14T19:18:55.037 INFO:tasks.cephadm.osd.9:Stopped osd.9 2023-07-14T19:18:55.038 INFO:tasks.cephadm.osd.10:Stopping osd.10... 2023-07-14T19:18:55.038 DEBUG:teuthology.orchestra.run.smithi130:> sudo systemctl stop ceph-adc52f04-2279-11ee-9b33-001a4aab830c@osd.10 2023-07-14T19:18:55.415 INFO:journalctl@ceph.osd.10.smithi130.stdout:Jul 14 19:18:55 smithi130 systemd[1]: Stopping Ceph osd.10 for adc52f04-2279-11ee-9b33-001a4aab830c... 2023-07-14T19:18:55.472 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jul 14 19:18:54 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-11[120628]: 2023-07-14T19:18:54.998+0000 7f5ef7fea700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2023-07-14T19:17:44.480647+0000 front 2023-07-14T19:17:44.480662+0000 (oldest deadline 2023-07-14T19:18:06.780727+0000) 2023-07-14T19:18:55.472 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jul 14 19:18:54 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-11[120628]: 2023-07-14T19:18:54.998+0000 7f5ef7fea700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-07-14T19:17:50.782602+0000 front 2023-07-14T19:17:50.782583+0000 (oldest deadline 2023-07-14T19:18:14.282678+0000) 2023-07-14T19:18:55.473 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jul 14 19:18:54 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-11[120628]: 2023-07-14T19:18:54.998+0000 7f5ef7fea700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2023-07-14T19:17:58.983820+0000 front 2023-07-14T19:17:58.983785+0000 (oldest deadline 2023-07-14T19:18:25.984643+0000) 2023-07-14T19:18:55.473 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jul 14 19:18:54 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-11[120628]: 2023-07-14T19:18:54.998+0000 7f5ef7fea700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.40:6814 osd.3 since back 2023-07-14T19:18:05.985177+0000 front 2023-07-14T19:18:05.985194+0000 (oldest deadline 2023-07-14T19:18:30.685202+0000) 2023-07-14T19:18:55.473 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jul 14 19:18:54 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-11[120628]: 2023-07-14T19:18:54.998+0000 7f5ef7fea700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.130:6812 osd.4 since back 2023-07-14T19:18:14.186314+0000 front 2023-07-14T19:18:14.186278+0000 (oldest deadline 2023-07-14T19:18:38.886251+0000) 2023-07-14T19:18:55.474 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jul 14 19:18:54 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-11[120628]: 2023-07-14T19:18:54.998+0000 7f5ef7fea700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.191:6812 osd.5 since back 2023-07-14T19:18:21.787193+0000 front 2023-07-14T19:18:21.787218+0000 (oldest deadline 2023-07-14T19:18:43.487275+0000) 2023-07-14T19:18:55.742 INFO:journalctl@ceph.osd.10.smithi130.stdout:Jul 14 19:18:55 smithi130 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-10[119160]: 2023-07-14T19:18:55.414+0000 7fa2041c7700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-osd -n osd.10 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-07-14T19:18:55.743 INFO:journalctl@ceph.osd.10.smithi130.stdout:Jul 14 19:18:55 smithi130 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-10[119160]: 2023-07-14T19:18:55.414+0000 7fa2041c7700 -1 osd.10 86 *** Got signal Terminated *** 2023-07-14T19:18:55.743 INFO:journalctl@ceph.osd.10.smithi130.stdout:Jul 14 19:18:55 smithi130 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-10[119160]: 2023-07-14T19:18:55.414+0000 7fa2041c7700 -1 osd.10 86 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-07-14T19:18:55.743 INFO:journalctl@ceph.osd.10.smithi130.stdout:Jul 14 19:18:55 smithi130 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-10[119160]: 2023-07-14T19:18:55.497+0000 7fa1fdfb9700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2023-07-14T19:17:41.727090+0000 front 2023-07-14T19:17:41.727036+0000 (oldest deadline 2023-07-14T19:18:06.427174+0000) 2023-07-14T19:18:55.744 INFO:journalctl@ceph.osd.10.smithi130.stdout:Jul 14 19:18:55 smithi130 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-10[119160]: 2023-07-14T19:18:55.497+0000 7fa1fdfb9700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-07-14T19:17:52.828799+0000 front 2023-07-14T19:17:52.828840+0000 (oldest deadline 2023-07-14T19:18:14.528908+0000) 2023-07-14T19:18:55.744 INFO:journalctl@ceph.osd.10.smithi130.stdout:Jul 14 19:18:55 smithi130 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-10[119160]: 2023-07-14T19:18:55.497+0000 7fa1fdfb9700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2023-07-14T19:17:56.730545+0000 front 2023-07-14T19:17:56.730590+0000 (oldest deadline 2023-07-14T19:18:22.630599+0000) 2023-07-14T19:18:55.744 INFO:journalctl@ceph.osd.10.smithi130.stdout:Jul 14 19:18:55 smithi130 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-10[119160]: 2023-07-14T19:18:55.497+0000 7fa1fdfb9700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.40:6814 osd.3 since back 2023-07-14T19:18:05.432244+0000 front 2023-07-14T19:18:05.432347+0000 (oldest deadline 2023-07-14T19:18:31.331990+0000) 2023-07-14T19:18:55.745 INFO:journalctl@ceph.osd.10.smithi130.stdout:Jul 14 19:18:55 smithi130 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-10[119160]: 2023-07-14T19:18:55.497+0000 7fa1fdfb9700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.130:6812 osd.4 since back 2023-07-14T19:18:14.232731+0000 front 2023-07-14T19:18:14.232687+0000 (oldest deadline 2023-07-14T19:18:37.132833+0000) 2023-07-14T19:18:55.745 INFO:journalctl@ceph.osd.10.smithi130.stdout:Jul 14 19:18:55 smithi130 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-10[119160]: 2023-07-14T19:18:55.497+0000 7fa1fdfb9700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.191:6812 osd.5 since back 2023-07-14T19:18:22.635891+0000 front 2023-07-14T19:18:22.636012+0000 (oldest deadline 2023-07-14T19:18:46.636128+0000) 2023-07-14T19:18:55.746 INFO:journalctl@ceph.osd.10.smithi130.stdout:Jul 14 19:18:55 smithi130 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-10[119160]: 2023-07-14T19:18:55.497+0000 7fa1fdfb9700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.40:6822 osd.6 since back 2023-07-14T19:18:28.937071+0000 front 2023-07-14T19:18:28.937134+0000 (oldest deadline 2023-07-14T19:18:52.436878+0000) 2023-07-14T19:18:56.471 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jul 14 19:18:55 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-11[120628]: 2023-07-14T19:18:55.983+0000 7f5ef7fea700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2023-07-14T19:17:44.480647+0000 front 2023-07-14T19:17:44.480662+0000 (oldest deadline 2023-07-14T19:18:06.780727+0000) 2023-07-14T19:18:56.472 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jul 14 19:18:55 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-11[120628]: 2023-07-14T19:18:55.983+0000 7f5ef7fea700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-07-14T19:17:50.782602+0000 front 2023-07-14T19:17:50.782583+0000 (oldest deadline 2023-07-14T19:18:14.282678+0000) 2023-07-14T19:18:56.472 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jul 14 19:18:55 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-11[120628]: 2023-07-14T19:18:55.983+0000 7f5ef7fea700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2023-07-14T19:17:58.983820+0000 front 2023-07-14T19:17:58.983785+0000 (oldest deadline 2023-07-14T19:18:25.984643+0000) 2023-07-14T19:18:56.472 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jul 14 19:18:55 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-11[120628]: 2023-07-14T19:18:55.983+0000 7f5ef7fea700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.40:6814 osd.3 since back 2023-07-14T19:18:05.985177+0000 front 2023-07-14T19:18:05.985194+0000 (oldest deadline 2023-07-14T19:18:30.685202+0000) 2023-07-14T19:18:56.473 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jul 14 19:18:55 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-11[120628]: 2023-07-14T19:18:55.983+0000 7f5ef7fea700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.130:6812 osd.4 since back 2023-07-14T19:18:14.186314+0000 front 2023-07-14T19:18:14.186278+0000 (oldest deadline 2023-07-14T19:18:38.886251+0000) 2023-07-14T19:18:56.473 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jul 14 19:18:55 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-11[120628]: 2023-07-14T19:18:55.983+0000 7f5ef7fea700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.191:6812 osd.5 since back 2023-07-14T19:18:21.787193+0000 front 2023-07-14T19:18:21.787218+0000 (oldest deadline 2023-07-14T19:18:43.487275+0000) 2023-07-14T19:18:56.742 INFO:journalctl@ceph.osd.10.smithi130.stdout:Jul 14 19:18:56 smithi130 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-10[119160]: 2023-07-14T19:18:56.467+0000 7fa1fdfb9700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2023-07-14T19:17:41.727090+0000 front 2023-07-14T19:17:41.727036+0000 (oldest deadline 2023-07-14T19:18:06.427174+0000) 2023-07-14T19:18:56.743 INFO:journalctl@ceph.osd.10.smithi130.stdout:Jul 14 19:18:56 smithi130 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-10[119160]: 2023-07-14T19:18:56.467+0000 7fa1fdfb9700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-07-14T19:17:52.828799+0000 front 2023-07-14T19:17:52.828840+0000 (oldest deadline 2023-07-14T19:18:14.528908+0000) 2023-07-14T19:18:56.743 INFO:journalctl@ceph.osd.10.smithi130.stdout:Jul 14 19:18:56 smithi130 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-10[119160]: 2023-07-14T19:18:56.467+0000 7fa1fdfb9700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2023-07-14T19:17:56.730545+0000 front 2023-07-14T19:17:56.730590+0000 (oldest deadline 2023-07-14T19:18:22.630599+0000) 2023-07-14T19:18:56.743 INFO:journalctl@ceph.osd.10.smithi130.stdout:Jul 14 19:18:56 smithi130 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-10[119160]: 2023-07-14T19:18:56.467+0000 7fa1fdfb9700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.40:6814 osd.3 since back 2023-07-14T19:18:05.432244+0000 front 2023-07-14T19:18:05.432347+0000 (oldest deadline 2023-07-14T19:18:31.331990+0000) 2023-07-14T19:18:56.743 INFO:journalctl@ceph.osd.10.smithi130.stdout:Jul 14 19:18:56 smithi130 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-10[119160]: 2023-07-14T19:18:56.467+0000 7fa1fdfb9700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.130:6812 osd.4 since back 2023-07-14T19:18:14.232731+0000 front 2023-07-14T19:18:14.232687+0000 (oldest deadline 2023-07-14T19:18:37.132833+0000) 2023-07-14T19:18:56.743 INFO:journalctl@ceph.osd.10.smithi130.stdout:Jul 14 19:18:56 smithi130 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-10[119160]: 2023-07-14T19:18:56.467+0000 7fa1fdfb9700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.191:6812 osd.5 since back 2023-07-14T19:18:22.635891+0000 front 2023-07-14T19:18:22.636012+0000 (oldest deadline 2023-07-14T19:18:46.636128+0000) 2023-07-14T19:18:56.744 INFO:journalctl@ceph.osd.10.smithi130.stdout:Jul 14 19:18:56 smithi130 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-10[119160]: 2023-07-14T19:18:56.467+0000 7fa1fdfb9700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.40:6822 osd.6 since back 2023-07-14T19:18:28.937071+0000 front 2023-07-14T19:18:28.937134+0000 (oldest deadline 2023-07-14T19:18:52.436878+0000) 2023-07-14T19:18:57.472 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jul 14 19:18:56 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-11[120628]: 2023-07-14T19:18:56.986+0000 7f5ef7fea700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2023-07-14T19:17:44.480647+0000 front 2023-07-14T19:17:44.480662+0000 (oldest deadline 2023-07-14T19:18:06.780727+0000) 2023-07-14T19:18:57.472 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jul 14 19:18:56 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-11[120628]: 2023-07-14T19:18:56.986+0000 7f5ef7fea700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-07-14T19:17:50.782602+0000 front 2023-07-14T19:17:50.782583+0000 (oldest deadline 2023-07-14T19:18:14.282678+0000) 2023-07-14T19:18:57.472 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jul 14 19:18:56 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-11[120628]: 2023-07-14T19:18:56.986+0000 7f5ef7fea700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2023-07-14T19:17:58.983820+0000 front 2023-07-14T19:17:58.983785+0000 (oldest deadline 2023-07-14T19:18:25.984643+0000) 2023-07-14T19:18:57.473 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jul 14 19:18:56 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-11[120628]: 2023-07-14T19:18:56.986+0000 7f5ef7fea700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.40:6814 osd.3 since back 2023-07-14T19:18:05.985177+0000 front 2023-07-14T19:18:05.985194+0000 (oldest deadline 2023-07-14T19:18:30.685202+0000) 2023-07-14T19:18:57.473 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jul 14 19:18:56 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-11[120628]: 2023-07-14T19:18:56.986+0000 7f5ef7fea700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.130:6812 osd.4 since back 2023-07-14T19:18:14.186314+0000 front 2023-07-14T19:18:14.186278+0000 (oldest deadline 2023-07-14T19:18:38.886251+0000) 2023-07-14T19:18:57.473 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jul 14 19:18:56 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-11[120628]: 2023-07-14T19:18:56.986+0000 7f5ef7fea700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.191:6812 osd.5 since back 2023-07-14T19:18:21.787193+0000 front 2023-07-14T19:18:21.787218+0000 (oldest deadline 2023-07-14T19:18:43.487275+0000) 2023-07-14T19:18:57.742 INFO:journalctl@ceph.osd.10.smithi130.stdout:Jul 14 19:18:57 smithi130 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-10[119160]: 2023-07-14T19:18:57.476+0000 7fa1fdfb9700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2023-07-14T19:17:41.727090+0000 front 2023-07-14T19:17:41.727036+0000 (oldest deadline 2023-07-14T19:18:06.427174+0000) 2023-07-14T19:18:57.742 INFO:journalctl@ceph.osd.10.smithi130.stdout:Jul 14 19:18:57 smithi130 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-10[119160]: 2023-07-14T19:18:57.476+0000 7fa1fdfb9700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-07-14T19:17:52.828799+0000 front 2023-07-14T19:17:52.828840+0000 (oldest deadline 2023-07-14T19:18:14.528908+0000) 2023-07-14T19:18:57.742 INFO:journalctl@ceph.osd.10.smithi130.stdout:Jul 14 19:18:57 smithi130 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-10[119160]: 2023-07-14T19:18:57.476+0000 7fa1fdfb9700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2023-07-14T19:17:56.730545+0000 front 2023-07-14T19:17:56.730590+0000 (oldest deadline 2023-07-14T19:18:22.630599+0000) 2023-07-14T19:18:57.743 INFO:journalctl@ceph.osd.10.smithi130.stdout:Jul 14 19:18:57 smithi130 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-10[119160]: 2023-07-14T19:18:57.476+0000 7fa1fdfb9700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.40:6814 osd.3 since back 2023-07-14T19:18:05.432244+0000 front 2023-07-14T19:18:05.432347+0000 (oldest deadline 2023-07-14T19:18:31.331990+0000) 2023-07-14T19:18:57.743 INFO:journalctl@ceph.osd.10.smithi130.stdout:Jul 14 19:18:57 smithi130 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-10[119160]: 2023-07-14T19:18:57.476+0000 7fa1fdfb9700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.130:6812 osd.4 since back 2023-07-14T19:18:14.232731+0000 front 2023-07-14T19:18:14.232687+0000 (oldest deadline 2023-07-14T19:18:37.132833+0000) 2023-07-14T19:18:57.743 INFO:journalctl@ceph.osd.10.smithi130.stdout:Jul 14 19:18:57 smithi130 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-10[119160]: 2023-07-14T19:18:57.476+0000 7fa1fdfb9700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.191:6812 osd.5 since back 2023-07-14T19:18:22.635891+0000 front 2023-07-14T19:18:22.636012+0000 (oldest deadline 2023-07-14T19:18:46.636128+0000) 2023-07-14T19:18:57.743 INFO:journalctl@ceph.osd.10.smithi130.stdout:Jul 14 19:18:57 smithi130 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-10[119160]: 2023-07-14T19:18:57.476+0000 7fa1fdfb9700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.40:6822 osd.6 since back 2023-07-14T19:18:28.937071+0000 front 2023-07-14T19:18:28.937134+0000 (oldest deadline 2023-07-14T19:18:52.436878+0000) 2023-07-14T19:18:58.222 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jul 14 19:18:57 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-11[120628]: 2023-07-14T19:18:57.967+0000 7f5ef7fea700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2023-07-14T19:17:44.480647+0000 front 2023-07-14T19:17:44.480662+0000 (oldest deadline 2023-07-14T19:18:06.780727+0000) 2023-07-14T19:18:58.222 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jul 14 19:18:57 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-11[120628]: 2023-07-14T19:18:57.967+0000 7f5ef7fea700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-07-14T19:17:50.782602+0000 front 2023-07-14T19:17:50.782583+0000 (oldest deadline 2023-07-14T19:18:14.282678+0000) 2023-07-14T19:18:58.223 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jul 14 19:18:57 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-11[120628]: 2023-07-14T19:18:57.967+0000 7f5ef7fea700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2023-07-14T19:17:58.983820+0000 front 2023-07-14T19:17:58.983785+0000 (oldest deadline 2023-07-14T19:18:25.984643+0000) 2023-07-14T19:18:58.223 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jul 14 19:18:57 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-11[120628]: 2023-07-14T19:18:57.967+0000 7f5ef7fea700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.40:6814 osd.3 since back 2023-07-14T19:18:05.985177+0000 front 2023-07-14T19:18:05.985194+0000 (oldest deadline 2023-07-14T19:18:30.685202+0000) 2023-07-14T19:18:58.224 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jul 14 19:18:57 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-11[120628]: 2023-07-14T19:18:57.967+0000 7f5ef7fea700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.130:6812 osd.4 since back 2023-07-14T19:18:14.186314+0000 front 2023-07-14T19:18:14.186278+0000 (oldest deadline 2023-07-14T19:18:38.886251+0000) 2023-07-14T19:18:58.224 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jul 14 19:18:57 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-11[120628]: 2023-07-14T19:18:57.967+0000 7f5ef7fea700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.191:6812 osd.5 since back 2023-07-14T19:18:21.787193+0000 front 2023-07-14T19:18:21.787218+0000 (oldest deadline 2023-07-14T19:18:43.487275+0000) 2023-07-14T19:18:58.742 INFO:journalctl@ceph.osd.10.smithi130.stdout:Jul 14 19:18:58 smithi130 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-10[119160]: 2023-07-14T19:18:58.465+0000 7fa1fdfb9700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2023-07-14T19:17:41.727090+0000 front 2023-07-14T19:17:41.727036+0000 (oldest deadline 2023-07-14T19:18:06.427174+0000) 2023-07-14T19:18:58.742 INFO:journalctl@ceph.osd.10.smithi130.stdout:Jul 14 19:18:58 smithi130 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-10[119160]: 2023-07-14T19:18:58.465+0000 7fa1fdfb9700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-07-14T19:17:52.828799+0000 front 2023-07-14T19:17:52.828840+0000 (oldest deadline 2023-07-14T19:18:14.528908+0000) 2023-07-14T19:18:58.742 INFO:journalctl@ceph.osd.10.smithi130.stdout:Jul 14 19:18:58 smithi130 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-10[119160]: 2023-07-14T19:18:58.465+0000 7fa1fdfb9700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2023-07-14T19:17:56.730545+0000 front 2023-07-14T19:17:56.730590+0000 (oldest deadline 2023-07-14T19:18:22.630599+0000) 2023-07-14T19:18:58.743 INFO:journalctl@ceph.osd.10.smithi130.stdout:Jul 14 19:18:58 smithi130 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-10[119160]: 2023-07-14T19:18:58.465+0000 7fa1fdfb9700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.40:6814 osd.3 since back 2023-07-14T19:18:05.432244+0000 front 2023-07-14T19:18:05.432347+0000 (oldest deadline 2023-07-14T19:18:31.331990+0000) 2023-07-14T19:18:58.743 INFO:journalctl@ceph.osd.10.smithi130.stdout:Jul 14 19:18:58 smithi130 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-10[119160]: 2023-07-14T19:18:58.465+0000 7fa1fdfb9700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.130:6812 osd.4 since back 2023-07-14T19:18:14.232731+0000 front 2023-07-14T19:18:14.232687+0000 (oldest deadline 2023-07-14T19:18:37.132833+0000) 2023-07-14T19:18:58.743 INFO:journalctl@ceph.osd.10.smithi130.stdout:Jul 14 19:18:58 smithi130 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-10[119160]: 2023-07-14T19:18:58.465+0000 7fa1fdfb9700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.191:6812 osd.5 since back 2023-07-14T19:18:22.635891+0000 front 2023-07-14T19:18:22.636012+0000 (oldest deadline 2023-07-14T19:18:46.636128+0000) 2023-07-14T19:18:58.743 INFO:journalctl@ceph.osd.10.smithi130.stdout:Jul 14 19:18:58 smithi130 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-10[119160]: 2023-07-14T19:18:58.465+0000 7fa1fdfb9700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.40:6822 osd.6 since back 2023-07-14T19:18:28.937071+0000 front 2023-07-14T19:18:28.937134+0000 (oldest deadline 2023-07-14T19:18:52.436878+0000) 2023-07-14T19:18:59.221 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jul 14 19:18:58 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-11[120628]: 2023-07-14T19:18:58.963+0000 7f5ef7fea700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2023-07-14T19:17:44.480647+0000 front 2023-07-14T19:17:44.480662+0000 (oldest deadline 2023-07-14T19:18:06.780727+0000) 2023-07-14T19:18:59.222 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jul 14 19:18:58 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-11[120628]: 2023-07-14T19:18:58.963+0000 7f5ef7fea700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-07-14T19:17:50.782602+0000 front 2023-07-14T19:17:50.782583+0000 (oldest deadline 2023-07-14T19:18:14.282678+0000) 2023-07-14T19:18:59.222 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jul 14 19:18:58 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-11[120628]: 2023-07-14T19:18:58.963+0000 7f5ef7fea700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2023-07-14T19:17:58.983820+0000 front 2023-07-14T19:17:58.983785+0000 (oldest deadline 2023-07-14T19:18:25.984643+0000) 2023-07-14T19:18:59.222 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jul 14 19:18:58 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-11[120628]: 2023-07-14T19:18:58.963+0000 7f5ef7fea700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.40:6814 osd.3 since back 2023-07-14T19:18:05.985177+0000 front 2023-07-14T19:18:05.985194+0000 (oldest deadline 2023-07-14T19:18:30.685202+0000) 2023-07-14T19:18:59.223 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jul 14 19:18:58 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-11[120628]: 2023-07-14T19:18:58.963+0000 7f5ef7fea700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.130:6812 osd.4 since back 2023-07-14T19:18:14.186314+0000 front 2023-07-14T19:18:14.186278+0000 (oldest deadline 2023-07-14T19:18:38.886251+0000) 2023-07-14T19:18:59.223 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jul 14 19:18:58 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-11[120628]: 2023-07-14T19:18:58.963+0000 7f5ef7fea700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.191:6812 osd.5 since back 2023-07-14T19:18:21.787193+0000 front 2023-07-14T19:18:21.787218+0000 (oldest deadline 2023-07-14T19:18:43.487275+0000) 2023-07-14T19:18:59.223 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jul 14 19:18:58 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-11[120628]: 2023-07-14T19:18:58.963+0000 7f5ef7fea700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.130:6820 osd.7 since back 2023-07-14T19:18:34.589145+0000 front 2023-07-14T19:18:34.589216+0000 (oldest deadline 2023-07-14T19:18:58.689113+0000) 2023-07-14T19:18:59.992 INFO:journalctl@ceph.osd.10.smithi130.stdout:Jul 14 19:18:59 smithi130 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-10[119160]: 2023-07-14T19:18:59.499+0000 7fa1fdfb9700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2023-07-14T19:17:41.727090+0000 front 2023-07-14T19:17:41.727036+0000 (oldest deadline 2023-07-14T19:18:06.427174+0000) 2023-07-14T19:18:59.992 INFO:journalctl@ceph.osd.10.smithi130.stdout:Jul 14 19:18:59 smithi130 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-10[119160]: 2023-07-14T19:18:59.499+0000 7fa1fdfb9700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-07-14T19:17:52.828799+0000 front 2023-07-14T19:17:52.828840+0000 (oldest deadline 2023-07-14T19:18:14.528908+0000) 2023-07-14T19:18:59.992 INFO:journalctl@ceph.osd.10.smithi130.stdout:Jul 14 19:18:59 smithi130 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-10[119160]: 2023-07-14T19:18:59.499+0000 7fa1fdfb9700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2023-07-14T19:17:56.730545+0000 front 2023-07-14T19:17:56.730590+0000 (oldest deadline 2023-07-14T19:18:22.630599+0000) 2023-07-14T19:18:59.993 INFO:journalctl@ceph.osd.10.smithi130.stdout:Jul 14 19:18:59 smithi130 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-10[119160]: 2023-07-14T19:18:59.499+0000 7fa1fdfb9700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.40:6814 osd.3 since back 2023-07-14T19:18:05.432244+0000 front 2023-07-14T19:18:05.432347+0000 (oldest deadline 2023-07-14T19:18:31.331990+0000) 2023-07-14T19:18:59.993 INFO:journalctl@ceph.osd.10.smithi130.stdout:Jul 14 19:18:59 smithi130 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-10[119160]: 2023-07-14T19:18:59.499+0000 7fa1fdfb9700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.130:6812 osd.4 since back 2023-07-14T19:18:14.232731+0000 front 2023-07-14T19:18:14.232687+0000 (oldest deadline 2023-07-14T19:18:37.132833+0000) 2023-07-14T19:18:59.993 INFO:journalctl@ceph.osd.10.smithi130.stdout:Jul 14 19:18:59 smithi130 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-10[119160]: 2023-07-14T19:18:59.499+0000 7fa1fdfb9700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.191:6812 osd.5 since back 2023-07-14T19:18:22.635891+0000 front 2023-07-14T19:18:22.636012+0000 (oldest deadline 2023-07-14T19:18:46.636128+0000) 2023-07-14T19:18:59.993 INFO:journalctl@ceph.osd.10.smithi130.stdout:Jul 14 19:18:59 smithi130 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-10[119160]: 2023-07-14T19:18:59.499+0000 7fa1fdfb9700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.40:6822 osd.6 since back 2023-07-14T19:18:28.937071+0000 front 2023-07-14T19:18:28.937134+0000 (oldest deadline 2023-07-14T19:18:52.436878+0000) 2023-07-14T19:19:00.222 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jul 14 19:18:59 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-11[120628]: 2023-07-14T19:18:59.917+0000 7f5ef7fea700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2023-07-14T19:17:44.480647+0000 front 2023-07-14T19:17:44.480662+0000 (oldest deadline 2023-07-14T19:18:06.780727+0000) 2023-07-14T19:19:00.222 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jul 14 19:18:59 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-11[120628]: 2023-07-14T19:18:59.917+0000 7f5ef7fea700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-07-14T19:17:50.782602+0000 front 2023-07-14T19:17:50.782583+0000 (oldest deadline 2023-07-14T19:18:14.282678+0000) 2023-07-14T19:19:00.222 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jul 14 19:18:59 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-11[120628]: 2023-07-14T19:18:59.917+0000 7f5ef7fea700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2023-07-14T19:17:58.983820+0000 front 2023-07-14T19:17:58.983785+0000 (oldest deadline 2023-07-14T19:18:25.984643+0000) 2023-07-14T19:19:00.223 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jul 14 19:18:59 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-11[120628]: 2023-07-14T19:18:59.917+0000 7f5ef7fea700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.40:6814 osd.3 since back 2023-07-14T19:18:05.985177+0000 front 2023-07-14T19:18:05.985194+0000 (oldest deadline 2023-07-14T19:18:30.685202+0000) 2023-07-14T19:19:00.223 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jul 14 19:18:59 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-11[120628]: 2023-07-14T19:18:59.917+0000 7f5ef7fea700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.130:6812 osd.4 since back 2023-07-14T19:18:14.186314+0000 front 2023-07-14T19:18:14.186278+0000 (oldest deadline 2023-07-14T19:18:38.886251+0000) 2023-07-14T19:19:00.223 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jul 14 19:18:59 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-11[120628]: 2023-07-14T19:18:59.917+0000 7f5ef7fea700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.191:6812 osd.5 since back 2023-07-14T19:18:21.787193+0000 front 2023-07-14T19:18:21.787218+0000 (oldest deadline 2023-07-14T19:18:43.487275+0000) 2023-07-14T19:19:00.223 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jul 14 19:18:59 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-11[120628]: 2023-07-14T19:18:59.917+0000 7f5ef7fea700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.130:6820 osd.7 since back 2023-07-14T19:18:34.589145+0000 front 2023-07-14T19:18:34.589216+0000 (oldest deadline 2023-07-14T19:18:58.689113+0000) 2023-07-14T19:19:01.222 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jul 14 19:19:00 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-11[120628]: 2023-07-14T19:19:00.876+0000 7f5ef7fea700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2023-07-14T19:17:44.480647+0000 front 2023-07-14T19:17:44.480662+0000 (oldest deadline 2023-07-14T19:18:06.780727+0000) 2023-07-14T19:19:01.222 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jul 14 19:19:00 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-11[120628]: 2023-07-14T19:19:00.876+0000 7f5ef7fea700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-07-14T19:17:50.782602+0000 front 2023-07-14T19:17:50.782583+0000 (oldest deadline 2023-07-14T19:18:14.282678+0000) 2023-07-14T19:19:01.223 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jul 14 19:19:00 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-11[120628]: 2023-07-14T19:19:00.876+0000 7f5ef7fea700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2023-07-14T19:17:58.983820+0000 front 2023-07-14T19:17:58.983785+0000 (oldest deadline 2023-07-14T19:18:25.984643+0000) 2023-07-14T19:19:01.223 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jul 14 19:19:00 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-11[120628]: 2023-07-14T19:19:00.876+0000 7f5ef7fea700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.40:6814 osd.3 since back 2023-07-14T19:18:05.985177+0000 front 2023-07-14T19:18:05.985194+0000 (oldest deadline 2023-07-14T19:18:30.685202+0000) 2023-07-14T19:19:01.223 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jul 14 19:19:00 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-11[120628]: 2023-07-14T19:19:00.876+0000 7f5ef7fea700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.130:6812 osd.4 since back 2023-07-14T19:18:14.186314+0000 front 2023-07-14T19:18:14.186278+0000 (oldest deadline 2023-07-14T19:18:38.886251+0000) 2023-07-14T19:19:01.223 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jul 14 19:19:00 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-11[120628]: 2023-07-14T19:19:00.876+0000 7f5ef7fea700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.191:6812 osd.5 since back 2023-07-14T19:18:21.787193+0000 front 2023-07-14T19:18:21.787218+0000 (oldest deadline 2023-07-14T19:18:43.487275+0000) 2023-07-14T19:19:01.224 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jul 14 19:19:00 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-11[120628]: 2023-07-14T19:19:00.876+0000 7f5ef7fea700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.130:6820 osd.7 since back 2023-07-14T19:18:34.589145+0000 front 2023-07-14T19:18:34.589216+0000 (oldest deadline 2023-07-14T19:18:58.689113+0000) 2023-07-14T19:19:01.492 INFO:journalctl@ceph.osd.10.smithi130.stdout:Jul 14 19:19:01 smithi130 bash[125943]: ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-10 2023-07-14T19:19:02.222 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jul 14 19:19:01 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-11[120628]: 2023-07-14T19:19:01.881+0000 7f5ef7fea700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2023-07-14T19:17:44.480647+0000 front 2023-07-14T19:17:44.480662+0000 (oldest deadline 2023-07-14T19:18:06.780727+0000) 2023-07-14T19:19:02.222 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jul 14 19:19:01 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-11[120628]: 2023-07-14T19:19:01.881+0000 7f5ef7fea700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-07-14T19:17:50.782602+0000 front 2023-07-14T19:17:50.782583+0000 (oldest deadline 2023-07-14T19:18:14.282678+0000) 2023-07-14T19:19:02.222 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jul 14 19:19:01 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-11[120628]: 2023-07-14T19:19:01.881+0000 7f5ef7fea700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2023-07-14T19:17:58.983820+0000 front 2023-07-14T19:17:58.983785+0000 (oldest deadline 2023-07-14T19:18:25.984643+0000) 2023-07-14T19:19:02.223 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jul 14 19:19:01 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-11[120628]: 2023-07-14T19:19:01.881+0000 7f5ef7fea700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.40:6814 osd.3 since back 2023-07-14T19:18:05.985177+0000 front 2023-07-14T19:18:05.985194+0000 (oldest deadline 2023-07-14T19:18:30.685202+0000) 2023-07-14T19:19:02.223 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jul 14 19:19:01 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-11[120628]: 2023-07-14T19:19:01.881+0000 7f5ef7fea700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.130:6812 osd.4 since back 2023-07-14T19:18:14.186314+0000 front 2023-07-14T19:18:14.186278+0000 (oldest deadline 2023-07-14T19:18:38.886251+0000) 2023-07-14T19:19:02.223 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jul 14 19:19:01 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-11[120628]: 2023-07-14T19:19:01.881+0000 7f5ef7fea700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.191:6812 osd.5 since back 2023-07-14T19:18:21.787193+0000 front 2023-07-14T19:18:21.787218+0000 (oldest deadline 2023-07-14T19:18:43.487275+0000) 2023-07-14T19:19:02.223 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jul 14 19:19:01 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-11[120628]: 2023-07-14T19:19:01.881+0000 7f5ef7fea700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.130:6820 osd.7 since back 2023-07-14T19:18:34.589145+0000 front 2023-07-14T19:18:34.589216+0000 (oldest deadline 2023-07-14T19:18:58.689113+0000) 2023-07-14T19:19:02.610 DEBUG:teuthology.orchestra.run.smithi130:> sudo pkill -f 'journalctl -f -n 0 -u ceph-adc52f04-2279-11ee-9b33-001a4aab830c@osd.10.service' 2023-07-14T19:19:02.651 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-14T19:19:02.652 INFO:tasks.cephadm.osd.10:Stopped osd.10 2023-07-14T19:19:02.652 INFO:tasks.cephadm.osd.11:Stopping osd.11... 2023-07-14T19:19:02.653 DEBUG:teuthology.orchestra.run.smithi191:> sudo systemctl stop ceph-adc52f04-2279-11ee-9b33-001a4aab830c@osd.11 2023-07-14T19:19:02.972 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jul 14 19:19:02 smithi191 systemd[1]: Stopping Ceph osd.11 for adc52f04-2279-11ee-9b33-001a4aab830c... 2023-07-14T19:19:02.973 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jul 14 19:19:02 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-11[120628]: 2023-07-14T19:19:02.855+0000 7f5ef7fea700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2023-07-14T19:17:44.480647+0000 front 2023-07-14T19:17:44.480662+0000 (oldest deadline 2023-07-14T19:18:06.780727+0000) 2023-07-14T19:19:02.973 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jul 14 19:19:02 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-11[120628]: 2023-07-14T19:19:02.855+0000 7f5ef7fea700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-07-14T19:17:50.782602+0000 front 2023-07-14T19:17:50.782583+0000 (oldest deadline 2023-07-14T19:18:14.282678+0000) 2023-07-14T19:19:02.974 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jul 14 19:19:02 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-11[120628]: 2023-07-14T19:19:02.855+0000 7f5ef7fea700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2023-07-14T19:17:58.983820+0000 front 2023-07-14T19:17:58.983785+0000 (oldest deadline 2023-07-14T19:18:25.984643+0000) 2023-07-14T19:19:02.974 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jul 14 19:19:02 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-11[120628]: 2023-07-14T19:19:02.855+0000 7f5ef7fea700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.40:6814 osd.3 since back 2023-07-14T19:18:05.985177+0000 front 2023-07-14T19:18:05.985194+0000 (oldest deadline 2023-07-14T19:18:30.685202+0000) 2023-07-14T19:19:02.975 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jul 14 19:19:02 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-11[120628]: 2023-07-14T19:19:02.855+0000 7f5ef7fea700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.130:6812 osd.4 since back 2023-07-14T19:18:14.186314+0000 front 2023-07-14T19:18:14.186278+0000 (oldest deadline 2023-07-14T19:18:38.886251+0000) 2023-07-14T19:19:02.975 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jul 14 19:19:02 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-11[120628]: 2023-07-14T19:19:02.855+0000 7f5ef7fea700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.191:6812 osd.5 since back 2023-07-14T19:18:21.787193+0000 front 2023-07-14T19:18:21.787218+0000 (oldest deadline 2023-07-14T19:18:43.487275+0000) 2023-07-14T19:19:02.975 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jul 14 19:19:02 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-11[120628]: 2023-07-14T19:19:02.855+0000 7f5ef7fea700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.130:6820 osd.7 since back 2023-07-14T19:18:34.589145+0000 front 2023-07-14T19:18:34.589216+0000 (oldest deadline 2023-07-14T19:18:58.689113+0000) 2023-07-14T19:19:03.472 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jul 14 19:19:03 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-11[120628]: 2023-07-14T19:19:03.020+0000 7f5efd9f7700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-osd -n osd.11 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-07-14T19:19:03.472 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jul 14 19:19:03 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-11[120628]: 2023-07-14T19:19:03.020+0000 7f5efd9f7700 -1 osd.11 86 *** Got signal Terminated *** 2023-07-14T19:19:03.472 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jul 14 19:19:03 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-11[120628]: 2023-07-14T19:19:03.020+0000 7f5efd9f7700 -1 osd.11 86 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-07-14T19:19:04.221 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jul 14 19:19:03 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-11[120628]: 2023-07-14T19:19:03.898+0000 7f5ef7fea700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2023-07-14T19:17:44.480647+0000 front 2023-07-14T19:17:44.480662+0000 (oldest deadline 2023-07-14T19:18:06.780727+0000) 2023-07-14T19:19:04.222 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jul 14 19:19:03 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-11[120628]: 2023-07-14T19:19:03.898+0000 7f5ef7fea700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-07-14T19:17:50.782602+0000 front 2023-07-14T19:17:50.782583+0000 (oldest deadline 2023-07-14T19:18:14.282678+0000) 2023-07-14T19:19:04.222 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jul 14 19:19:03 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-11[120628]: 2023-07-14T19:19:03.898+0000 7f5ef7fea700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2023-07-14T19:17:58.983820+0000 front 2023-07-14T19:17:58.983785+0000 (oldest deadline 2023-07-14T19:18:25.984643+0000) 2023-07-14T19:19:04.222 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jul 14 19:19:03 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-11[120628]: 2023-07-14T19:19:03.898+0000 7f5ef7fea700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.40:6814 osd.3 since back 2023-07-14T19:18:05.985177+0000 front 2023-07-14T19:18:05.985194+0000 (oldest deadline 2023-07-14T19:18:30.685202+0000) 2023-07-14T19:19:04.223 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jul 14 19:19:03 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-11[120628]: 2023-07-14T19:19:03.898+0000 7f5ef7fea700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.130:6812 osd.4 since back 2023-07-14T19:18:14.186314+0000 front 2023-07-14T19:18:14.186278+0000 (oldest deadline 2023-07-14T19:18:38.886251+0000) 2023-07-14T19:19:04.223 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jul 14 19:19:03 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-11[120628]: 2023-07-14T19:19:03.898+0000 7f5ef7fea700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.191:6812 osd.5 since back 2023-07-14T19:18:21.787193+0000 front 2023-07-14T19:18:21.787218+0000 (oldest deadline 2023-07-14T19:18:43.487275+0000) 2023-07-14T19:19:04.223 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jul 14 19:19:03 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-11[120628]: 2023-07-14T19:19:03.898+0000 7f5ef7fea700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.130:6820 osd.7 since back 2023-07-14T19:18:34.589145+0000 front 2023-07-14T19:18:34.589216+0000 (oldest deadline 2023-07-14T19:18:58.689113+0000) 2023-07-14T19:19:05.222 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jul 14 19:19:04 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-11[120628]: 2023-07-14T19:19:04.883+0000 7f5ef7fea700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2023-07-14T19:17:44.480647+0000 front 2023-07-14T19:17:44.480662+0000 (oldest deadline 2023-07-14T19:18:06.780727+0000) 2023-07-14T19:19:05.222 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jul 14 19:19:04 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-11[120628]: 2023-07-14T19:19:04.883+0000 7f5ef7fea700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-07-14T19:17:50.782602+0000 front 2023-07-14T19:17:50.782583+0000 (oldest deadline 2023-07-14T19:18:14.282678+0000) 2023-07-14T19:19:05.223 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jul 14 19:19:04 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-11[120628]: 2023-07-14T19:19:04.883+0000 7f5ef7fea700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2023-07-14T19:17:58.983820+0000 front 2023-07-14T19:17:58.983785+0000 (oldest deadline 2023-07-14T19:18:25.984643+0000) 2023-07-14T19:19:05.223 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jul 14 19:19:04 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-11[120628]: 2023-07-14T19:19:04.883+0000 7f5ef7fea700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.40:6814 osd.3 since back 2023-07-14T19:18:05.985177+0000 front 2023-07-14T19:18:05.985194+0000 (oldest deadline 2023-07-14T19:18:30.685202+0000) 2023-07-14T19:19:05.224 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jul 14 19:19:04 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-11[120628]: 2023-07-14T19:19:04.883+0000 7f5ef7fea700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.130:6812 osd.4 since back 2023-07-14T19:18:14.186314+0000 front 2023-07-14T19:18:14.186278+0000 (oldest deadline 2023-07-14T19:18:38.886251+0000) 2023-07-14T19:19:05.224 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jul 14 19:19:04 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-11[120628]: 2023-07-14T19:19:04.883+0000 7f5ef7fea700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.191:6812 osd.5 since back 2023-07-14T19:18:21.787193+0000 front 2023-07-14T19:18:21.787218+0000 (oldest deadline 2023-07-14T19:18:43.487275+0000) 2023-07-14T19:19:05.224 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jul 14 19:19:04 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-11[120628]: 2023-07-14T19:19:04.883+0000 7f5ef7fea700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.130:6820 osd.7 since back 2023-07-14T19:18:34.589145+0000 front 2023-07-14T19:18:34.589216+0000 (oldest deadline 2023-07-14T19:18:58.689113+0000) 2023-07-14T19:19:06.221 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jul 14 19:19:05 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-11[120628]: 2023-07-14T19:19:05.906+0000 7f5ef7fea700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2023-07-14T19:17:44.480647+0000 front 2023-07-14T19:17:44.480662+0000 (oldest deadline 2023-07-14T19:18:06.780727+0000) 2023-07-14T19:19:06.222 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jul 14 19:19:05 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-11[120628]: 2023-07-14T19:19:05.906+0000 7f5ef7fea700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-07-14T19:17:50.782602+0000 front 2023-07-14T19:17:50.782583+0000 (oldest deadline 2023-07-14T19:18:14.282678+0000) 2023-07-14T19:19:06.222 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jul 14 19:19:05 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-11[120628]: 2023-07-14T19:19:05.906+0000 7f5ef7fea700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2023-07-14T19:17:58.983820+0000 front 2023-07-14T19:17:58.983785+0000 (oldest deadline 2023-07-14T19:18:25.984643+0000) 2023-07-14T19:19:06.222 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jul 14 19:19:05 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-11[120628]: 2023-07-14T19:19:05.906+0000 7f5ef7fea700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.40:6814 osd.3 since back 2023-07-14T19:18:05.985177+0000 front 2023-07-14T19:18:05.985194+0000 (oldest deadline 2023-07-14T19:18:30.685202+0000) 2023-07-14T19:19:06.223 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jul 14 19:19:05 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-11[120628]: 2023-07-14T19:19:05.906+0000 7f5ef7fea700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.130:6812 osd.4 since back 2023-07-14T19:18:14.186314+0000 front 2023-07-14T19:18:14.186278+0000 (oldest deadline 2023-07-14T19:18:38.886251+0000) 2023-07-14T19:19:06.223 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jul 14 19:19:05 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-11[120628]: 2023-07-14T19:19:05.906+0000 7f5ef7fea700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.191:6812 osd.5 since back 2023-07-14T19:18:21.787193+0000 front 2023-07-14T19:18:21.787218+0000 (oldest deadline 2023-07-14T19:18:43.487275+0000) 2023-07-14T19:19:06.223 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jul 14 19:19:05 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-11[120628]: 2023-07-14T19:19:05.906+0000 7f5ef7fea700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.130:6820 osd.7 since back 2023-07-14T19:18:34.589145+0000 front 2023-07-14T19:18:34.589216+0000 (oldest deadline 2023-07-14T19:18:58.689113+0000) 2023-07-14T19:19:07.221 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jul 14 19:19:06 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-11[120628]: 2023-07-14T19:19:06.872+0000 7f5ef7fea700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2023-07-14T19:17:44.480647+0000 front 2023-07-14T19:17:44.480662+0000 (oldest deadline 2023-07-14T19:18:06.780727+0000) 2023-07-14T19:19:07.222 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jul 14 19:19:06 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-11[120628]: 2023-07-14T19:19:06.872+0000 7f5ef7fea700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-07-14T19:17:50.782602+0000 front 2023-07-14T19:17:50.782583+0000 (oldest deadline 2023-07-14T19:18:14.282678+0000) 2023-07-14T19:19:07.222 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jul 14 19:19:06 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-11[120628]: 2023-07-14T19:19:06.872+0000 7f5ef7fea700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2023-07-14T19:17:58.983820+0000 front 2023-07-14T19:17:58.983785+0000 (oldest deadline 2023-07-14T19:18:25.984643+0000) 2023-07-14T19:19:07.222 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jul 14 19:19:06 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-11[120628]: 2023-07-14T19:19:06.872+0000 7f5ef7fea700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.40:6814 osd.3 since back 2023-07-14T19:18:05.985177+0000 front 2023-07-14T19:18:05.985194+0000 (oldest deadline 2023-07-14T19:18:30.685202+0000) 2023-07-14T19:19:07.223 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jul 14 19:19:06 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-11[120628]: 2023-07-14T19:19:06.872+0000 7f5ef7fea700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.130:6812 osd.4 since back 2023-07-14T19:18:14.186314+0000 front 2023-07-14T19:18:14.186278+0000 (oldest deadline 2023-07-14T19:18:38.886251+0000) 2023-07-14T19:19:07.223 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jul 14 19:19:06 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-11[120628]: 2023-07-14T19:19:06.872+0000 7f5ef7fea700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.191:6812 osd.5 since back 2023-07-14T19:18:21.787193+0000 front 2023-07-14T19:18:21.787218+0000 (oldest deadline 2023-07-14T19:18:43.487275+0000) 2023-07-14T19:19:07.223 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jul 14 19:19:06 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-11[120628]: 2023-07-14T19:19:06.872+0000 7f5ef7fea700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.130:6820 osd.7 since back 2023-07-14T19:18:34.589145+0000 front 2023-07-14T19:18:34.589216+0000 (oldest deadline 2023-07-14T19:18:58.689113+0000) 2023-07-14T19:19:08.222 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jul 14 19:19:07 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-11[120628]: 2023-07-14T19:19:07.874+0000 7f5ef7fea700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2023-07-14T19:17:44.480647+0000 front 2023-07-14T19:17:44.480662+0000 (oldest deadline 2023-07-14T19:18:06.780727+0000) 2023-07-14T19:19:08.222 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jul 14 19:19:07 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-11[120628]: 2023-07-14T19:19:07.874+0000 7f5ef7fea700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.130:6804 osd.1 since back 2023-07-14T19:17:50.782602+0000 front 2023-07-14T19:17:50.782583+0000 (oldest deadline 2023-07-14T19:18:14.282678+0000) 2023-07-14T19:19:08.223 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jul 14 19:19:07 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-11[120628]: 2023-07-14T19:19:07.874+0000 7f5ef7fea700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.191:6804 osd.2 since back 2023-07-14T19:17:58.983820+0000 front 2023-07-14T19:17:58.983785+0000 (oldest deadline 2023-07-14T19:18:25.984643+0000) 2023-07-14T19:19:08.223 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jul 14 19:19:07 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-11[120628]: 2023-07-14T19:19:07.874+0000 7f5ef7fea700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.40:6814 osd.3 since back 2023-07-14T19:18:05.985177+0000 front 2023-07-14T19:18:05.985194+0000 (oldest deadline 2023-07-14T19:18:30.685202+0000) 2023-07-14T19:19:08.223 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jul 14 19:19:07 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-11[120628]: 2023-07-14T19:19:07.874+0000 7f5ef7fea700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.130:6812 osd.4 since back 2023-07-14T19:18:14.186314+0000 front 2023-07-14T19:18:14.186278+0000 (oldest deadline 2023-07-14T19:18:38.886251+0000) 2023-07-14T19:19:08.223 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jul 14 19:19:07 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-11[120628]: 2023-07-14T19:19:07.874+0000 7f5ef7fea700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.191:6812 osd.5 since back 2023-07-14T19:18:21.787193+0000 front 2023-07-14T19:18:21.787218+0000 (oldest deadline 2023-07-14T19:18:43.487275+0000) 2023-07-14T19:19:08.224 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jul 14 19:19:07 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-11[120628]: 2023-07-14T19:19:07.874+0000 7f5ef7fea700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.130:6820 osd.7 since back 2023-07-14T19:18:34.589145+0000 front 2023-07-14T19:18:34.589216+0000 (oldest deadline 2023-07-14T19:18:58.689113+0000) 2023-07-14T19:19:08.894 INFO:journalctl@ceph.osd.11.smithi191.stdout:Jul 14 19:19:08 smithi191 bash[127652]: ceph-adc52f04-2279-11ee-9b33-001a4aab830c-osd-11 2023-07-14T19:19:10.156 DEBUG:teuthology.orchestra.run.smithi191:> sudo pkill -f 'journalctl -f -n 0 -u ceph-adc52f04-2279-11ee-9b33-001a4aab830c@osd.11.service' 2023-07-14T19:19:10.200 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-14T19:19:10.200 INFO:tasks.cephadm.osd.11:Stopped osd.11 2023-07-14T19:19:10.201 INFO:tasks.cephadm.mds.a:Stopping mds.a... 2023-07-14T19:19:10.201 DEBUG:teuthology.orchestra.run.smithi040:> sudo systemctl stop ceph-adc52f04-2279-11ee-9b33-001a4aab830c@mds.a 2023-07-14T19:19:10.579 INFO:journalctl@ceph.mds.a.smithi040.stdout:Jul 14 19:19:10 smithi040 systemd[1]: Stopping Ceph mds.a for adc52f04-2279-11ee-9b33-001a4aab830c... 2023-07-14T19:19:10.970 INFO:journalctl@ceph.mds.a.smithi040.stdout:Jul 14 19:19:10 smithi040 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-mds-a[131223]: 2023-07-14T19:19:10.577+0000 7fee9058e700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mds -n mds.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-07-14T19:19:10.971 INFO:journalctl@ceph.mds.a.smithi040.stdout:Jul 14 19:19:10 smithi040 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-mds-a[131223]: 2023-07-14T19:19:10.577+0000 7fee9058e700 -1 mds.a *** got signal Terminated *** 2023-07-14T19:19:14.220 INFO:journalctl@ceph.mds.a.smithi040.stdout:Jul 14 19:19:14 smithi040 bash[147274]: ceph-adc52f04-2279-11ee-9b33-001a4aab830c-mds-a 2023-07-14T19:19:14.433 DEBUG:teuthology.orchestra.run.smithi040:> sudo pkill -f 'journalctl -f -n 0 -u ceph-adc52f04-2279-11ee-9b33-001a4aab830c@mds.a.service' 2023-07-14T19:19:14.471 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-14T19:19:14.472 INFO:tasks.cephadm.mds.a:Stopped mds.a 2023-07-14T19:19:14.472 INFO:tasks.cephadm.mds.d:Stopping mds.d... 2023-07-14T19:19:14.472 DEBUG:teuthology.orchestra.run.smithi040:> sudo systemctl stop ceph-adc52f04-2279-11ee-9b33-001a4aab830c@mds.d 2023-07-14T19:19:14.721 INFO:journalctl@ceph.mds.d.smithi040.stdout:Jul 14 19:19:14 smithi040 systemd[1]: Stopping Ceph mds.d for adc52f04-2279-11ee-9b33-001a4aab830c... 2023-07-14T19:19:15.220 INFO:journalctl@ceph.mds.d.smithi040.stdout:Jul 14 19:19:14 smithi040 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-mds-d[130252]: 2023-07-14T19:19:14.859+0000 7fa87285f700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mds -n mds.d -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-07-14T19:19:15.221 INFO:journalctl@ceph.mds.d.smithi040.stdout:Jul 14 19:19:14 smithi040 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-mds-d[130252]: 2023-07-14T19:19:14.859+0000 7fa87285f700 -1 mds.d *** got signal Terminated *** 2023-07-14T19:19:19.432 INFO:journalctl@ceph.mds.d.smithi040.stdout:Jul 14 19:19:19 smithi040 bash[147417]: ceph-adc52f04-2279-11ee-9b33-001a4aab830c-mds-d 2023-07-14T19:19:19.438 DEBUG:teuthology.orchestra.run.smithi040:> sudo pkill -f 'journalctl -f -n 0 -u ceph-adc52f04-2279-11ee-9b33-001a4aab830c@mds.d.service' 2023-07-14T19:19:19.474 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-14T19:19:19.474 INFO:tasks.cephadm.mds.d:Stopped mds.d 2023-07-14T19:19:19.475 INFO:tasks.cephadm.mds.g:Stopping mds.g... 2023-07-14T19:19:19.475 DEBUG:teuthology.orchestra.run.smithi040:> sudo systemctl stop ceph-adc52f04-2279-11ee-9b33-001a4aab830c@mds.g 2023-07-14T19:19:19.720 INFO:journalctl@ceph.mds.g.smithi040.stdout:Jul 14 19:19:19 smithi040 systemd[1]: Stopping Ceph mds.g for adc52f04-2279-11ee-9b33-001a4aab830c... 2023-07-14T19:19:20.221 INFO:journalctl@ceph.mds.g.smithi040.stdout:Jul 14 19:19:19 smithi040 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-mds-g[130836]: 2023-07-14T19:19:19.855+0000 7f0bbf601700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mds -n mds.g -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-07-14T19:19:20.221 INFO:journalctl@ceph.mds.g.smithi040.stdout:Jul 14 19:19:19 smithi040 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-mds-g[130836]: 2023-07-14T19:19:19.855+0000 7f0bbf601700 -1 mds.g *** got signal Terminated *** 2023-07-14T19:19:22.219 INFO:journalctl@ceph.mds.g.smithi040.stdout:Jul 14 19:19:22 smithi040 bash[147561]: ceph-adc52f04-2279-11ee-9b33-001a4aab830c-mds-g 2023-07-14T19:19:22.417 DEBUG:teuthology.orchestra.run.smithi040:> sudo pkill -f 'journalctl -f -n 0 -u ceph-adc52f04-2279-11ee-9b33-001a4aab830c@mds.g.service' 2023-07-14T19:19:22.452 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-14T19:19:22.453 INFO:tasks.cephadm.mds.g:Stopped mds.g 2023-07-14T19:19:22.453 INFO:tasks.cephadm.mds.j:Stopping mds.j... 2023-07-14T19:19:22.454 DEBUG:teuthology.orchestra.run.smithi040:> sudo systemctl stop ceph-adc52f04-2279-11ee-9b33-001a4aab830c@mds.j 2023-07-14T19:19:22.721 INFO:journalctl@ceph.mds.j.smithi040.stdout:Jul 14 19:19:22 smithi040 systemd[1]: Stopping Ceph mds.j for adc52f04-2279-11ee-9b33-001a4aab830c... 2023-07-14T19:19:23.221 INFO:journalctl@ceph.mds.j.smithi040.stdout:Jul 14 19:19:22 smithi040 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-mds-j[129876]: 2023-07-14T19:19:22.832+0000 7f7fc4e3f700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mds -n mds.j -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-07-14T19:19:23.221 INFO:journalctl@ceph.mds.j.smithi040.stdout:Jul 14 19:19:22 smithi040 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-mds-j[129876]: 2023-07-14T19:19:22.832+0000 7f7fc4e3f700 -1 mds.j *** got signal Terminated *** 2023-07-14T19:19:25.011 INFO:journalctl@ceph.mds.j.smithi040.stdout:Jul 14 19:19:24 smithi040 bash[147702]: ceph-adc52f04-2279-11ee-9b33-001a4aab830c-mds-j 2023-07-14T19:19:25.158 DEBUG:teuthology.orchestra.run.smithi040:> sudo pkill -f 'journalctl -f -n 0 -u ceph-adc52f04-2279-11ee-9b33-001a4aab830c@mds.j.service' 2023-07-14T19:19:25.195 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-14T19:19:25.196 INFO:tasks.cephadm.mds.j:Stopped mds.j 2023-07-14T19:19:25.197 INFO:tasks.cephadm.mds.b:Stopping mds.b... 2023-07-14T19:19:25.198 DEBUG:teuthology.orchestra.run.smithi130:> sudo systemctl stop ceph-adc52f04-2279-11ee-9b33-001a4aab830c@mds.b 2023-07-14T19:19:25.492 INFO:journalctl@ceph.mds.b.smithi130.stdout:Jul 14 19:19:25 smithi130 systemd[1]: Stopping Ceph mds.b for adc52f04-2279-11ee-9b33-001a4aab830c... 2023-07-14T19:19:25.992 INFO:journalctl@ceph.mds.b.smithi130.stdout:Jul 14 19:19:25 smithi130 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-mds-b[122843]: 2023-07-14T19:19:25.553+0000 7fa520197700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mds -n mds.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-07-14T19:19:25.992 INFO:journalctl@ceph.mds.b.smithi130.stdout:Jul 14 19:19:25 smithi130 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-mds-b[122843]: 2023-07-14T19:19:25.553+0000 7fa520197700 -1 mds.b *** got signal Terminated *** 2023-07-14T19:19:29.992 INFO:journalctl@ceph.mds.b.smithi130.stdout:Jul 14 19:19:29 smithi130 bash[126212]: ceph-adc52f04-2279-11ee-9b33-001a4aab830c-mds-b 2023-07-14T19:19:30.121 DEBUG:teuthology.orchestra.run.smithi130:> sudo pkill -f 'journalctl -f -n 0 -u ceph-adc52f04-2279-11ee-9b33-001a4aab830c@mds.b.service' 2023-07-14T19:19:30.158 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-14T19:19:30.158 INFO:tasks.cephadm.mds.b:Stopped mds.b 2023-07-14T19:19:30.159 INFO:tasks.cephadm.mds.e:Stopping mds.e... 2023-07-14T19:19:30.159 DEBUG:teuthology.orchestra.run.smithi130:> sudo systemctl stop ceph-adc52f04-2279-11ee-9b33-001a4aab830c@mds.e 2023-07-14T19:19:30.492 INFO:journalctl@ceph.mds.e.smithi130.stdout:Jul 14 19:19:30 smithi130 systemd[1]: Stopping Ceph mds.e for adc52f04-2279-11ee-9b33-001a4aab830c... 2023-07-14T19:19:30.992 INFO:journalctl@ceph.mds.e.smithi130.stdout:Jul 14 19:19:30 smithi130 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-mds-e[122160]: 2023-07-14T19:19:30.543+0000 7fc2ec028700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mds -n mds.e -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-07-14T19:19:30.992 INFO:journalctl@ceph.mds.e.smithi130.stdout:Jul 14 19:19:30 smithi130 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-mds-e[122160]: 2023-07-14T19:19:30.543+0000 7fc2ec028700 -1 mds.e *** got signal Terminated *** 2023-07-14T19:19:34.483 INFO:journalctl@ceph.mds.e.smithi130.stdout:Jul 14 19:19:34 smithi130 bash[126358]: ceph-adc52f04-2279-11ee-9b33-001a4aab830c-mds-e 2023-07-14T19:19:34.489 DEBUG:teuthology.orchestra.run.smithi130:> sudo pkill -f 'journalctl -f -n 0 -u ceph-adc52f04-2279-11ee-9b33-001a4aab830c@mds.e.service' 2023-07-14T19:19:34.524 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-14T19:19:34.524 INFO:tasks.cephadm.mds.e:Stopped mds.e 2023-07-14T19:19:34.525 INFO:tasks.cephadm.mds.h:Stopping mds.h... 2023-07-14T19:19:34.525 DEBUG:teuthology.orchestra.run.smithi130:> sudo systemctl stop ceph-adc52f04-2279-11ee-9b33-001a4aab830c@mds.h 2023-07-14T19:19:34.742 INFO:journalctl@ceph.mds.h.smithi130.stdout:Jul 14 19:19:34 smithi130 systemd[1]: Stopping Ceph mds.h for adc52f04-2279-11ee-9b33-001a4aab830c... 2023-07-14T19:19:35.242 INFO:journalctl@ceph.mds.h.smithi130.stdout:Jul 14 19:19:34 smithi130 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-mds-h[122505]: 2023-07-14T19:19:34.902+0000 7f03bf40c700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mds -n mds.h -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-07-14T19:19:35.242 INFO:journalctl@ceph.mds.h.smithi130.stdout:Jul 14 19:19:34 smithi130 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-mds-h[122505]: 2023-07-14T19:19:34.902+0000 7f03bf40c700 -1 mds.h *** got signal Terminated *** 2023-07-14T19:19:38.242 INFO:journalctl@ceph.mds.h.smithi130.stdout:Jul 14 19:19:38 smithi130 bash[126514]: ceph-adc52f04-2279-11ee-9b33-001a4aab830c-mds-h 2023-07-14T19:19:38.457 DEBUG:teuthology.orchestra.run.smithi130:> sudo pkill -f 'journalctl -f -n 0 -u ceph-adc52f04-2279-11ee-9b33-001a4aab830c@mds.h.service' 2023-07-14T19:19:38.492 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-14T19:19:38.493 INFO:tasks.cephadm.mds.h:Stopped mds.h 2023-07-14T19:19:38.493 INFO:tasks.cephadm.mds.k:Stopping mds.k... 2023-07-14T19:19:38.494 DEBUG:teuthology.orchestra.run.smithi130:> sudo systemctl stop ceph-adc52f04-2279-11ee-9b33-001a4aab830c@mds.k 2023-07-14T19:19:38.742 INFO:journalctl@ceph.mds.k.smithi130.stdout:Jul 14 19:19:38 smithi130 systemd[1]: Stopping Ceph mds.k for adc52f04-2279-11ee-9b33-001a4aab830c... 2023-07-14T19:19:39.242 INFO:journalctl@ceph.mds.k.smithi130.stdout:Jul 14 19:19:38 smithi130 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-mds-k[121818]: 2023-07-14T19:19:38.902+0000 7f20a0a2e700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mds -n mds.k -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-07-14T19:19:39.243 INFO:journalctl@ceph.mds.k.smithi130.stdout:Jul 14 19:19:38 smithi130 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-mds-k[121818]: 2023-07-14T19:19:38.902+0000 7f20a0a2e700 -1 mds.k *** got signal Terminated *** 2023-07-14T19:19:42.460 INFO:journalctl@ceph.mds.k.smithi130.stdout:Jul 14 19:19:42 smithi130 bash[126657]: ceph-adc52f04-2279-11ee-9b33-001a4aab830c-mds-k 2023-07-14T19:19:42.467 DEBUG:teuthology.orchestra.run.smithi130:> sudo pkill -f 'journalctl -f -n 0 -u ceph-adc52f04-2279-11ee-9b33-001a4aab830c@mds.k.service' 2023-07-14T19:19:42.504 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-14T19:19:42.505 INFO:tasks.cephadm.mds.k:Stopped mds.k 2023-07-14T19:19:42.505 INFO:tasks.cephadm.mds.c:Stopping mds.c... 2023-07-14T19:19:42.505 DEBUG:teuthology.orchestra.run.smithi191:> sudo systemctl stop ceph-adc52f04-2279-11ee-9b33-001a4aab830c@mds.c 2023-07-14T19:19:42.863 INFO:journalctl@ceph.mds.c.smithi191.stdout:Jul 14 19:19:42 smithi191 systemd[1]: Stopping Ceph mds.c for adc52f04-2279-11ee-9b33-001a4aab830c... 2023-07-14T19:19:43.222 INFO:journalctl@ceph.mds.c.smithi191.stdout:Jul 14 19:19:42 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-mds-c[124204]: 2023-07-14T19:19:42.862+0000 7fe0fd7a4700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mds -n mds.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-07-14T19:19:43.222 INFO:journalctl@ceph.mds.c.smithi191.stdout:Jul 14 19:19:42 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-mds-c[124204]: 2023-07-14T19:19:42.862+0000 7fe0fd7a4700 -1 mds.c *** got signal Terminated *** 2023-07-14T19:19:46.229 INFO:journalctl@ceph.mds.c.smithi191.stdout:Jul 14 19:19:46 smithi191 bash[127914]: ceph-adc52f04-2279-11ee-9b33-001a4aab830c-mds-c 2023-07-14T19:19:46.420 DEBUG:teuthology.orchestra.run.smithi191:> sudo pkill -f 'journalctl -f -n 0 -u ceph-adc52f04-2279-11ee-9b33-001a4aab830c@mds.c.service' 2023-07-14T19:19:46.456 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-14T19:19:46.456 INFO:tasks.cephadm.mds.c:Stopped mds.c 2023-07-14T19:19:46.457 INFO:tasks.cephadm.mds.f:Stopping mds.f... 2023-07-14T19:19:46.457 DEBUG:teuthology.orchestra.run.smithi191:> sudo systemctl stop ceph-adc52f04-2279-11ee-9b33-001a4aab830c@mds.f 2023-07-14T19:19:46.722 INFO:journalctl@ceph.mds.f.smithi191.stdout:Jul 14 19:19:46 smithi191 systemd[1]: Stopping Ceph mds.f for adc52f04-2279-11ee-9b33-001a4aab830c... 2023-07-14T19:19:47.221 INFO:journalctl@ceph.mds.f.smithi191.stdout:Jul 14 19:19:46 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-mds-f[123863]: 2023-07-14T19:19:46.844+0000 7f07d79b7700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mds -n mds.f -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-07-14T19:19:47.222 INFO:journalctl@ceph.mds.f.smithi191.stdout:Jul 14 19:19:46 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-mds-f[123863]: 2023-07-14T19:19:46.844+0000 7f07d79b7700 -1 mds.f *** got signal Terminated *** 2023-07-14T19:19:50.471 INFO:journalctl@ceph.mds.f.smithi191.stdout:Jul 14 19:19:50 smithi191 bash[128053]: ceph-adc52f04-2279-11ee-9b33-001a4aab830c-mds-f 2023-07-14T19:19:50.483 DEBUG:teuthology.orchestra.run.smithi191:> sudo pkill -f 'journalctl -f -n 0 -u ceph-adc52f04-2279-11ee-9b33-001a4aab830c@mds.f.service' 2023-07-14T19:19:50.517 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-14T19:19:50.518 INFO:tasks.cephadm.mds.f:Stopped mds.f 2023-07-14T19:19:50.518 INFO:tasks.cephadm.mds.i:Stopping mds.i... 2023-07-14T19:19:50.518 DEBUG:teuthology.orchestra.run.smithi191:> sudo systemctl stop ceph-adc52f04-2279-11ee-9b33-001a4aab830c@mds.i 2023-07-14T19:19:50.920 INFO:journalctl@ceph.mds.i.smithi191.stdout:Jul 14 19:19:50 smithi191 systemd[1]: Stopping Ceph mds.i for adc52f04-2279-11ee-9b33-001a4aab830c... 2023-07-14T19:19:51.221 INFO:journalctl@ceph.mds.i.smithi191.stdout:Jul 14 19:19:50 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-mds-i[124544]: 2023-07-14T19:19:50.918+0000 7f19a06c4700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mds -n mds.i -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-07-14T19:19:51.222 INFO:journalctl@ceph.mds.i.smithi191.stdout:Jul 14 19:19:50 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-mds-i[124544]: 2023-07-14T19:19:50.918+0000 7f19a06c4700 -1 mds.i *** got signal Terminated *** 2023-07-14T19:19:54.471 INFO:journalctl@ceph.mds.i.smithi191.stdout:Jul 14 19:19:54 smithi191 bash[128192]: ceph-adc52f04-2279-11ee-9b33-001a4aab830c-mds-i 2023-07-14T19:19:54.531 DEBUG:teuthology.orchestra.run.smithi191:> sudo pkill -f 'journalctl -f -n 0 -u ceph-adc52f04-2279-11ee-9b33-001a4aab830c@mds.i.service' 2023-07-14T19:19:54.565 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-14T19:19:54.565 INFO:tasks.cephadm.mds.i:Stopped mds.i 2023-07-14T19:19:54.566 INFO:tasks.cephadm.mds.l:Stopping mds.l... 2023-07-14T19:19:54.566 DEBUG:teuthology.orchestra.run.smithi191:> sudo systemctl stop ceph-adc52f04-2279-11ee-9b33-001a4aab830c@mds.l 2023-07-14T19:19:54.953 INFO:journalctl@ceph.mds.l.smithi191.stdout:Jul 14 19:19:54 smithi191 systemd[1]: Stopping Ceph mds.l for adc52f04-2279-11ee-9b33-001a4aab830c... 2023-07-14T19:19:55.221 INFO:journalctl@ceph.mds.l.smithi191.stdout:Jul 14 19:19:54 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-mds-l[123521]: 2023-07-14T19:19:54.950+0000 7f2afabcf700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mds -n mds.l -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-07-14T19:19:55.222 INFO:journalctl@ceph.mds.l.smithi191.stdout:Jul 14 19:19:54 smithi191 ceph-adc52f04-2279-11ee-9b33-001a4aab830c-mds-l[123521]: 2023-07-14T19:19:54.950+0000 7f2afabcf700 -1 mds.l *** got signal Terminated *** 2023-07-14T19:19:58.471 INFO:journalctl@ceph.mds.l.smithi191.stdout:Jul 14 19:19:58 smithi191 bash[128338]: ceph-adc52f04-2279-11ee-9b33-001a4aab830c-mds-l 2023-07-14T19:19:58.543 DEBUG:teuthology.orchestra.run.smithi191:> sudo pkill -f 'journalctl -f -n 0 -u ceph-adc52f04-2279-11ee-9b33-001a4aab830c@mds.l.service' 2023-07-14T19:19:58.578 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-14T19:19:58.578 INFO:tasks.cephadm.mds.l:Stopped mds.l 2023-07-14T19:19:58.579 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid adc52f04-2279-11ee-9b33-001a4aab830c --force --keep-logs 2023-07-14T19:19:58.887 INFO:teuthology.orchestra.run.smithi040.stdout:Deleting cluster with fsid: adc52f04-2279-11ee-9b33-001a4aab830c 2023-07-14T19:20:00.546 DEBUG:teuthology.orchestra.run.smithi130:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid adc52f04-2279-11ee-9b33-001a4aab830c --force --keep-logs 2023-07-14T19:20:00.845 INFO:teuthology.orchestra.run.smithi130.stdout:Deleting cluster with fsid: adc52f04-2279-11ee-9b33-001a4aab830c 2023-07-14T19:20:02.416 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid adc52f04-2279-11ee-9b33-001a4aab830c --force --keep-logs 2023-07-14T19:20:03.065 INFO:teuthology.orchestra.run.smithi191.stdout:Deleting cluster with fsid: adc52f04-2279-11ee-9b33-001a4aab830c 2023-07-14T19:20:04.627 DEBUG:teuthology.orchestra.run.smithi040:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2023-07-14T19:20:04.665 DEBUG:teuthology.orchestra.run.smithi130:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2023-07-14T19:20:04.702 DEBUG:teuthology.orchestra.run.smithi191:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2023-07-14T19:20:04.732 INFO:tasks.cephadm:Archiving crash dumps... 2023-07-14T19:20:04.734 DEBUG:teuthology.misc:Transferring archived files from smithi040:/var/lib/ceph/adc52f04-2279-11ee-9b33-001a4aab830c/crash to /home/teuthworker/archive/pdonnell-2023-07-14_16:44:47-fs:workload-wip-pdonnell-testing-20230714.134335-distro-default-smithi/7338049/remote/smithi040/crash 2023-07-14T19:20:04.735 DEBUG:teuthology.orchestra.run.smithi040:> sudo tar c -f - -C /var/lib/ceph/adc52f04-2279-11ee-9b33-001a4aab830c/crash -- . 2023-07-14T19:20:04.780 INFO:teuthology.orchestra.run.smithi040.stderr:tar: /var/lib/ceph/adc52f04-2279-11ee-9b33-001a4aab830c/crash: Cannot open: No such file or directory 2023-07-14T19:20:04.780 INFO:teuthology.orchestra.run.smithi040.stderr:tar: Error is not recoverable: exiting now 2023-07-14T19:20:04.782 DEBUG:teuthology.misc:Transferring archived files from smithi130:/var/lib/ceph/adc52f04-2279-11ee-9b33-001a4aab830c/crash to /home/teuthworker/archive/pdonnell-2023-07-14_16:44:47-fs:workload-wip-pdonnell-testing-20230714.134335-distro-default-smithi/7338049/remote/smithi130/crash 2023-07-14T19:20:04.783 DEBUG:teuthology.orchestra.run.smithi130:> sudo tar c -f - -C /var/lib/ceph/adc52f04-2279-11ee-9b33-001a4aab830c/crash -- . 2023-07-14T19:20:04.826 INFO:teuthology.orchestra.run.smithi130.stderr:tar: /var/lib/ceph/adc52f04-2279-11ee-9b33-001a4aab830c/crash: Cannot open: No such file or directory 2023-07-14T19:20:04.827 INFO:teuthology.orchestra.run.smithi130.stderr:tar: Error is not recoverable: exiting now 2023-07-14T19:20:04.832 DEBUG:teuthology.misc:Transferring archived files from smithi191:/var/lib/ceph/adc52f04-2279-11ee-9b33-001a4aab830c/crash to /home/teuthworker/archive/pdonnell-2023-07-14_16:44:47-fs:workload-wip-pdonnell-testing-20230714.134335-distro-default-smithi/7338049/remote/smithi191/crash 2023-07-14T19:20:04.833 DEBUG:teuthology.orchestra.run.smithi191:> sudo tar c -f - -C /var/lib/ceph/adc52f04-2279-11ee-9b33-001a4aab830c/crash -- . 2023-07-14T19:20:04.890 INFO:teuthology.orchestra.run.smithi191.stderr:tar: /var/lib/ceph/adc52f04-2279-11ee-9b33-001a4aab830c/crash: Cannot open: No such file or directory 2023-07-14T19:20:04.890 INFO:teuthology.orchestra.run.smithi191.stderr:tar: Error is not recoverable: exiting now 2023-07-14T19:20:04.891 INFO:tasks.cephadm:Checking cluster log for badness... 2023-07-14T19:20:04.892 DEBUG:teuthology.orchestra.run.smithi040:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/adc52f04-2279-11ee-9b33-001a4aab830c/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'overall HEALTH_' | egrep -v '\(FS_DEGRADED\)' | egrep -v '\(MDS_FAILED\)' | egrep -v '\(MDS_DEGRADED\)' | egrep -v '\(FS_WITH_FAILED_MDS\)' | egrep -v '\(MDS_DAMAGE\)' | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(FS_INLINE_DATA_DEPRECATED\)' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v 'slow metadata IO' | egrep -v SLOW_OPS | egrep -v 'slow request' | head -n 1 2023-07-14T19:20:04.930 INFO:teuthology.orchestra.run.smithi040.stderr:grep: /var/log/ceph/adc52f04-2279-11ee-9b33-001a4aab830c/ceph.log: No such file or directory 2023-07-14T19:20:04.933 INFO:tasks.cephadm:Compressing logs... 2023-07-14T19:20:04.933 DEBUG:teuthology.orchestra.run.smithi040:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2023-07-14T19:20:04.976 DEBUG:teuthology.orchestra.run.smithi130:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2023-07-14T19:20:04.978 DEBUG:teuthology.orchestra.run.smithi191:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2023-07-14T19:20:05.005 INFO:teuthology.orchestra.run.smithi191.stderr:find: ‘/var/log/rbd-target-api’: No such file or directory 2023-07-14T19:20:05.005 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cephadm.log 2023-07-14T19:20:05.006 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/adc52f04-2279-11ee-9b33-001a4aab830c/ceph-osd.5.log 2023-07-14T19:20:05.006 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/adc52f04-2279-11ee-9b33-001a4aab830c/ceph-osd.2.log 2023-07-14T19:20:05.008 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/adc52f04-2279-11ee-9b33-001a4aab830c/ceph-osd.11.log 2023-07-14T19:20:05.008 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/adc52f04-2279-11ee-9b33-001a4aab830c/ceph-osd.5.log: /var/log/ceph/adc52f04-2279-11ee-9b33-001a4aab830c/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/adc52f04-2279-11ee-9b33-001a4aab830c/ceph-mgr.z.log 2023-07-14T19:20:05.009 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/adc52f04-2279-11ee-9b33-001a4aab830c/ceph-osd.11.log: gzip -5 --verbose -- /var/log/ceph/adc52f04-2279-11ee-9b33-001a4aab830c/ceph-osd.8.log 2023-07-14T19:20:05.009 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/adc52f04-2279-11ee-9b33-001a4aab830c/ceph-mgr.z.log: gzip -5 --verbose -- /var/log/ceph/adc52f04-2279-11ee-9b33-001a4aab830c/ceph-mds.f.log 2023-07-14T19:20:05.009 INFO:teuthology.orchestra.run.smithi191.stderr: 92.1% -- replaced with /var/log/ceph/cephadm.log.gz 2023-07-14T19:20:05.009 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/adc52f04-2279-11ee-9b33-001a4aab830c/ceph-osd.8.log: gzip -5 --verbose -- /var/log/ceph/adc52f04-2279-11ee-9b33-001a4aab830c/ceph-mds.c.log 2023-07-14T19:20:05.009 INFO:teuthology.orchestra.run.smithi040.stderr:find: ‘/var/log/rbd-target-api’: No such file or directory 2023-07-14T19:20:05.010 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/cephadm.log 2023-07-14T19:20:05.010 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/adc52f04-2279-11ee-9b33-001a4aab830c/ceph-osd.0.log 2023-07-14T19:20:05.010 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/adc52f04-2279-11ee-9b33-001a4aab830c/ceph-mds.g.log 2023-07-14T19:20:05.010 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/adc52f04-2279-11ee-9b33-001a4aab830c/ceph-mds.a.log 2023-07-14T19:20:05.011 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/adc52f04-2279-11ee-9b33-001a4aab830c/ceph-osd.0.log: /var/log/ceph/adc52f04-2279-11ee-9b33-001a4aab830c/ceph-mds.g.log: gzip -5 --verbose -- /var/log/ceph/adc52f04-2279-11ee-9b33-001a4aab830c/ceph-osd.6.log 2023-07-14T19:20:05.011 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/adc52f04-2279-11ee-9b33-001a4aab830c/ceph-mds.a.log: gzip -5 --verbose -- /var/log/ceph/adc52f04-2279-11ee-9b33-001a4aab830c/ceph-osd.3.log 2023-07-14T19:20:05.012 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/adc52f04-2279-11ee-9b33-001a4aab830c/ceph-osd.6.log: /var/log/ceph/adc52f04-2279-11ee-9b33-001a4aab830c/ceph-osd.3.log: 93.4% -- replaced with /var/log/ceph/cephadm.log.gz 2023-07-14T19:20:05.012 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/adc52f04-2279-11ee-9b33-001a4aab830c/ceph-mds.f.log: /var/log/ceph/adc52f04-2279-11ee-9b33-001a4aab830c/ceph-mds.c.log: 91.5% -- replaced with /var/log/ceph/adc52f04-2279-11ee-9b33-001a4aab830c/ceph-mgr.z.log.gz 2023-07-14T19:20:05.013 INFO:teuthology.orchestra.run.smithi130.stderr:find: ‘/var/log/rbd-target-api’: No such file or directory 2023-07-14T19:20:05.013 INFO:teuthology.orchestra.run.smithi130.stderr:gzip -5 --verbose -- /var/log/ceph/cephadm.log 2023-07-14T19:20:05.013 INFO:teuthology.orchestra.run.smithi130.stderr:gzip -5 --verbose -- /var/log/ceph/adc52f04-2279-11ee-9b33-001a4aab830c/ceph-osd.10.log 2023-07-14T19:20:05.014 INFO:teuthology.orchestra.run.smithi130.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/adc52f04-2279-11ee-9b33-001a4aab830c/ceph-mds.h.log 2023-07-14T19:20:05.014 INFO:teuthology.orchestra.run.smithi130.stderr:/var/log/ceph/adc52f04-2279-11ee-9b33-001a4aab830c/ceph-osd.10.log: gzip -5 --verbose -- /var/log/ceph/adc52f04-2279-11ee-9b33-001a4aab830c/ceph-mds.e.log 2023-07-14T19:20:05.014 INFO:teuthology.orchestra.run.smithi130.stderr:/var/log/ceph/adc52f04-2279-11ee-9b33-001a4aab830c/ceph-mds.h.log: gzip -5 --verbose -- /var/log/ceph/adc52f04-2279-11ee-9b33-001a4aab830c/ceph-osd.4.log 2023-07-14T19:20:05.014 INFO:teuthology.orchestra.run.smithi130.stderr: 91.6% -- replaced with /var/log/ceph/cephadm.log.gz 2023-07-14T19:20:05.014 INFO:teuthology.orchestra.run.smithi130.stderr:/var/log/ceph/adc52f04-2279-11ee-9b33-001a4aab830c/ceph-mds.e.log: gzip -5 --verbose -- /var/log/ceph/adc52f04-2279-11ee-9b33-001a4aab830c/ceph-osd.1.log 2023-07-14T19:20:05.015 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/adc52f04-2279-11ee-9b33-001a4aab830c/ceph-mds.i.log 2023-07-14T19:20:05.016 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/adc52f04-2279-11ee-9b33-001a4aab830c/ceph-mds.i.log: gzip -5 --verbose -- /var/log/ceph/adc52f04-2279-11ee-9b33-001a4aab830c/ceph-mon.c.log 2023-07-14T19:20:05.016 INFO:teuthology.orchestra.run.smithi130.stderr:/var/log/ceph/adc52f04-2279-11ee-9b33-001a4aab830c/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/adc52f04-2279-11ee-9b33-001a4aab830c/ceph-mds.k.log 2023-07-14T19:20:05.016 INFO:teuthology.orchestra.run.smithi130.stderr:/var/log/ceph/adc52f04-2279-11ee-9b33-001a4aab830c/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/adc52f04-2279-11ee-9b33-001a4aab830c/ceph-volume.log 2023-07-14T19:20:05.017 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/adc52f04-2279-11ee-9b33-001a4aab830c/ceph-mds.d.log 2023-07-14T19:20:05.017 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/adc52f04-2279-11ee-9b33-001a4aab830c/ceph-mgr.x.log 2023-07-14T19:20:05.017 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/adc52f04-2279-11ee-9b33-001a4aab830c/ceph-mds.d.log: gzip -5 --verbose -- /var/log/ceph/adc52f04-2279-11ee-9b33-001a4aab830c/ceph-osd.9.log 2023-07-14T19:20:05.018 INFO:teuthology.orchestra.run.smithi130.stderr:/var/log/ceph/adc52f04-2279-11ee-9b33-001a4aab830c/ceph-mds.k.log: /var/log/ceph/adc52f04-2279-11ee-9b33-001a4aab830c/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/adc52f04-2279-11ee-9b33-001a4aab830c/ceph-osd.7.log 2023-07-14T19:20:05.018 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/adc52f04-2279-11ee-9b33-001a4aab830c/ceph-mon.c.log: gzip -5 --verbose -- /var/log/ceph/adc52f04-2279-11ee-9b33-001a4aab830c/ceph-volume.log 2023-07-14T19:20:05.019 INFO:teuthology.orchestra.run.smithi130.stderr: 91.5% -- replaced with /var/log/ceph/adc52f04-2279-11ee-9b33-001a4aab830c/ceph-volume.log.gz 2023-07-14T19:20:05.024 INFO:teuthology.orchestra.run.smithi191.stderr: 91.4% -- replaced with /var/log/ceph/adc52f04-2279-11ee-9b33-001a4aab830c/ceph-mds.c.log.gz 2023-07-14T19:20:05.024 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/adc52f04-2279-11ee-9b33-001a4aab830c/ceph-mgr.x.log: /var/log/ceph/adc52f04-2279-11ee-9b33-001a4aab830c/ceph-osd.9.log: 94.8% -- replaced with /var/log/ceph/adc52f04-2279-11ee-9b33-001a4aab830c/ceph-mds.d.log.gz 2023-07-14T19:20:05.025 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/adc52f04-2279-11ee-9b33-001a4aab830c/ceph-volume.log 2023-07-14T19:20:05.025 INFO:teuthology.orchestra.run.smithi040.stderr: 92.7% -- replaced with /var/log/ceph/adc52f04-2279-11ee-9b33-001a4aab830c/ceph-mds.g.log.gz 2023-07-14T19:20:05.025 INFO:teuthology.orchestra.run.smithi040.stderr: 92.6% -- replaced with /var/log/ceph/adc52f04-2279-11ee-9b33-001a4aab830c/ceph-mds.a.log.gz 2023-07-14T19:20:05.025 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/adc52f04-2279-11ee-9b33-001a4aab830c/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/adc52f04-2279-11ee-9b33-001a4aab830c/ceph-mds.j.log 2023-07-14T19:20:05.026 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/adc52f04-2279-11ee-9b33-001a4aab830c/ceph-mds.j.log: 94.5% -- replaced with /var/log/ceph/adc52f04-2279-11ee-9b33-001a4aab830c/ceph-mds.j.log.gz 2023-07-14T19:20:05.026 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/adc52f04-2279-11ee-9b33-001a4aab830c/ceph-mon.a.log 2023-07-14T19:20:05.026 INFO:teuthology.orchestra.run.smithi130.stderr:/var/log/ceph/adc52f04-2279-11ee-9b33-001a4aab830c/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/adc52f04-2279-11ee-9b33-001a4aab830c/ceph-mon.b.log 2023-07-14T19:20:05.026 INFO:teuthology.orchestra.run.smithi130.stderr: 92.7% 91.1% -- replaced with /var/log/ceph/adc52f04-2279-11ee-9b33-001a4aab830c/ceph-mds.k.log.gz 2023-07-14T19:20:05.026 INFO:teuthology.orchestra.run.smithi130.stderr: -- replaced with /var/log/ceph/adc52f04-2279-11ee-9b33-001a4aab830c/ceph-mds.h.log.gz 2023-07-14T19:20:05.027 INFO:teuthology.orchestra.run.smithi130.stderr: 91.5% -- replaced with /var/log/ceph/adc52f04-2279-11ee-9b33-001a4aab830c/ceph-mds.e.log.gz 2023-07-14T19:20:05.027 INFO:teuthology.orchestra.run.smithi130.stderr:gzip -5 --verbose -- /var/log/ceph/adc52f04-2279-11ee-9b33-001a4aab830c/ceph-mds.b.log 2023-07-14T19:20:05.027 INFO:teuthology.orchestra.run.smithi191.stderr: 92.6% -- replaced with /var/log/ceph/adc52f04-2279-11ee-9b33-001a4aab830c/ceph-mds.f.log.gz 2023-07-14T19:20:05.028 INFO:teuthology.orchestra.run.smithi130.stderr:/var/log/ceph/adc52f04-2279-11ee-9b33-001a4aab830c/ceph-mon.b.log: gzip -5 --verbose -- /var/log/ceph/adc52f04-2279-11ee-9b33-001a4aab830c/ceph-mgr.y.log 2023-07-14T19:20:05.028 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/adc52f04-2279-11ee-9b33-001a4aab830c/ceph-mon.a.log: 91.5% -- replaced with /var/log/ceph/adc52f04-2279-11ee-9b33-001a4aab830c/ceph-volume.log.gz 2023-07-14T19:20:05.029 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/adc52f04-2279-11ee-9b33-001a4aab830c/ceph-volume.log: 91.6% -- replaced with /var/log/ceph/adc52f04-2279-11ee-9b33-001a4aab830c/ceph-mds.i.log.gz 2023-07-14T19:20:05.029 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/adc52f04-2279-11ee-9b33-001a4aab830c/ceph-mds.l.log 2023-07-14T19:20:05.030 INFO:teuthology.orchestra.run.smithi130.stderr:/var/log/ceph/adc52f04-2279-11ee-9b33-001a4aab830c/ceph-mds.b.log: /var/log/ceph/adc52f04-2279-11ee-9b33-001a4aab830c/ceph-mgr.y.log: 91.5% -- replaced with /var/log/ceph/adc52f04-2279-11ee-9b33-001a4aab830c/ceph-mgr.y.log.gz 2023-07-14T19:20:05.034 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/adc52f04-2279-11ee-9b33-001a4aab830c/ceph-mds.l.log: 91.5% -- replaced with /var/log/ceph/adc52f04-2279-11ee-9b33-001a4aab830c/ceph-volume.log.gz 2023-07-14T19:20:05.043 INFO:teuthology.orchestra.run.smithi191.stderr: 92.6% -- replaced with /var/log/ceph/adc52f04-2279-11ee-9b33-001a4aab830c/ceph-mds.l.log.gz 2023-07-14T19:20:05.081 INFO:teuthology.orchestra.run.smithi130.stderr: 92.4% -- replaced with /var/log/ceph/adc52f04-2279-11ee-9b33-001a4aab830c/ceph-mds.b.log.gz 2023-07-14T19:20:05.122 INFO:teuthology.orchestra.run.smithi040.stderr: 89.0% -- replaced with /var/log/ceph/adc52f04-2279-11ee-9b33-001a4aab830c/ceph-mgr.x.log.gz 2023-07-14T19:20:05.152 INFO:teuthology.orchestra.run.smithi130.stderr: 92.0% -- replaced with /var/log/ceph/adc52f04-2279-11ee-9b33-001a4aab830c/ceph-mon.b.log.gz 2023-07-14T19:20:05.154 INFO:teuthology.orchestra.run.smithi191.stderr: 92.0% -- replaced with /var/log/ceph/adc52f04-2279-11ee-9b33-001a4aab830c/ceph-mon.c.log.gz 2023-07-14T19:20:05.157 INFO:teuthology.orchestra.run.smithi191.stderr: 93.0% -- replaced with /var/log/ceph/adc52f04-2279-11ee-9b33-001a4aab830c/ceph-osd.11.log.gz 2023-07-14T19:20:05.236 INFO:teuthology.orchestra.run.smithi040.stderr: 92.2% -- replaced with /var/log/ceph/adc52f04-2279-11ee-9b33-001a4aab830c/ceph-osd.3.log.gz 2023-07-14T19:20:05.243 INFO:teuthology.orchestra.run.smithi130.stderr: 92.3% -- replaced with /var/log/ceph/adc52f04-2279-11ee-9b33-001a4aab830c/ceph-osd.4.log.gz 2023-07-14T19:20:05.244 INFO:teuthology.orchestra.run.smithi040.stderr: 92.5% -- replaced with /var/log/ceph/adc52f04-2279-11ee-9b33-001a4aab830c/ceph-osd.6.log.gz 2023-07-14T19:20:05.244 INFO:teuthology.orchestra.run.smithi191.stderr: 92.3% -- replaced with /var/log/ceph/adc52f04-2279-11ee-9b33-001a4aab830c/ceph-osd.5.log.gz 2023-07-14T19:20:05.260 INFO:teuthology.orchestra.run.smithi191.stderr: 92.4% -- replaced with /var/log/ceph/adc52f04-2279-11ee-9b33-001a4aab830c/ceph-osd.8.log.gz 2023-07-14T19:20:05.282 INFO:teuthology.orchestra.run.smithi040.stderr: 92.4% -- replaced with /var/log/ceph/adc52f04-2279-11ee-9b33-001a4aab830c/ceph-osd.9.log.gz 2023-07-14T19:20:05.283 INFO:teuthology.orchestra.run.smithi130.stderr: 92.5% -- replaced with /var/log/ceph/adc52f04-2279-11ee-9b33-001a4aab830c/ceph-osd.10.log.gz 2023-07-14T19:20:05.290 INFO:teuthology.orchestra.run.smithi130.stderr: 92.3% -- replaced with /var/log/ceph/adc52f04-2279-11ee-9b33-001a4aab830c/ceph-osd.7.log.gz 2023-07-14T19:20:05.315 INFO:teuthology.orchestra.run.smithi191.stderr: 92.2% -- replaced with /var/log/ceph/adc52f04-2279-11ee-9b33-001a4aab830c/ceph-osd.2.log.gz 2023-07-14T19:20:05.317 INFO:teuthology.orchestra.run.smithi191.stderr: 2023-07-14T19:20:05.318 INFO:teuthology.orchestra.run.smithi191.stderr:real 0m0.322s 2023-07-14T19:20:05.318 INFO:teuthology.orchestra.run.smithi191.stderr:user 0m1.140s 2023-07-14T19:20:05.318 INFO:teuthology.orchestra.run.smithi191.stderr:sys 0m0.052s 2023-07-14T19:20:05.323 INFO:teuthology.orchestra.run.smithi130.stderr: 92.2% -- replaced with /var/log/ceph/adc52f04-2279-11ee-9b33-001a4aab830c/ceph-osd.1.log.gz 2023-07-14T19:20:05.325 INFO:teuthology.orchestra.run.smithi130.stderr: 2023-07-14T19:20:05.325 INFO:teuthology.orchestra.run.smithi130.stderr:real 0m0.330s 2023-07-14T19:20:05.326 INFO:teuthology.orchestra.run.smithi130.stderr:user 0m1.300s 2023-07-14T19:20:05.327 INFO:teuthology.orchestra.run.smithi130.stderr:sys 0m0.064s 2023-07-14T19:20:05.329 INFO:teuthology.orchestra.run.smithi040.stderr: 92.4% -- replaced with /var/log/ceph/adc52f04-2279-11ee-9b33-001a4aab830c/ceph-osd.0.log.gz 2023-07-14T19:20:05.450 INFO:teuthology.orchestra.run.smithi040.stderr: 90.9% -- replaced with /var/log/ceph/adc52f04-2279-11ee-9b33-001a4aab830c/ceph-mon.a.log.gz 2023-07-14T19:20:05.451 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-07-14T19:20:05.452 INFO:teuthology.orchestra.run.smithi040.stderr:real 0m0.459s 2023-07-14T19:20:05.452 INFO:teuthology.orchestra.run.smithi040.stderr:user 0m1.589s 2023-07-14T19:20:05.452 INFO:teuthology.orchestra.run.smithi040.stderr:sys 0m0.066s 2023-07-14T19:20:05.452 INFO:tasks.cephadm:Archiving logs... 2023-07-14T19:20:05.453 DEBUG:teuthology.misc:Transferring archived files from smithi040:/var/log/ceph to /home/teuthworker/archive/pdonnell-2023-07-14_16:44:47-fs:workload-wip-pdonnell-testing-20230714.134335-distro-default-smithi/7338049/remote/smithi040/log 2023-07-14T19:20:05.454 DEBUG:teuthology.orchestra.run.smithi040:> sudo tar c -f - -C /var/log/ceph -- . 2023-07-14T19:20:05.711 DEBUG:teuthology.misc:Transferring archived files from smithi130:/var/log/ceph to /home/teuthworker/archive/pdonnell-2023-07-14_16:44:47-fs:workload-wip-pdonnell-testing-20230714.134335-distro-default-smithi/7338049/remote/smithi130/log 2023-07-14T19:20:05.712 DEBUG:teuthology.orchestra.run.smithi130:> sudo tar c -f - -C /var/log/ceph -- . 2023-07-14T19:20:05.906 DEBUG:teuthology.misc:Transferring archived files from smithi191:/var/log/ceph to /home/teuthworker/archive/pdonnell-2023-07-14_16:44:47-fs:workload-wip-pdonnell-testing-20230714.134335-distro-default-smithi/7338049/remote/smithi191/log 2023-07-14T19:20:05.907 DEBUG:teuthology.orchestra.run.smithi191:> sudo tar c -f - -C /var/log/ceph -- . 2023-07-14T19:20:06.075 INFO:tasks.cephadm:Removing cluster... 2023-07-14T19:20:06.075 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid adc52f04-2279-11ee-9b33-001a4aab830c --force 2023-07-14T19:20:06.371 INFO:teuthology.orchestra.run.smithi040.stdout:Deleting cluster with fsid: adc52f04-2279-11ee-9b33-001a4aab830c 2023-07-14T19:20:06.837 DEBUG:teuthology.orchestra.run.smithi130:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid adc52f04-2279-11ee-9b33-001a4aab830c --force 2023-07-14T19:20:07.136 INFO:teuthology.orchestra.run.smithi130.stdout:Deleting cluster with fsid: adc52f04-2279-11ee-9b33-001a4aab830c 2023-07-14T19:20:07.595 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid adc52f04-2279-11ee-9b33-001a4aab830c --force 2023-07-14T19:20:07.906 INFO:teuthology.orchestra.run.smithi191.stdout:Deleting cluster with fsid: adc52f04-2279-11ee-9b33-001a4aab830c 2023-07-14T19:20:08.357 INFO:tasks.cephadm:Removing cephadm ... 2023-07-14T19:20:08.358 DEBUG:teuthology.orchestra.run.smithi040:> rm -rf /home/ubuntu/cephtest/cephadm 2023-07-14T19:20:08.380 DEBUG:teuthology.orchestra.run.smithi130:> rm -rf /home/ubuntu/cephtest/cephadm 2023-07-14T19:20:08.401 DEBUG:teuthology.orchestra.run.smithi191:> rm -rf /home/ubuntu/cephtest/cephadm 2023-07-14T19:20:08.418 INFO:tasks.cephadm:Teardown complete 2023-07-14T19:20:08.419 DEBUG:teuthology.run_tasks:Unwinding manager install 2023-07-14T19:20:08.429 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2023-07-14T19:20:08.430 DEBUG:teuthology.orchestra.run.smithi040:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-07-14T19:20:08.432 DEBUG:teuthology.orchestra.run.smithi130:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-07-14T19:20:08.443 DEBUG:teuthology.orchestra.run.smithi191:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-07-14T19:20:23.577 DEBUG:teuthology.run_tasks:Unwinding manager clock 2023-07-14T19:20:23.588 INFO:teuthology.task.clock:Checking final clock skew... 2023-07-14T19:20:23.589 DEBUG:teuthology.orchestra.run.smithi040:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-07-14T19:20:23.592 DEBUG:teuthology.orchestra.run.smithi130:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-07-14T19:20:23.595 DEBUG:teuthology.orchestra.run.smithi191:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-07-14T19:20:23.609 INFO:teuthology.orchestra.run.smithi040.stderr:bash: ntpq: command not found 2023-07-14T19:20:23.613 INFO:teuthology.orchestra.run.smithi130.stderr:bash: ntpq: command not found 2023-07-14T19:20:23.615 INFO:teuthology.orchestra.run.smithi191.stderr:bash: ntpq: command not found 2023-07-14T19:20:23.616 INFO:teuthology.orchestra.run.smithi040.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-07-14T19:20:23.616 INFO:teuthology.orchestra.run.smithi040.stdout:=============================================================================== 2023-07-14T19:20:23.618 INFO:teuthology.orchestra.run.smithi040.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 27 +619us[ +727us] +/- 86ms 2023-07-14T19:20:23.618 INFO:teuthology.orchestra.run.smithi040.stdout:^* hv02.front.sepia.ceph.com 3 6 377 27 -4881us[-4774us] +/- 80ms 2023-07-14T19:20:23.619 INFO:teuthology.orchestra.run.smithi040.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 27 +4113us[+4220us] +/- 75ms 2023-07-14T19:20:23.619 INFO:teuthology.orchestra.run.smithi040.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2023-07-14T19:20:23.621 INFO:teuthology.orchestra.run.smithi130.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-07-14T19:20:23.621 INFO:teuthology.orchestra.run.smithi130.stdout:=============================================================================== 2023-07-14T19:20:23.622 INFO:teuthology.orchestra.run.smithi130.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 28 +610us[ +585us] +/- 86ms 2023-07-14T19:20:23.622 INFO:teuthology.orchestra.run.smithi130.stdout:^* hv02.front.sepia.ceph.com 3 6 377 27 -4906us[-4931us] +/- 80ms 2023-07-14T19:20:23.622 INFO:teuthology.orchestra.run.smithi130.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 27 +4104us[+4104us] +/- 75ms 2023-07-14T19:20:23.622 INFO:teuthology.orchestra.run.smithi130.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2023-07-14T19:20:23.622 INFO:teuthology.orchestra.run.smithi191.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-07-14T19:20:23.623 INFO:teuthology.orchestra.run.smithi191.stdout:=============================================================================== 2023-07-14T19:20:23.623 INFO:teuthology.orchestra.run.smithi191.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 28 +537us[ +537us] +/- 86ms 2023-07-14T19:20:23.623 INFO:teuthology.orchestra.run.smithi191.stdout:^* hv02.front.sepia.ceph.com 3 6 377 30 -4961us[-4920us] +/- 80ms 2023-07-14T19:20:23.623 INFO:teuthology.orchestra.run.smithi191.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 29 +4059us[+4059us] +/- 75ms 2023-07-14T19:20:23.623 INFO:teuthology.orchestra.run.smithi191.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2023-07-14T19:20:23.624 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2023-07-14T19:20:23.639 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2023-07-14T19:20:23.641 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2023-07-14T19:20:23.666 DEBUG:teuthology.orchestra.run.smithi040:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2023-07-14T19:20:23.773 DEBUG:teuthology.orchestra.run.smithi130:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2023-07-14T19:20:23.875 DEBUG:teuthology.orchestra.run.smithi191:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2023-07-14T19:20:23.977 DEBUG:teuthology.orchestra.run.smithi040:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-07-14T19:20:24.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-07-14T19:20:24.015 DEBUG:teuthology.orchestra.run.smithi130:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-07-14T19:20:24.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-07-14T19:20:24.050 DEBUG:teuthology.orchestra.run.smithi191:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-07-14T19:20:24.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-07-14T19:20:24.084 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2023-07-14T19:20:24.094 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2023-07-14T19:20:24.137 INFO:teuthology.task.internal:Duration was 1600.871500 seconds 2023-07-14T19:20:24.138 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2023-07-14T19:20:24.162 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2023-07-14T19:20:24.163 DEBUG:teuthology.orchestra.run.smithi040:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-07-14T19:20:24.165 DEBUG:teuthology.orchestra.run.smithi130:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-07-14T19:20:24.167 DEBUG:teuthology.orchestra.run.smithi191:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-07-14T19:20:24.214 INFO:teuthology.orchestra.run.smithi040.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-07-14T19:20:24.215 INFO:teuthology.orchestra.run.smithi191.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-07-14T19:20:24.215 INFO:teuthology.orchestra.run.smithi130.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-07-14T19:20:24.503 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2023-07-14T19:20:24.503 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi040.front.sepia.ceph.com 2023-07-14T19:20:24.504 DEBUG:teuthology.orchestra.run.smithi040:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-07-14T19:20:24.528 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi130.front.sepia.ceph.com 2023-07-14T19:20:24.529 DEBUG:teuthology.orchestra.run.smithi130:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-07-14T19:20:24.552 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi191.front.sepia.ceph.com 2023-07-14T19:20:24.552 DEBUG:teuthology.orchestra.run.smithi191:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-07-14T19:20:24.574 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2023-07-14T19:20:24.575 DEBUG:teuthology.orchestra.run.smithi040:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-07-14T19:20:24.577 DEBUG:teuthology.orchestra.run.smithi130:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-07-14T19:20:24.595 DEBUG:teuthology.orchestra.run.smithi191:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-07-14T19:20:24.661 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2023-07-14T19:20:24.661 DEBUG:teuthology.orchestra.run.smithi040:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-07-14T19:20:24.664 DEBUG:teuthology.orchestra.run.smithi130:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-07-14T19:20:24.669 DEBUG:teuthology.orchestra.run.smithi191:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-07-14T19:20:24.767 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2023-07-14T19:20:24.782 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2023-07-14T19:20:24.783 DEBUG:teuthology.orchestra.run.smithi040:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-07-14T19:20:24.810 DEBUG:teuthology.orchestra.run.smithi130:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-07-14T19:20:24.838 DEBUG:teuthology.orchestra.run.smithi191:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-07-14T19:20:24.865 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2023-07-14T19:20:24.882 DEBUG:teuthology.orchestra.run.smithi040:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-07-14T19:20:24.885 DEBUG:teuthology.orchestra.run.smithi130:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-07-14T19:20:24.888 DEBUG:teuthology.orchestra.run.smithi191:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-07-14T19:20:24.909 INFO:teuthology.orchestra.run.smithi040.stdout:kernel.core_pattern = core 2023-07-14T19:20:24.911 INFO:teuthology.orchestra.run.smithi130.stdout:kernel.core_pattern = core 2023-07-14T19:20:24.931 INFO:teuthology.orchestra.run.smithi191.stdout:kernel.core_pattern = core 2023-07-14T19:20:24.957 DEBUG:teuthology.orchestra.run.smithi040:> test -e /home/ubuntu/cephtest/archive/coredump 2023-07-14T19:20:24.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-07-14T19:20:24.981 DEBUG:teuthology.orchestra.run.smithi130:> test -e /home/ubuntu/cephtest/archive/coredump 2023-07-14T19:20:24.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-07-14T19:20:24.998 DEBUG:teuthology.orchestra.run.smithi191:> test -e /home/ubuntu/cephtest/archive/coredump 2023-07-14T19:20:25.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-07-14T19:20:25.016 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2023-07-14T19:20:25.028 INFO:teuthology.task.internal:Transferring archived files... 2023-07-14T19:20:25.029 DEBUG:teuthology.misc:Transferring archived files from smithi040:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/pdonnell-2023-07-14_16:44:47-fs:workload-wip-pdonnell-testing-20230714.134335-distro-default-smithi/7338049/remote/smithi040 2023-07-14T19:20:25.029 DEBUG:teuthology.orchestra.run.smithi040:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-07-14T19:20:25.067 DEBUG:teuthology.misc:Transferring archived files from smithi130:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/pdonnell-2023-07-14_16:44:47-fs:workload-wip-pdonnell-testing-20230714.134335-distro-default-smithi/7338049/remote/smithi130 2023-07-14T19:20:25.068 DEBUG:teuthology.orchestra.run.smithi130:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-07-14T19:20:25.110 DEBUG:teuthology.misc:Transferring archived files from smithi191:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/pdonnell-2023-07-14_16:44:47-fs:workload-wip-pdonnell-testing-20230714.134335-distro-default-smithi/7338049/remote/smithi191 2023-07-14T19:20:25.111 DEBUG:teuthology.orchestra.run.smithi191:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-07-14T19:20:25.154 INFO:teuthology.task.internal:Removing archive directory... 2023-07-14T19:20:25.154 DEBUG:teuthology.orchestra.run.smithi040:> rm -rf -- /home/ubuntu/cephtest/archive 2023-07-14T19:20:25.157 DEBUG:teuthology.orchestra.run.smithi130:> rm -rf -- /home/ubuntu/cephtest/archive 2023-07-14T19:20:25.159 DEBUG:teuthology.orchestra.run.smithi191:> rm -rf -- /home/ubuntu/cephtest/archive 2023-07-14T19:20:25.198 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2023-07-14T19:20:25.211 INFO:teuthology.task.internal:Not uploading archives. 2023-07-14T19:20:25.212 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2023-07-14T19:20:25.238 INFO:teuthology.task.internal:Tidying up after the test... 2023-07-14T19:20:25.238 DEBUG:teuthology.orchestra.run.smithi040:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-07-14T19:20:25.241 DEBUG:teuthology.orchestra.run.smithi130:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-07-14T19:20:25.243 DEBUG:teuthology.orchestra.run.smithi191:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-07-14T19:20:25.255 INFO:teuthology.orchestra.run.smithi040.stdout: 262307 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Jul 14 19:20 /home/ubuntu/cephtest 2023-07-14T19:20:25.258 INFO:teuthology.orchestra.run.smithi130.stdout: 262169 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Jul 14 19:20 /home/ubuntu/cephtest 2023-07-14T19:20:25.259 INFO:teuthology.orchestra.run.smithi191.stdout: 262169 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Jul 14 19:20 /home/ubuntu/cephtest 2023-07-14T19:20:25.260 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2023-07-14T19:20:25.280 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2023-07-14T19:20:25.315 INFO:teuthology.nuke:Checking targets against current locks 2023-07-14T19:20:30.352 DEBUG:teuthology.nuke:shortname: smithi040 2023-07-14T19:20:30.352 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-07-14T19:20:39.028 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi040.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2023-07-14_16:44:47-fs:workload-wip-pdonnell-testing-20230714.134335-distro-default-smithi/7338049', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.6', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-07-14 18:46:28.583416', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCpUxo5ay/ucdkX9rleNMm8Ofp24XRXUtJPOK3Ji3XBpTiUgmmwneaY6316+3CG+k4K0p3IebanuyTvZApjm02/dNCvu8yD3rjEpWVgIuqYH8O3lEdN9DXcCX+7xHJE0m18+ZPsJQVsNrNJxY+5zqyNnBc8jpZq2LStsWwPOKPWhauANRkOBFSGDoLppWZbLx/yJs+R0qoNLLK9LpKQTUnqsapXvbD27OpPahzj/qbNxFPPyY6XrrldZ6qEO+EjJOujFLH3gzv0kJu10DCuzxrpz6ston4RPi1O3GgcHnF4Mqr/W9VwZ94UTMh+k4dIFRVqJEKrcwRHa5kFsvH/WhVx0Ygr/MiV/Q86mmV7JT+/MpzIRTtWyei5L9IBeZIE9cwzj+mOk6SD/t1roIVEGVV/uF2AyFDVBPr6F8oZ822OGpUyyaDsWTitPNdTX4TC3zdSrLc9tyDa1P74eH/ZLeTSf+C6pM/cepOhQaa6XzpDiPSuJUm9HOQxUlEAvcqa7Ac='} 2023-07-14T19:20:39.033 DEBUG:teuthology.nuke:shortname: smithi130 2023-07-14T19:20:39.034 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-07-14T19:20:39.063 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi130.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2023-07-14_16:44:47-fs:workload-wip-pdonnell-testing-20230714.134335-distro-default-smithi/7338049', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.6', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-07-14 18:46:28.580568', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLVvN3+LmzPZ4ixiwyCkS3KZLZ9R6MOPm+kPtaJbl8fExpzbk8TudXCcJWHBL7DGPWFvye2jT6sei9eiX0EUQYs='} 2023-07-14T19:20:39.065 DEBUG:teuthology.nuke:shortname: smithi191 2023-07-14T19:20:39.065 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-07-14T19:20:39.092 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi191.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2023-07-14_16:44:47-fs:workload-wip-pdonnell-testing-20230714.134335-distro-default-smithi/7338049', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.6', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-07-14 18:46:28.582158', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCpUxo5ay/ucdkX9rleNMm8Ofp24XRXUtJPOK3Ji3XBpTiUgmmwneaY6316+3CG+k4K0p3IebanuyTvZApjm02/dNCvu8yD3rjEpWVgIuqYH8O3lEdN9DXcCX+7xHJE0m18+ZPsJQVsNrNJxY+5zqyNnBc8jpZq2LStsWwPOKPWhauANRkOBFSGDoLppWZbLx/yJs+R0qoNLLK9LpKQTUnqsapXvbD27OpPahzj/qbNxFPPyY6XrrldZ6qEO+EjJOujFLH3gzv0kJu10DCuzxrpz6ston4RPi1O3GgcHnF4Mqr/W9VwZ94UTMh+k4dIFRVqJEKrcwRHa5kFsvH/WhVx0Ygr/MiV/Q86mmV7JT+/MpzIRTtWyei5L9IBeZIE9cwzj+mOk6SD/t1roIVEGVV/uF2AyFDVBPr6F8oZ822OGpUyyaDsWTitPNdTX4TC3zdSrLc9tyDa1P74eH/ZLeTSf+C6pM/cepOhQaa6XzpDiPSuJUm9HOQxUlEAvcqa7Ac='} 2023-07-14T19:20:39.103 INFO:teuthology.orchestra.console:Power off smithi040 2023-07-14T19:20:39.105 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi040.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-07-14T19:20:39.130 INFO:teuthology.orchestra.console:Power off smithi130 2023-07-14T19:20:39.132 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi130.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-07-14T19:20:39.157 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi040.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-14T19:20:39.174 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi130.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-14T19:20:39.203 INFO:teuthology.orchestra.console:Power off smithi191 2023-07-14T19:20:39.204 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi191.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-07-14T19:20:39.231 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi191.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-14T19:20:43.190 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi040.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-14T19:20:43.218 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi130.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-14T19:20:43.263 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi191.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-14T19:20:48.305 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi040.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-14T19:20:48.333 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi130.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-14T19:20:48.375 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi191.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-14T19:20:53.424 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi040.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-14T19:20:53.447 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi130.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-14T19:20:53.490 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi191.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-14T19:20:53.702 INFO:teuthology.orchestra.console:Power off for smithi191 completed 2023-07-14T19:20:58.540 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi040.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-14T19:20:58.559 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi130.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-14T19:21:03.655 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi040.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-14T19:21:03.670 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi130.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-14T19:21:08.772 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi040.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-14T19:21:08.783 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi130.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-14T19:21:13.888 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi040.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-14T19:21:13.900 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi130.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-14T19:21:19.003 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi040.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-14T19:21:19.013 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi130.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-14T19:21:24.117 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi040.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-14T19:21:24.128 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi130.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-14T19:21:29.232 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi040.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-14T19:21:29.243 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi130.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-14T19:21:34.348 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi040.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-14T19:21:34.359 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi130.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-14T19:21:39.463 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi040.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-14T19:21:39.474 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi130.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-14T19:21:44.577 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi040.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-14T19:21:44.588 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi130.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-14T19:21:49.692 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi040.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-14T19:21:49.704 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi130.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-14T19:21:50.806 WARNING:teuthology.contextutil:'wait for power off' reached maximum tries (16) after waiting for 60.0 seconds 2023-07-14T19:21:50.808 ERROR:teuthology.orchestra.console:Failed to power off smithi040 2023-07-14T19:21:50.824 WARNING:teuthology.contextutil:'wait for power off' reached maximum tries (16) after waiting for 60.0 seconds 2023-07-14T19:21:50.825 ERROR:teuthology.orchestra.console:Failed to power off smithi130 2023-07-14T19:21:55.939 INFO:teuthology.run:Summary data: description: fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-comp omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/random export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/direct_io}} duration: 1600.8715000152588 failure_reason: 'Command failed on smithi040 with status 1: "sudo TESTDIR=/home/ubuntu/cephtest bash -c ''ceph fs subvolumegroup pin cephfs _nogroup random 0.01''"' flavor: default owner: scheduled_pdonnell@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=8dd2d741af5b472b961c3ef46ed37445 status: fail success: false 2023-07-14T19:21:55.940 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-07-14T19:22:01.045 INFO:teuthology.run:FAIL